JP2022186598A - Information processing apparatus, imaging apparatus, information processing method and control method, and program - Google Patents

Abstract

To appropriately select frames of a photographed image to easily check a motion shake that may occur in main photographing.SOLUTION: An information processing apparatus comprises: acquisition means that acquires a plurality of first photographed images obtained by performing first photographing according to a first photographic parameter and motion information on a subject in the plurality of first photographed images; estimation means that, when second photographing is performed according to a second photographic parameter set independently of the first photographic parameter, estimates a motion shake of the subject in a second photographed image obtained through the second photographing from the motion information and the second photographic parameter; setting means that sets the first photographic parameter and the second photographic parameter; and notification means that notifies information on the motion shake. The acquisition means acquires the motion information from the plurality of first photographed images obtained according to the first photographic parameter that is changed in correspondence with an exposure time in the second photographing related to the second photographic parameter.SELECTED DRAWING: Figure 7

Description

The present invention relates to a technique for notifying subject blur that occurs in a captured image.

2. Description of the Related Art Some imaging devices such as digital still cameras are equipped with a shooting mode that gives priority to shutter speed (hereinafter referred to as "shutter speed priority mode"). In the shutter speed priority mode, the photographer sets a desired shutter speed, and the imaging apparatus automatically sets exposure setting values other than the shutter speed, such as the aperture value and ISO sensitivity. A photographer can shoot at a desired shutter speed by using the shutter speed priority mode. For example, by setting a high shutter speed before shooting and shooting in the shutter speed priority mode, it is possible to shoot an image with less motion blur.

Japanese Patent Application Laid-Open No. 2002-200002 discloses a technique of detecting a moving area between time-series images captured during preparatory shooting and emphasizing the moving area. Here, the preparatory photographing is photographing in which the image composition is adjusted and the photographing conditions are set while looking at the electronic viewfinder and the rear liquid crystal of the imaging device before the actual photographing. In addition, the main shooting is shooting that is executed by the imaging apparatus based on the composition and shooting condition settings determined in the preliminary shooting, triggered by the photographer's action such as pressing the shutter button. According to Japanese Patent Application Laid-Open No. 2002-100000, the photographer can visually confirm the motion area during preparatory photographing.

JP 2008-172667 A

According to Patent Literature 1, it is possible to detect a motion area between time-series images captured during preparatory shooting and emphasize the motion area, thereby prompting the user to act to reduce blurring.

However, in Patent Document 1, in order to extract a moving region between images in time series, shooting at an appropriate frame rate and shutter speed according to the speed and amount of movement of the subject is not taken into consideration.

For example, when portrait photography is performed at night, the amount of light that can be captured by the imaging device is smaller than that during the daytime, resulting in a longer exposure time. As a result, even a small shake or movement of a person causes blurring of the subject. At this time, in order to appropriately notify the user of subject blur, it is necessary to detect subject movement that affects the exposure time (corresponding to the set shutter speed) for photographing. However, a frame rate value sufficient to detect small movements of the subject is not always set in advance, and there is a problem that the user cannot be appropriately notified of the occurrence of subject blurring.

The present invention has been made in view of the above problems. to detect Accordingly, it is an object of the present invention to provide an information processing device, an imaging device, an information processing method, a control method, and a program that enable confirmation of motion blur that occurs in actual photography.

In order to solve the above problems, one information processing apparatus of the present invention provides a plurality of first photographed images obtained by first photographing with a first photographing parameter and the plurality of first photographed images. acquisition means for acquiring motion information of a subject in an image; estimating means for estimating motion blurring of a subject in a second captured image obtained by the second capturing from the capturing parameters; setting means for setting the first capturing parameter and the second capturing parameter; and the motion notification means for notifying blur information, wherein the obtaining means obtains the first photographing parameter changed corresponding to the exposure time in the second photographing related to the second photographing parameter. The motion information is obtained from the plurality of first captured images.

Further, one of the information processing apparatuses of the present invention acquires a plurality of first captured images obtained by performing a first photographing with a first photographing parameter and motion information of a subject in the plurality of first captured images. and an obtaining means for obtaining a second photographing from the motion information and the second photographing parameters when the second photographing is performed with the second photographing parameters set independently of the first photographing parameters. estimating means for estimating motion blurring of the subject in the second captured image obtained in step; setting means for setting the first shooting parameter and the second shooting parameter; and reporting means for reporting information on the motion blurring. and wherein the obtaining means determines an image to be used for obtaining the motion information from the plurality of first captured images based on the exposure time in the second capturing according to the second capturing parameter. characterized by

ADVANTAGE OF THE INVENTION According to the present invention, an information processing device, an imaging device, and an information processing device capable of displaying a blurring notification image that enables accurate confirmation of motion blurring that occurs in main photography by appropriately selecting a frame of a preparatory photography image for detecting motion. It becomes possible to provide a processing method, a control method, and a program.

A diagram showing a configuration example of the implementation of the present invention Flowchart showing shooting processing of the imaging device 100 A diagram showing a configuration example of a motion blur notification image generation unit FIG. 2 is a diagram for explaining the processing of S202 in FIG. 2 of the first embodiment; A diagram explaining the relationship between the amount of motion of the subject and the allowable amount of motion Diagram showing preparatory captured images and motion vectors Flowchart showing motion vector calculation method Diagram showing how motion vectors are calculated Diagram showing motion blur notification method Acquisition timing chart of preparatory image and main image Diagram explaining the exposure time of the main shooting and the frame interval of the preparatory shooting A diagram explaining the integration of the amount of motion between frames Diagram explaining frame rate control FIG. 2 is a diagram for explaining the processing of S208 in FIG. 2 of the second embodiment;

Preferred embodiments of the present invention will be described below with reference to the drawings.

In the first embodiment of the present invention, motion information calculated from images during preparatory shooting is used to estimate motion blur equivalent to actual shooting, and the motion blur expected to occur in main shooting is displayed as a motion blur notification image. An imaging device that notifies the user will be described.

FIG. 1 is a block diagram showing a configuration example of an imaging device as an example of an information processing device and an imaging device according to Embodiment 1 of the present invention. A configuration example of Embodiment 1 of the present invention will be described below with reference to FIG. The information processing apparatus according to the present invention can be applied to any electronic device capable of processing captured images. These electronic devices may include, for example, mobile phones, game machines, tablet terminals, personal computers, watch-type or glasses-type information terminals, and the like.

The control unit 101 is, for example, a CPU, and reads a control program for each block included in the imaging apparatus 100 (eg, a camera) from the ROM 102 described later, develops it in the RAM 103 described later, and executes it. Thereby, the control unit 101 controls the operation of each block included in the imaging device 100 . The ROM 102 is an electrically erasable/recordable non-volatile memory, and stores an operation program for each block included in the imaging apparatus 100 as well as parameters required for the operation of each block. A RAM 103 is a rewritable volatile memory, and is used for development of programs executed by the control unit 101 and the like, temporary storage of data generated by the operation of each block included in the imaging apparatus 100, and the like.

The optical system 104 is composed of a lens group including a zoom lens and a focus lens, and forms a subject image on an imaging surface of an imaging unit 105, which will be described later. The imaging unit 105 is, for example, an imaging device such as a CCD or CMOS sensor, photoelectrically converts an optical image formed on the imaging surface of the imaging unit 105 by the optical system 104, and converts the obtained analog image signal into an A/D signal. Output to conversion unit 106 . The A/D converter 106 converts the input analog image signal into digital image data. Digital image data output from the A/D converter 106 is temporarily stored in the RAM 103 .

The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 103 . The image processing unit 107 includes a motion blurring notification image generation unit 300, which will be described later. Generate. The recording unit 108 is a storage medium such as a removable memory card. A recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103 . A display unit 109 is a display device such as an LCD, and displays images stored in the RAM 103 and images recorded in the recording unit 108, and displays an operation user interface for receiving instructions from the user. In addition, the display unit 109 displays an image captured by the imaging unit 105 for composition adjustment or the like during preparatory shooting. The instruction input unit 110 is a touch panel, a mouse, or the like, and inputs instructions from the user.

Next, the first embodiment will be described in detail with reference to the flowchart of FIG. The following processing is realized by the control unit 101 controlling each unit of the apparatus according to the program stored in the ROM 102 . The flowchart in FIG. 2 is executed when preparatory photographing is started.

In S<b>201 , the user turns on the imaging device 100 . Then, in S201, the control unit 101 controls the optical system 104 and the imaging unit 105 in response to the power of the imaging apparatus 100 being turned on to start preparatory photographing such as composition adjustment. The control unit 101 maintains a predetermined frame rate during this preparatory photographing period, and the image capturing apparatus 100 sequentially captures images and displays the acquired captured images on the display unit 109 . The user can adjust the composition while viewing the displayed preparatory photographed image. Here, an image captured during preparatory photographing is defined as a preparatory photographed image.

In S202, the imaging apparatus 100 sets shooting conditions (shooting parameters) for a pre-shooting image to be taken to detect the amount of movement of the subject in the composition. The shooting conditions here mainly indicate frame rate and shutter speed. Then, the highest frame rate and the highest speed within the range that does not affect the processing for calculating the evaluation value used for controlling the automatic functions performed by general imaging devices such as AE (auto exposure) and AF (auto focus) control Sets the shutter speed of the . Also, even when the shutter speed is set to a high speed, the optical system 104 is controlled in accordance with the shutter speed, the aperture of the lens is changed, and the ISO sensitivity of the imaging unit 105 is set so that shooting can be performed under appropriate exposure conditions. By changing the exposure, control is performed so that shooting can be performed with an appropriate exposure. The imaging apparatus 100 captures images that are continuous in time series during preparatory imaging under the imaging conditions set in S202. The faster the shutter speed, the less accumulated blurring of a moving subject in the captured images, and the smaller the amount of movement of the subject between consecutive captured images, making it possible to capture the movement of the subject in detail.

In S<b>203 , the user uses the instruction input unit 110 to set shooting parameters for actual shooting. The control unit 101 can set the exposure time as a shooting parameter for preparatory shooting according to the input from the instruction input unit 110 . Note that the imaging apparatus 100 may be configured to automatically set the imaging parameters for the actual imaging. Also, the imaging parameters for main imaging may be set independently of the imaging parameters for preparatory imaging. In this embodiment, it is assumed that the set shooting parameters for the main shooting are used as the shooting parameters for the main shooting after pressing of the shutter button (instruction for main shooting), which will be described later, is detected.

In S204, the control unit 101 determines whether the motion blur notification is set to ON or OFF. The setting of ON or OFF of the motion blur notification is set by the user, for example, using the instruction input unit 110, and once set, the set value is retained. When the control unit 101 determines in S204 that the motion blurring notification is set to ON, the motion blurring notification image generating unit 300 described later generates motion blurring notification by superimposing a motion blurring notification plane on the preparatory photographed image in S205. Generate an image. Then, the control unit 101 displays the motion blur notification image generated in S206 on the display unit 109 .

When the control unit 101 determines in S204 that the motion blurring notification is set to OFF, no motion blurring notification image is generated. Therefore, in S<b>206 , the display unit 109 displays the preparatory photographed image without any motion blurring notification.

In S207, the control unit 101 determines whether or not the shutter button of the instruction input unit 110 has been pressed by the user's operation. Here, if the shutter button depression input is configured to receive two steps of input methods, such as half-press for instructing shooting preparation operation and full-press for instructing actual shooting, whether or not full-press has been performed is determined. shall judge. If only a simple one-step input is accepted, it is determined whether or not the one-step input has been made.

If the control unit 101 determines that the shutter button has not been pressed, the process returns to S203 and repeats the processes from S203 to S206. As a result, when the motion blur notification image is displayed on the display unit 109, the user can prevent the motion blur that would occur in the subject when the actual shooting is performed with the currently set shooting parameters for the actual shooting during the preparatory shooting. can be easily verified. If the confirmed motion blur is not the motion blur that the user likes, the user returns to S202 without pressing the shutter button, and resets the shutter speed (exposure time) for the actual photographing again in S203. good too. In this manner, during preparatory photography, the user repeatedly sets the exposure time for actual photography while confirming the motion blurring notification image displayed on the display unit 109 until the desired motion blur is achieved, and presses the shutter button at a photo opportunity. Press.

If the control unit 101 determines in S207 that the shutter button has been pressed down, it assumes that the shooting instruction for main shooting has been received, and proceeds to S208 to control the optical system 104, the imaging unit 105, etc., and set up before the preparatory shooting. Actual photography is performed based on the obtained photography parameters. A main photographed image obtained by main photographing is output to the display unit 109 and the recording unit 108 by the control unit 101, and is displayed on the display device of the display unit 109 and recorded on a recording medium by the recording unit 108 or transferred to an external device. Output is done.

Next, a configuration example of the motion blurring notification image generation unit 300 included in the image processing unit 107, which is a feature of the present invention, will be described with reference to FIG.

FIG. 3 is a diagram showing a configuration example of the motion blurring notification image generation unit 300. As shown in FIG. A motion blur notification image generation unit 300 includes a motion vector calculation unit 301 that calculates a motion vector of a subject from a comparison between images, and an estimated motion blur calculation that estimates the motion blur of the subject at the time of actual shooting based on the calculated motion vector. A portion 302 is included. Furthermore, a notification plane generation unit 303 that creates data for notifying motion blur based on the estimated motion blur of the subject, and an image that undergoes superimposition processing in order to superimpose the motion blur notification plane on the captured image in this embodiment. A superimposed portion 304 is included. Details of the operation of each unit will be described later.

One or more of the functional blocks shown in FIG. 3 may be implemented by hardware such as an ASIC or programmable logic array (PLA), or implemented by a programmable processor such as a CPU or MPU executing software. may It may also be implemented by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as main entities, the same hardware can be implemented as the main entities.

Next, the processing performed by the motion blur notification image generation unit 300 in S205 of FIG. 2 will be described with reference to the flowchart of FIG. Each step of this flowchart is executed by the control unit 101 or by each unit of the imaging device 100 including the motion blur notification image generation unit 300 according to instructions from the control unit 101 .

In S401, the control unit 101 sets the imaging conditions set in S202. Then, the control unit 101 controls the optical system 104 and the image capturing unit 105, and the image capturing apparatus 100 continuously captures preparatory captured images for detecting motion blurring of the subject during actual capturing.

In S402, the control unit 101 acquires shutter speed information automatically set by the user or the camera to perform actual photography.

In S403, the control unit 101 selects a preparatory image frame to be used for detecting the movement of the subject (S404 described later) from the frame rate information of the preparatory photographing and the shutter speed of the main photographing. A method of selecting the preparation image frames will be described in detail with reference to FIGS.

FIG. 10 is a timing chart showing acquisition timings of a frame group (1001) of preparatory captured images and a main captured image (1011), with time T on the horizontal axis.

The horizontal axis represents the shooting time, and 1021 in (a) of FIG. 10 and 1022 in (b) of FIG. frame is used to calculate the amount of motion at that moment). Reference numeral 1021 denotes control for selecting two frames before and after movement is detected. Reference numeral 1022 denotes the frame rate information of the preparatory photographing in this embodiment and the shutter speed of the main photographing to select the preparatory image frames. This is an example of selection of The selection of a captured image frame 1021 is a method generally used when calculating the amount of motion of a subject.

A method of selecting a preparation image frame in this embodiment will be described with reference to FIG. FIG. 11 is a diagram showing the relationship between the frame intervals of a plurality of preparatory photographed images (1111 to 1114) and the exposure time of the main photographed image (1101) photographed at the shutter speed of the main photographing. A description will be given below using specific numerical values. If the shutter speed set in the actual shooting is set to 1/30, an image with an exposure time of 1/30 second is obtained in the actual shooting. In other words, if the movement of the subject in 1/30 second is small, an image with little accumulated blur can be obtained. On the other hand, the pre-shot images are shot at a shutter speed at which accumulated blurring is less likely to occur between frames in order to analyze motion. For example, assume that multiple frames are shot with an exposure time shorter than 1/120 second. At this time, it is assumed that the imaging frame rate of the preparatory shot images for continuous shooting is 120 fps (continuous shooting of 120 images per second). In order to detect the movement of the subject during the exposure time of the main shot image, the control unit 101 selects frames in which the frame interval of the preparatory shot image is close to the exposure time of the main shot image. Since the intervals between adjacent preparatory shot image frames 1111 to 1114 are uniform, the interval between four frames (1111 and 1114) is 1/30 second, which is equivalent to the exposure time of the main shot image. In other words, it means that a frame containing the movement of the subject for a time period equivalent to 1/30 second of the exposure time of the captured image is selected. The smaller the difference between the selected frame interval of the preparatory photographed image and the shutter speed of the main photographed image, the smaller the error in the process of S405, which will be described later, and the more accurate estimation can be performed. Therefore, the control unit 101 selects frames in which the shutter speed set in the actual photographing and the frame interval of the preparatory photographed image are as close as possible. The control unit 101 selects the preparatory captured images having frame intervals that satisfy the following equation (2).
F_Period (frame interval)≧(shutter speed for main shooting×frame rate for preparatory shooting)×α
(F_Period is the smallest integer that satisfies the formula, α is a constant) Formula (2)

By selecting the preparatory photographed image frames for calculating the amount of movement of the subject that satisfies Equation (2), it is possible to select frames at intervals close to the same time interval as the shutter speed set in the main photographing.

The relationship between the frame interval of preparatory photographed images and the shutter speed of actual photographing has been described above with reference to FIG. 10 . In this way, in the present embodiment, the control unit 101 controls the selection of frame 1022 in FIG. 10B as the frame interval of the preparatory shot images. Also, it is desirable to set α in equation (2) so that it decreases as the ISO sensitivity increases. By using α, it is possible to prevent the frame interval from becoming too long when the shooting conditions are severe. Also, α may be changed for each shutter speed of the actual photographing. For example, when the shutter speed of the main photographing is long, if the frame interval of the preparatory photographed images becomes too long, complex movements of the subject are likely to be included between the frames, and the accuracy of detecting the movement of the subject may decrease. Here, α can be changed for each shutter speed of the actual photographing, thereby suppressing a decrease in the accuracy of motion detection of the subject.

In S404 of FIG. 4, the frame of the pre-shot image selected in S403 is input to the motion blur notification image generation unit 300, and the motion vector calculation unit 301 uses the input image to calculate the amount of motion of the subject. As for the processing for calculating the motion amount of the subject by the motion vector calculation unit 303, as shown in FIG. explain.

As the motion amount of the subject, the motion vector calculation unit 301 calculates a motion vector between successive pre-shot images in time series. A motion vector is a vector representing the amount of horizontal movement and the amount of vertical movement of a subject between pre-captured images. A motion vector calculation method will be described in detail with reference to FIGS. 7 and 8. FIG.

FIG. 7 is a flowchart showing motion vector calculation processing. In the present invention, the block matching method will be described as an example of the motion vector calculation method, but the motion vector calculation method is not limited to this example, and may be, for example, a gradient method. Each step of this flowchart is executed by the control unit 101 or by each unit of the imaging apparatus 100 including the motion blur notification image generation unit 300 according to instructions from the control unit 101 .

In step S<b>701 , the control unit 101 receives two temporally adjacent preparatory photographed images acquired during preparatory photographing, sets the preparatory photographed image of the Mth frame as a reference frame, and sets the preparatory photographed image of the M+1th frame. as the reference frame.

In S702, the control unit 101 arranges a reference block 802 of N×N pixels in a reference frame 801 as shown in FIG.

In S703, the control unit 101 designates (N+n)×(N+n) pixels around the same coordinates 804 as the center coordinates of the reference block 802 of the reference frame 801 as a search range 805, as shown in FIG. set.

In step S<b>704 , the control unit 101 controls the motion vector calculation unit 301 to determine the correlation between the reference block 802 of the reference frame 801 and the reference block 806 of N×N pixels with different coordinates existing within the search range 805 of the reference frame 803 . Calculation is performed to calculate the correlation value. A correlation value is calculated based on the sum of absolute differences (SAD) between frames for the pixels of the reference block 802 and the reference block 806 . In other words, the coordinate with the smallest sum of absolute differences between frames is the coordinate with the highest correlation value. Note that the method of calculating the correlation value is not limited to the method of obtaining the sum of the inter-frame absolute difference values, and may be, for example, a method of calculating the correlation value based on the inter-frame difference sum of squares or the normal cross-correlation value. In the example of FIG. 8, it is assumed that reference block 806 indicates the highest correlation.

In S705, the control unit 101 controls the motion vector calculation unit 301 to calculate a motion vector based on the reference block coordinates indicating the highest correlation value obtained in S704. In the example of FIG. 8, a motion vector is obtained based on the coordinates 804 corresponding to the central coordinates of the reference block 802 of the reference frame 801 and the central coordinates of the reference block 806 in the search range 805 of the reference frame 803 . That is, the inter-coordinate distance and direction from the same coordinates 804 to the center coordinates of the reference block 806 are obtained as motion vectors.

In S<b>706 , the control unit 101 determines whether motion vectors have been calculated for all pixels of the reference frame 801 . If it is determined in S706 that motion vectors for all pixels have not been calculated, the process returns to S702. Then, in step 602, a reference block 802 of N×N pixels is placed in the reference frame 801 described above, centering on pixels for which motion vectors have not been calculated. That is, the motion vector calculation unit 301 repeats the processing from S702 to S705 while moving the reference block 802 in FIG. 8 to calculate the motion vectors of all pixels of the reference frame 801. An example of this motion vector is shown in FIG. 6(b). FIG. 6(b) is a diagram showing an example of motion vectors of the preparatory photographed image of FIG. 6(a).

The preparatory photographed image in FIG. 6A shows an example in which a dog 601 is running leftward. FIG. 6B shows a representative example of motion vectors when the subject is moving in this way. In the example of FIG. 6B, a running dog 601 is detected as a motion vector in the left direction, and a resting dog 602 and a fence in the background are detected as motion vectors 0, so the motion vectors are illustrated. not

Also, the motion vector calculation unit 301 may calculate a motion vector for each predetermined pixel instead of calculating the motion vector for all pixels. Through the above processing, a motion vector between frames of temporally adjacent pre-shot images is calculated.

In S405 of FIG. 4, the estimated motion blur calculation unit 302 calculates the relationship between the calculated subject motion amount and the target motion blur amount. At this time, if the frame-to-frame time of the preparatory photographed image selected in S403 is not equal to the shutter speed (exposure time) of the actual photographing, it is impossible to accurately calculate the subject blur in the actual photographing. Therefore, the amount of motion of the object detected in S404 is multiplied by the correction gain obtained by the equation (4) to convert the amount of motion, thereby estimating the motion blur in the actual photographing.
Correction gain=Shutter speed for main shooting/Between frames of selected preparatory shot image (Equation (4))

Subsequently, the control unit 101 determines whether the amount of motion blur in the actual shooting estimated by multiplying the correction gain of equation (4), that is, the estimated amount of motion blur, is larger or smaller than the target amount of motion blur. judgment processing is performed. The target amount of motion blur referred to here is an allowable amount of motion blur that can be contained to such an extent that motion blur is inconspicuous when photographing is performed at a predetermined shutter speed. The allowable amount of motion blur is determined by the size of an imaging device such as a CCD or CMOS sensor, the number of pixels, and the resolution of a display. For example, if the image sensor is APS-C, the number of pixels is 200,000 pixels, and the allowable amount of motion blur is 5 pixels or less on a full HD (1920 x 1080 pixels) PC display, it should be possible to detect motion of 5 pixels or less between frames. It is necessary to In other words, a resolution that is sufficient to detect motion that is equal to or less than the allowable motion blur amount is required. A specific example will be described with reference to FIG.

FIGS. 5A and 5B express the relationship between the motion amount of the subject calculated from the preparatory photographed image and the allowable motion blur amount using motion vectors. FIG. 5(a) shows the case where the amount of motion of the subject exceeds the allowable amount of motion blur, and FIG. 5(b) shows the case where the amount of motion of the subject is equal to or less than the allowable amount of motion blur. The relationship between the estimated amount of motion of the subject and the target amount of motion blur can be represented by Equation (1).
n=subject motion amount/permissible motion blur amount (1)

At this time, if n is larger than 1, the allowable amount of motion blur is exceeded, so the control unit 101 determines that subject blur occurs. Conversely, if n is 1 or less, the control unit 101 determines that the allowable motion blur amount is not exceeded. That is, it can be determined that subject blur is within the allowable range. The target permissible amount of motion blurring can also be determined by the user's settings.

In S406 of FIG. 4, based on an instruction from the control unit 101, the idle plane generation unit 303 generates an image for notifying motion blur based on the relationship between the amount of motion of the subject obtained in S405 and the target amount of motion blur. The processing of S406 will be explained using FIG. The notification of motion blur is a process of notifying the user of the degree of motion blur during actual shooting. Therefore, the amount of motion calculated from the preparatory shot image is estimated as the amount of motion (estimated motion blur amount) when converted under the exposure conditions of the actual shooting, and a blur notification image is created that allows the estimated amount of motion to be visually confirmed. .

First, for each amount of movement of the subject, the notification plane generation unit 303 uses the result of equation (1) calculated in step S405 to create an image plane for notification of motion blur. After that, the image superimposing unit 304 superimposes the motion blurring notification plane on the preparatory photographed image to generate a motion blurring notification image. An example of the blur notification image will be described with reference to FIG.

FIG. 9 shows three examples of motion blur notification images. By displaying the motion blurring notification image on the display unit 109 during preparatory shooting, the user can easily confirm motion blurring.

First, FIG. 9(a) shows an example in which motion blur is notified by icon display. In S<b>405 , the control unit 101 calculates the ratio of the amount of motion blur of the subject for which n in Equation (1) is greater than 1 to the total amount of motion blur of the subject. When the ratio is equal to or greater than a predetermined ratio, the motion blurring notification image generating unit 300 creates a motion blurring notification plane 901 as shown in FIG. A motion blur notification image such as (a) is generated.

FIG. 9B shows an example of notification of motion blurring by motion blurring frame display. Here, a method of generating a motion blurring notification image by displaying a motion blurring frame will be described. In step S405, the control unit 101 divides the image into regions, and calculates a number that makes n greater than 1 using equation (1) for the motion blur amount of the subject included in each divided region. That is, based on an instruction from the control unit 101, the notification plane generation unit 303 generates a motion blurring frame 902 as a motion blurring notification plane as shown in FIG. A motion blur notification image such as that shown in FIG.

FIG. 9(c) shows an example in which motion blurring is reported by highlighting an edge where motion blurring has occurred. Here, a method of generating a motion blurring notification image by highlighting motion blurring edges will be described. First, in step S405, the notification plane generation unit 303 detects the edge strength of the preparatory photographed image. An existing method such as a Sobel filter is used to calculate the edge strength, and detailed description thereof is omitted. Then, the notification plane generation unit 303 highlights the motion blurred edge as shown in 903 in FIG. It is created as a motion blurring notification plane like this. The image superimposing unit 304 superimposes the created motion blurring notification plane on the preparatory photographed image to generate a motion blurring notification image as shown in FIG. 9(c). An example 903 in FIG. 9C shows an example of thickening the motion blur edge. By emphasizing such motion blurring edges, it is possible to visually recognize motion blurring in fine parts of the subject in the actual shooting. Another example of the highlighting method is to extract pixels whose edge strength is greater than or equal to a predetermined value and whose estimated motion blur is greater than or equal to a predetermined value. display method.

The process of generating a motion blurring notification image by the motion blurring notification image generation unit 300 by performing the processes from S401 to S406 in FIG. 4 has been described above. The control unit 101 or each unit of the imaging apparatus 100 including the motion blur notification image generation unit 300 according to instructions from the control unit 101 performs the flowchart in FIG. It is possible to suitably determine the imaging conditions.

According to the present invention, it is possible to appropriately select a pre-photographed image for confirming motion blurring that occurs in main photography and detect motion, thereby making it possible to confirm appropriate motion blurring that occurs in main photography.

In the present invention, in order to estimate the amount of subject blurring that occurs during the exposure time in the actual photographing, the control for selecting frames in accordance with the frame intervals of preparatory photographing images and detecting the amount of motion has been described. In order to detect more detailed movement, it is also possible to detect the amount of movement of the subject using all the preparatory photographed images included within the frame interval of the preparatory photographed images corresponding to the exposure time of the main photographed image. . For example, referring to FIGS. 11 and 12, when the exposure time (1/30 second) for the main shooting and the frame rate for the preparatory shooting are 120 fps as described in FIG. Become. Therefore, in Example 1, the motion vector calculation unit 301 calculates the motion amount of the subject using 1111 and 1114 in FIG. 11 (1201 in FIG. 12). Here, the control unit 101 integrates the amounts of subject movement obtained respectively in 1111 and 1112, 1112 and 1113, and 1113 and 1114 (1202 in FIG. 12). As a result, even if the subject moves intricately during 1/30 second (rotational motion, acceleration motion, etc.) in the pre-photographed image, the motion of the subject in 1/30 second from 1111 to 1114 can be accurately calculated. can do.

Further, when the allowable amount of motion blur is not intentionally determined by the photographer in advance, the size of the motion vector within a range that does not exceed the limit range of motion detection may be set as the allowable amount of motion blur. At that time, the control unit 101 determines whether the motion of the subject exceeds the motion detection range. As a determination method, it can be obtained from a method of measuring the movement of the main body of the imaging apparatus 100 such as gyro information or the amount of change in the SAD value at the time of motion detection being smaller than a predetermined value. When the control unit 101 determines that the allowable motion blurring amount is exceeded, control is performed so that the shutter speed and the frame rate are increased.

In the present invention, as a motion blurring notification method, an example of displaying a motion blurring notification on the display unit 109 has been described, but the motion blurring notification method is not limited to this. For example, motion blurring may be notified by sound, light, or vibration. Specifically, when the number of pixels showing an estimated motion blur of a predetermined value or more among the estimated motion blur of each pixel occupies a predetermined ratio or more of the entire screen, motion blur notification sound, notification light, and notification vibration are generated. generate. In this embodiment, the configuration of the notification plane generating unit 303 and the image superimposing unit 304 and the processing flow of steps S405 and S406 are not required. Instead, a speaker is provided in the imaging apparatus 100, and in step S205, in conjunction with the display of the preparatory photographed image on the display unit 109, the control unit 101 sounds a notification sound from the speaker, lights a notification light, or generates notification vibration. should wake up.

In the present invention, the control unit 101 changes the selection of the frame for motion detection from the preparatory shot image according to the shutter speed of the main shooting. The selection of the frame of the preparatory photographed image to be performed may be changed. Only when the shutter speed of the actual shooting is stable, frame control of the pre-shot image for which motion detection is performed is performed. It is possible to prevent an undesired motion blur notification image from being displayed on the display unit 109 when it is not stable.

As an example of determining whether or not the motion of the subject is stable, it can be determined by determining whether or not the motion of the moving subject is stable as a change in the motion vector after changing the shooting conditions for the preparatory shooting. At this time, if the change in motion vector per unit time becomes large, it is possible to change the shooting conditions for the preparatory shooting again.

In the present invention, the control unit 101 changes the selection of the frame for motion detection from the preparatory photographed image according to the shutter speed of the main photographing. You can For example, when the shutter speed of the main photographing is longer than a predetermined value, if the ISO sensitivity of the main photographing is higher than the predetermined value, the shutter speed of the preparatory photographing is changed to be longer. Conversely, if the ISO sensitivity for the main shooting is low, the shutter speed for the preparatory shooting is shortened. This makes it possible to capture an appropriate preparatory photographed image for detecting movement by making a trade-off between the effects of noise and accumulated blur as the photographing conditions for preparatory photographing for detecting movement.

Next, an example of estimating the amount of subject blurring in main photography and generating a motion blurring notification image by changing the frame rate of preparatory photography in accordance with the shutter speed of main photography will be described. By realizing this embodiment, it is possible to efficiently control the frame rate of the preparatory photographed image in order to notify the subject blurring of the main photographing.

Since the present embodiment can be realized with the same configuration of the imaging apparatus 100 as that of the first embodiment shown in FIG. 1, description of the block diagram of the imaging apparatus 100 is omitted. Also, the overall processing performed by the control unit 101 in the second embodiment is the same as the processing shown in the flowchart of FIG. 2, and thus description thereof is omitted.

Next, the processing of S202 and S205 in FIG. 2, which is a feature of the present invention, will be described in detail.

In the processing of S202, the control unit 101 changes the frame rate of the preparatory photographing in accordance with the shutter speed of the main photographing. This will be described with reference to FIG. 13 .

FIG. 13A is a diagram showing a general timing chart of preparatory shooting and main shooting, and FIG. This is a modified example. As an example, it is assumed that the shutter speed of the main photographing is 1/30 second and the frame rate of the preparatory photographing before changing the frame rate shown in FIG. 13(a) is 120 fps. At this time, the actually shot image 1311 is an image with an exposure time of 1/30 second because the shutter speed is 1/30 second. In this embodiment, the control unit 101 changes the frame rate of the preparatory photographing so as to approach the exposure time of the actually photographed image as shown in FIG. 13(b). For example, when the frame rate of preparatory photography is 120 fps, the interval between adjacent frames is 1/120 second. Here, if the exposure time of the main shot image is 1/30 second, the control unit 101 changes the frame rate of the preparatory shooting from 120 fps to 30 fps. That is, the frame rate of the preparatory photographing is given by the following formula (3).
Post-change frame rate for preparatory photography = β/shutter speed for main photography Equation (3)
(β is a constant)

β is a preset constant that is changed according to the ISO sensitivity and shutter speed of the actual photographing, like α in the formula (2) of the first embodiment. The control unit 101 changes the frame rate of the preparatory photographing so as to satisfy the above formula (3), thereby setting the frame interval to appropriately detect subject blurring at the shutter speed of the main photographing. In FIG. 13B after the frame rate is changed, the case where the movement of the subject is detected using the adjacent frames when preparatory shooting is performed at the frame rate in FIG. 13A and the frame in FIG. Comparing the interval (1321), it can be seen that the frame interval of the preparatory photographing of 1321 is widened and the time interval is close to the exposure time of the main photographing 1311. Therefore, the motion of the subject corresponding to the exposure time of the main shooting 1311 can be estimated by controlling the frame rate of the preparatory shooting as shown in FIG. 13B. It should be noted that the frame rate does not have to match the exposure time of the actual photography here, and the frame rate may be changed to a value equal to or greater than the exposure time of the actual photography.

Next, the processing of S205 will be described using the flowchart of FIG. Each step of this flowchart is executed by the control unit 101 or by each unit of the imaging apparatus 100 including the motion blur notification image generation unit 300 according to instructions from the control unit 101 .

In S1401, the control unit 101 performs the same processing as in S401 to set imaging conditions. Then, the control unit 101 controls the optical system 104 and the image capturing unit 105, and the image capturing apparatus 100 continuously captures preparatory captured images.

In S1402, the control unit 101 performs processing equivalent to that of S402.

In S1403, the control unit 101 performs the same processing as in S404, and the motion vector calculation unit 301 calculates the motion amount of the subject from the pre-captured image captured in S1401.

In S1404, the estimated motion blur calculation unit 302 performs the same processing as in S405, and the imaging apparatus 100 calculates the relationship between the calculated subject motion amount and the target motion blur amount.

In S1305, the control unit 101 performs the same processing as S406 in FIG. The notification plane generation unit 303 creates an image plane for notification of motion blur based on the estimated subject blur amount of the main captured image in the process of S1404, and the image superimposition unit 304 superimposes the motion blur notification plane on the preparatory captured image. , to generate a motion blur notification image.

S202 and S205, which are characteristic processes, have been specifically described for the entire process of the flowchart in FIG. 2 controlled by the control unit 101 of the second embodiment.

By executing the configuration of the second embodiment, the control unit 101 can control the frame rate of the preparatory photographing for detecting the movement of the subject that affects the exposure time of the main photographing. Therefore, it is possible to accurately detect the motion blur that occurs in the actual shooting. In addition, since it is not necessary to continuously shoot preparatory shot images at a high frame rate in order to detect the motion of the subject, it is possible to achieve accurate motion detection with low power consumption.

In the present invention, the control unit 101 changes the selection of frames for motion detection from the preparatory photographed image according to the shutter speed of the main photographing. Here, if the shutter speed of the main shooting is higher than the maximum value of the frame rate that can be set in the imaging device 100, the frame rate in the preparatory shooting is fixed at the maximum value, and the preparatory shooting image with the closest frame interval is selected. It is good also as a structure which detects a motion by doing. Note that under this condition, motion blur does not occur at the time of actual shooting, so motion blur does not need to be estimated. In this way, when the shutter speed of the main photographing is higher than a certain value, that is, when the exposure time of the main photographing is shorter than a certain time, the frame rate may be fixed to a specific value.

Alternatively, a threshold value may be provided for the shutter speed of the main photographing, and the control unit may control the frame rate in the preparatory photographing according to the threshold value. For example, a first threshold value and a second threshold value larger than the first threshold value are provided for the shutter speed of the actual photographing. At this time, if the shutter speed for the actual shooting is less than the first threshold, the control unit 101 performs control at the first frame rate. Also, if the shutter speed of the actual shooting is greater than or equal to the first threshold and less than the second threshold, the second frame rate is controlled, and if the shutter speed of the actual photography is greater than or equal to the second threshold, the third frame rate is controlled. The unit 101 may control.

In the present invention, an example has been described in which motion blur is notified when the estimated motion blur is equal to or greater than a predetermined value. For example, it is conceivable to express in three colors (red, blue, and yellow) depending on the magnitude of the estimated amount of motion blur. In this case, the allowable amount of motion that can be detected is set to the narrowest range of the amount of motion so that the range of magnitude of the amount of motion represented by each color can be expressed. For example, when the expression colors of motion blur are yellow: 0 to 10 pixels, blue: 11 to 15 pixels, and red: 16 pixels or more with respect to the magnitude of the vector, the blue range is the narrowest section as the amount of motion. That is, it can be realized by setting 5 pixels in the blue range (11 to 15) as the allowable motion blur amount.

In the present invention, the allowable amount of motion blur is set, and the frame rate, which is the imaging condition of the preparatory shot image, is controlled so that the motion that is equal to or less than the allowable amount of motion blur can be detected. Also, the shooting conditions may be changed (sports priority mode, etc.).

The shooting conditions for the preparatory shot image may be changed when it is detected that the motion of the subject is stable. In addition, after changing the shooting conditions for the preparatory shooting once, whether or not the motion of the subject is stable is obtained as a change in the motion vector. may be changed.

When the allowable motion blur amount cannot be set in advance, the allowable motion amount may be set to the size of the motion vector within a range that does not exceed the limit range of motion detection. At that time, the control unit 101 determines whether the motion of the subject exceeds the motion detection range. As a determination method, it can be obtained from a method of measuring the movement of the main body of the imaging apparatus 100 such as gyro information or the amount of change in the SAD value at the time of motion detection being smaller than a predetermined value. When the control unit 101 determines that the allowable motion blur amount is exceeded, control is performed so that the shutter speed and the frame rate are increased.

In the present invention, as a motion blurring notification method, an example of displaying a motion blurring notification on the display unit 109 has been described, but the motion blurring notification method is not limited to this. For example, motion blurring may be notified by sound, light, or vibration. Specifically, the motion blur notification sound may be generated when the number of pixels showing an estimated motion blur of a predetermined value or more among the estimated motion blur of each pixel occupies a predetermined ratio or more of the entire screen. In such a case, blur notification sound generating means such as a microphone may be prepared as motion blur notification means.

(Other embodiments)
Although the present invention has been described above based on the embodiments, the present invention is not limited to these embodiments, and various forms without departing from the gist of the present invention are also included in the present invention.

The object of the present invention can also be achieved as follows. That is, a storage medium in which software program codes describing procedures for realizing the functions of the above-described embodiments are recorded is supplied to the system or device. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the novel functions of the present invention, and the storage medium and program storing the program code constitute the present invention.

Storage media for supplying program codes include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. Also, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, non-volatile memory card, ROM, etc. can be used.

Moreover, the functions of the above-described embodiments are realized by making the program code read by the computer executable. Furthermore, based on the instructions of the program code, the OS (operating system) running on the computer performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments. is also included.

It also includes the following cases: First, a program code read from a storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

100 imaging device 101 control unit 102 ROM
103 RAM
104 optical system 105 imaging unit 106 A/D conversion unit 107 image processing unit 108 recording unit 109 display unit 110 instruction input unit

Claims (36)

Acquisition means for acquiring a plurality of first shot images obtained by performing a first shooting with a first shooting parameter and motion information of a subject in the plurality of first shot images;
When the second imaging is performed with the second imaging parameters set independently of the first imaging parameters, the second imaging parameters obtained in the second imaging are obtained from the motion information and the second imaging parameters. an estimating means for estimating the motion blur of the subject in the captured image of
setting means for setting the first imaging parameter and the second imaging parameter;
and a notification means for notifying information of the motion blur,
The acquiring means acquires the motion information from the plurality of first captured images obtained by the first shooting parameters changed corresponding to the exposure time in the second shooting related to the second shooting parameters. An information processing device characterized by acquiring. 2. The imaging apparatus according to claim 1, wherein said first shooting parameter includes at least one of a frame rate and shutter speed in said first shooting. 3. An information processing apparatus according to claim 2, wherein said estimation means estimates said motion blur using said motion information and a frame rate which is one of said first imaging parameters. 4. The information processing apparatus according to claim 2, wherein said setting means changes a frame rate, which is one of said first photographing parameters, based on an exposure time in said second photographing. 5. An information processing apparatus according to claim 4, wherein said acquisition means changes said frame rate to a value equal to or greater than the exposure time in said second photographing. 5. The method according to claim 4, wherein said acquiring means acquires said motion information from two images that are consecutive in time series among said plurality of first images acquired at said changed frame rate. information processing equipment. 3. The information processing apparatus according to claim 1, wherein said notification means notifies said motion blur by displaying information corresponding to said motion blur on a display unit. 3. The information according to claim 1, wherein the notification means notifies the motion blur by superimposing information corresponding to the motion blur on the first captured image and displaying the information on the display unit. processing equipment. 3. The information processing apparatus according to claim 1, wherein when said estimation means estimates motion blur larger than a predetermined value, said notification means reports said motion blur. The notification means is
10. The information processing apparatus according to claim 9, wherein motion blur is notified by displaying an image in which an icon is drawn on the display unit. The notification means is
10. The information processing apparatus according to claim 9, wherein motion blurring is notified by displaying an image in which a motion blurring frame is drawn on said display unit. The notification means is
10. The information processing apparatus according to claim 9, wherein motion blurring is notified by displaying an image drawn by emphasizing a motion blurring edge on the display unit. 3. The information processing apparatus according to claim 1, wherein when the notification by said notification means is OFF, said notification means does not notify said motion blurring. 3. The information processing apparatus according to claim 1, wherein said first photographing is preparatory photographing for photographing an image at a time different from said second photographing. 15. The information processing apparatus according to claim 14, wherein said second photography is main photography, and said first photography is preparatory photography performed before said main photography. a motion blurring notification sound generating means for generating a motion blurring notification sound;
3. The information processing apparatus according to claim 1, wherein said motion blur notification means uses sound to notify motion blur. Acquisition means for acquiring a plurality of first shot images obtained by performing a first shooting with a first shooting parameter and motion information of a subject in the plurality of first shot images;
When the second imaging is performed with the second imaging parameters set independently of the first imaging parameters, the second imaging parameters obtained in the second imaging are obtained from the motion information and the second imaging parameters. an estimating means for estimating the motion blur of the subject in the captured image of
setting means for setting the first imaging parameter and the second imaging parameter;
and a notification means for notifying information of the motion blur,
The acquiring means determines an image to be used for acquiring the motion information from the plurality of first photographed images based on the exposure time in the second photographing related to the second photographing parameter. information processing equipment. 18. The information processing apparatus according to claim 17, wherein said first shooting parameter includes at least one of frame rate and shutter speed in said first shooting. The obtaining means obtains the motion information from the plurality of first images according to a difference between a frame rate, which is one of the first photographing parameters, and an exposure time value in the second photographing. 9. The information processing apparatus according to claim 8, wherein an image to be used for is determined. 20. The information processing apparatus according to claim 19, wherein said acquiring means selects two images from said plurality of first images so as to have a time interval corresponding to an exposure time in said second photographing. . 19. The information processing apparatus according to claim 17, wherein the notification means notifies the motion blur by displaying information corresponding to the motion blur on a display unit. 19. The information according to claim 17, wherein the notification means notifies the motion blur by superimposing information corresponding to the motion blur on the first captured image and displaying the information on the display unit. processing equipment. 19. The information processing apparatus according to claim 17 or 18, wherein when said estimation means estimates motion blur larger than a predetermined value, said notification means reports said motion blur. The notification means is
24. The information processing apparatus according to claim 23, wherein the motion blur is notified by displaying an image in which an icon is drawn on the display unit. The notification means is
24. The information processing apparatus according to claim 23, wherein motion blurring is notified by displaying an image in which a motion blurring frame is drawn on said display unit. The notification means is
24. The information processing apparatus according to claim 23, wherein motion blurring is notified by displaying an image drawn by emphasizing a motion blurring edge on said display unit. 19. The information processing apparatus according to claim 17, wherein when the notification by said notification means is OFF, said notification means does not notify said motion blurring. 19. The information processing apparatus according to claim 17, wherein said first photographing is preparatory photographing for photographing an image at a time different from said second photographing. 29. The information processing apparatus according to claim 28, wherein said second photography is main photography, and said first photography is preparatory photography performed before said main photography. a motion blurring notification sound generating means for generating a motion blurring notification sound;
19. The information processing apparatus according to claim 17, wherein said motion blur notification means notifies motion blur with sound. an imaging means for capturing an image of a subject formed through an optical system and outputting the image;
Acquisition means for acquiring a plurality of first shot images obtained by performing a first shooting with a first shooting parameter and motion information of a subject in the plurality of first shot images;
When the second imaging is performed with the second imaging parameters set independently of the first imaging parameters, the second imaging parameters obtained in the second imaging are obtained from the motion information and the second imaging parameters. an estimating means for estimating the motion blur of the subject in the captured image of
setting means for setting the first imaging parameter and the second imaging parameter;
and a notification means for notifying information of the motion blur,
The acquiring means acquires the motion information from the plurality of first captured images obtained by the first shooting parameters changed corresponding to the exposure time in the second shooting related to the second shooting parameters. An imaging device characterized by acquiring. an imaging means for capturing an image of a subject formed through an optical system and outputting the image;
Acquisition means for acquiring a plurality of first shot images obtained by performing a first shooting with a first shooting parameter and motion information of a subject in the plurality of first shot images;
When the second imaging is performed with the second imaging parameters set independently of the first imaging parameters, the second imaging parameters obtained in the second imaging are obtained from the motion information and the second imaging parameters. an estimating means for estimating the motion blur of the subject in the captured image of
setting means for setting the first imaging parameter and the second imaging parameter;
and a notification means for notifying information of the motion blur,
The acquiring means determines an image to be used for acquiring the motion information from the plurality of first photographed images based on the exposure time in the second photographing related to the second photographing parameter. imaging device. an acquisition step of acquiring a plurality of first captured images obtained by performing a first capturing with a first capturing parameter and motion information of a subject in the plurality of first captured images;
When the second imaging is performed with the second imaging parameters set independently of the first imaging parameters, the second imaging parameters obtained in the second imaging are obtained from the motion information and the second imaging parameters. an estimation step of estimating the motion blur of the subject in the captured image of
a setting step of setting the first imaging parameter and the second imaging parameter;
a notification step of notifying information of the motion blur,
obtaining the motion information from the plurality of first captured images obtained by the first imaging parameter changed corresponding to the exposure time in the second imaging according to the second imaging parameter in the obtaining step; A control method for an image processing apparatus, characterized by acquiring a an acquisition step of acquiring a plurality of first captured images obtained by performing a first capturing with a first capturing parameter and motion information of a subject in the plurality of first captured images;
When the second imaging is performed with the second imaging parameters set independently of the first imaging parameters, the second imaging parameters obtained in the second imaging are obtained from the motion information and the second imaging parameters. an estimation step of estimating the motion blur of the subject in the captured image of
a setting step of setting the first imaging parameter and the second imaging parameter;
a notification step of notifying information of the motion blur,
characterized in that, in the acquisition step, an image to be used for acquiring the motion information from the plurality of first captured images is determined based on the exposure time in the second imaging related to the second imaging parameter. A control method for an information processing device. A program for causing a computer to execute each step of the control method according to claim 33 or 34. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 35.

Images