JP2011223265A - Image evaluation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image evaluation device for evaluating an image frame-out.SOLUTION: A control device 104 extracts a main photogenic subject area including a main photogenic subject from a plurality of images photographed in a time sequence, determines whether at least a part of the extracted main photogenic subject area is framed out or not, specifies images determined to be framed out as frame-out images, and evaluates an image whose main photogenic subject is framed out as a failure image by relatively lowering the evaluation of the specified frame-out images.

Description

  The present invention relates to an image evaluation apparatus.

  The following image capturing apparatuses are known. This image photographing apparatus performs image evaluation such as exposure, contrast, and blur, and deletes an image whose evaluation value is a predetermined threshold value or less from a recording medium (for example, Patent Document 1).

JP 2006-50494 A

  However, when an image is evaluated by applying the technology in the conventional image capturing device to an image captured in time series, the subject or the image capturing device has moved, and thus the image where the subject is out of frame is identified and identified. The image could not be evaluated based on the results.

An image evaluation apparatus according to the present invention includes an extraction unit that extracts a main subject region including a main subject from an image photographed in time series, and at least a part of the main subject region extracted by the extraction unit is framed out of the image. Determining means for determining whether or not, specifying means for specifying an image determined to be out of frame by the determining means as a frame-out image, and evaluation of the frame-out image specified by the specifying means are acquired in time series And an evaluation means for lowering in the image.
In the present invention, the determination means detects a change in position of the main subject area in the images between a plurality of images taken in time series, the main subject area is located at the end point of the image, and the main subject area When the area decreases, it may be determined that at least a part of the main subject region is out of frame from the image.
The determining means uses an image captured at a certain time as a reference image, and calculates a difference between the reference image and a target image captured at a different time from the reference image, thereby obtaining a plurality of images captured in time series The position change of the main subject region in the image may be detected.
The determination unit may perform alignment between the reference image and the target image using the feature points in the reference image and the feature points in the target image, and calculate the difference after the alignment.
When the alignment unit performs alignment, if there is no corresponding region for calculating the difference from the reference image outside the target image, the determination unit corresponds to the corresponding region from the image acquired at a time different from the target image. And the difference may be calculated.
The determination unit may calculate the difference after normalizing the brightness of the reference image and the brightness of the target image.
Reference image update means for acquiring and updating the reference image at a predetermined update frequency is further provided, and the reference image update means is provided for at least one of the shooting time interval of images taken in time series and the movement information of the imaging device. Based on this, a predetermined update frequency may be determined.
The extraction means uses an image taken at a certain time as a reference image, extracts a main subject area from the reference image, and the determination means sets feature points in the main subject area extracted from the reference image, If there is a feature point that cannot be associated with a feature point in the target image that is captured after that, at least a part of the main subject area in the target image is out of the image. You may make it determine.
The determination means is a feature point that can be associated with a distribution of feature points that cannot be associated with feature points in the target image among feature points set in the main subject region, or with feature points in the target image. Based on the distribution in the main subject, it may be determined that at least a part of the main subject region in the target image is out of frame from the image.
The determination unit further distributes the feature points in the main subject that cannot be associated with the feature points in the target image or the feature points in the target image when the position of the main subject region in the target image satisfies a predetermined condition. It may be determined that occlusion has occurred in the main subject in the target image based on the distribution of the feature points that can be associated with the main subject in the target image.
The extraction means may calculate a motion vector between a plurality of images taken in time series, and extract a main subject region based on the calculated motion vector.

  According to the present invention, it is possible to specify an image in which the subject is out of frame due to the movement of the subject or the photographing apparatus, and to lower the evaluation of the image.

1 is a block diagram illustrating a configuration of an embodiment of a camera 100. FIG. FIG. 10 is a flowchart showing a flow of failure image identification processing in the form of the camera 100. It is the figure which showed typically the specific example of the defocus information acquisition area set to the imaging | photography screen in the form of the camera. It is the figure which showed typically the specific example of the feature point extracted from the reference | standard image in the form of the camera. It is the figure which showed typically the specific example of the difference calculation process result in 1st Embodiment. It is the 1st figure showing typically a specific example of a difference calculation processing result in a 2nd embodiment. It is the 2nd figure showing typically a specific example of a difference operation processing result in a 2nd embodiment. It is the figure which showed typically the example of extraction of the feature point in 3rd Embodiment. It is the figure which showed typically the example of calculation of the motion vector in 3rd Embodiment. It is the figure which showed typically the specific example of the difference calculation process result in the modification of 1st Embodiment. It is the figure which showed typically the specific example in 2nd Embodiment.

-First embodiment-
FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the first embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, a monitor 106, and a shake detection sensor 107. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a zoom button, a cross key, an enter button, a play button, and a delete button.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The lenses constituting the lens 102 include a zoom lens for changing the zoom magnification, a focus adjustment lens (AF lens) for performing focus adjustment, and the like. The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and captures a subject image formed by the lens 102. Then, an image signal obtained by imaging is output to the control device 104.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  In the present embodiment, the control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105. Note that the header information is, for example, Exif information, and information on the date and time of image capture, information on functions used at the time of image capture, and the like are recorded.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is inserted into the memory card slot 105 based on an instruction from the control device 104. Recorded on the memory card. Also, based on an instruction from the control device 104, an image file stored in the memory card inserted in the memory card slot 105 is read.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, when the user sets the mode of the camera 100 to the shooting mode, the control device 104 outputs image data for display of images acquired from the image sensor 103 in time series to the monitor 106. As a result, a through image is displayed on the monitor 106. The camera 100 further includes a shake detection sensor 107 that detects the shake of the camera 100. The output of the shake detection sensor 107 is input to the control device 104 and used for various controls.

  The camera 100 in the present embodiment can shoot a plurality of images in time series, and can perform, for example, moving image shooting, continuous shooting, or interval shooting. As described above, when a plurality of images are taken in time series, the subject may protrude from the frame as the subject moves. For example, when the photographer performs shooting while changing the shooting direction of the camera in accordance with the movement of the subject, if the movement of the camera relative to the movement of the subject is not appropriate, part or all of the subject to be originally photographed However, there is a possibility that an image protruding from the frame is taken. In addition, when the camera movement is not appropriate for the movement of the subject, the so-called unfavorable composition in which the subject to be photographed does not protrude from the frame but the subject to be photographed is extremely close to the end of the frame. May be taken.

  In the present embodiment, the control device 104 identifies an image in which a part or all of the subject protrudes from the frame, that is, an image in which the subject is out of the frame, and evaluates the identified image as an evaluation of other images. Relatively lower. For example, an image in which the subject is out of frame is evaluated as an image that has failed to be captured (failed image). Then, such a failed image is not recorded on the memory card. Hereinafter, the failure image specifying process according to the present embodiment will be described with reference to the flowchart shown in FIG. Note that the processing shown in FIG. 2 is executed by the control device 104 as a program that starts when an instruction to start moving image shooting, continuous shooting, or interval shooting is given based on an instruction from the user. First, assuming a case where shooting is performed with the camera 100 fixed on a tripod or the like, the flowchart shown in FIG. 2 will be described. It can be detected by monitoring the output of the shake detection sensor 107 by the control device 104 that the camera 100 is shooting in such a fixed state. That is, when the output of the shake detection sensor 107 during shooting is equal to or smaller than a predetermined value, the control device 104 determines that the camera 100 is fixed and shot.

  In step S10, the control device 104 specifies a main subject area for determining whether or not the frame has been out. In the present embodiment, the photographer specifies the in-focus area in the image as the main subject area in order to perform focus adjustment so as to focus on the main subject of interest. Specifically, for example, the control device 104 uses, as a reference image, an image captured first among a plurality of images input in time series, and a focused area in the reference image as a main subject area. Identify. For example, the control device 104 acquires a plurality of defocus information in synchronization with shooting from a plurality of phase difference AF defocus information acquisition areas set in association with the shooting screen, and the defocus amount is predetermined. The area less than the value is determined as the focus area, and the coarse area of the main subject is specified. Here, the rough region is expressed for the following reason. The phase difference AF used in the present embodiment is assumed to be a phase difference AF sensor that is roughly distributed (arranged) in the central portion relative to the shooting screen used in a normal single-lens reflex camera. ing. For this reason, it is assumed that the detection of the main subject at the screen edge and the dense detection cannot be performed only by the output of the phase difference AF sensor.

  In the example shown in FIG. 3, 31 defocus information acquisition areas indicated by × are set in the shooting screen, and the defocus information acquisition area indicated by ◯ has a defocus amount less than a predetermined value. The defocus information acquisition area represented by x indicates that the defocus amount is a predetermined value or more. Therefore, in FIG. 3, the area 3a is extracted as the in-focus area, that is, the main subject area.

  Thereafter, the process proceeds to step S20, and the control device 104 extracts feature points from each of the images input in time series. As the feature points to be extracted here, for example, feature points extracted using a known SIFT feature quantity (Scale Invariant Feature Transform) may be used, and detailed description of the feature point extraction method is omitted. FIG. 4 is a diagram illustrating an example of feature points extracted from the above-described reference image. In FIG. 4, the extracted feature points are indicated by Δ. Each extracted feature point has an image feature amount calculated for each feature point, and by comparing the feature amount between images, the feature points can be associated with feature points in images taken at different times. Possible configuration.

  Thereafter, the process proceeds to step S30, where the control device 104 extracts the feature points extracted in the area outside the main subject area of the reference image among the feature points extracted in step S20, and an image taken at a time different from the reference image. The reference image and an image photographed at a time different from the reference image are positioned with reference to a plurality of feature points in the background region other than the main subject region. Align. Thereafter, the process proceeds to step S40.

  In step S40, the control device 104 performs difference calculation processing (subtraction processing) between the reference image t0 and each of the images t1 to t3 taken at different times as shown in FIG. Calculate the absolute value. Note that the images t0 to t3 may have different brightness due to automatic exposure control. For this reason, it is preferable that the control device 104 performs the difference calculation process after performing normalization so that the minimum value and the maximum value of each pixel value in each image become a predetermined value. In addition, the normalization may be performed so that the RGB values of the associated feature points are aligned between a plurality of frames for which the difference calculation process is performed.

  In the example illustrated in FIG. 5, a difference image | t0−t1 |, a difference image | t0−t2 |, and a difference image | t0−t3 | In these difference calculation processes, when the difference calculation targets are background areas other than the main subject area described above, the difference calculation result is a value close to zero. Even when the difference calculation target is the main subject area, if the main subject area is uniform, the difference calculation result becomes a value close to zero. On the other hand, when one of the difference calculation targets is the main subject area and the other is the background area, the difference calculation result is a large value.

  In each of the difference images, the control device 104 extracts an area where the difference calculation result is larger than a predetermined threshold as a difference area, and determines whether or not the main subject is out of frame based on the difference area. In the example illustrated in FIG. 5, the difference area A1 and the difference area B1 are extracted from the difference image | t0-t1 |, and the difference area A2 and the difference area B2 are extracted from the difference image | t0-t2 |. In the difference image | t0−t3 | extracted, the difference area A3 and the difference area B3 are extracted.

  Thereafter, the process proceeds to step S50, and the control device 104 determines whether or not a part or all of the main subject has been out of frame by determining whether or not the difference area extracted in step S40 has reached the end of the shooting screen. Determine. For example, in the example shown in FIG. 5, in the difference image | t0-t1 | and the difference image | t0-t2 |, no difference area reaches the end of the shooting screen, but the difference image | t0-t3 | Since the difference area B3 has reached the end of the shooting screen at, it is determined that the main subject is out of frame in the image t3. Thereafter, the process ends.

  In the example shown in FIG. 5, when the difference image | t0-t1 | and the difference image | t0-t2 | are compared, the region B1 has moved to the region B2, and thus the main subject has moved to the right in the shooting screen. You can see that In addition, when the difference image | t0−t2 | and the difference image | t0−t3 | are compared, the region B2 has moved to the region B3, and the area of the region B3 is smaller than the area of the region B2. It can be seen that the subject has moved out of the frame as a result of moving further to the right in the shooting screen. As described above, it is also possible to detect a change in the position of the main subject area in the image between the images based on the change in the position of the difference area between the difference images.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The control device 104 extracts a main subject area including a main subject from a plurality of images acquired in time series, determines whether or not the main subject has been out of frame, and an image in which the main subject is out of frame Was evaluated as a failure image. As a result, an image in which the main subject is out of frame due to the movement of the subject or the camera can be specified as a failed image.

(2) The control device 104 determines that the main subject is out of frame when the difference area in the difference image reaches the end of the shooting screen. This makes it possible to determine with high accuracy whether or not the main subject is out of frame.

(3) The control device 104 uses an image acquired at a certain time as a reference image, and is acquired at a time different from the acquisition time of the reference image and the reference image (for example, after the above “certain time”). By calculating the difference from the image, the position change of the main subject region in the image between the plurality of images acquired in time series is detected. This makes it possible to detect a change in the position of the main subject region in the image with high accuracy.

(4) The control device 104 uses the feature points in the reference image and the feature points in the other images to perform alignment between the reference image and each image, and performs difference calculation processing after the alignment. . Thereby, the difference between corresponding pixels in the reference image and each image can be calculated.

(5) The control device 104 performs the difference calculation process after normalizing the brightness of the reference image and the brightness of other images. Thereby, even when the brightness of each image is different, the difference calculation can be performed with high accuracy.

-Second embodiment-
In the first embodiment described above, the example in which the process illustrated in FIG. 2 is executed on the assumption that the camera 100 is fixed with a tripod or the like to perform shooting has been described. On the other hand, in the second embodiment, a process when the user moves the camera 100 in accordance with the movement of the subject or when the camera 100 moves against the user's intention due to a hand shake will be described. In the second embodiment, the drawings in FIGS. 1 to 4 are the same as those in the first embodiment, and thus the description thereof is omitted.

  First, processing when the subject moves in the opposite direction on the shooting screen with respect to the shake direction of the camera 100 will be described. Whether or not the subject is moving in the direction opposite to the shake direction of the camera 100 is determined based on the shake direction of the camera 100 detected using the output of the shake detection sensor 107 and the motion vector of the main subject obtained from the captured image. It can be determined by comparing the direction of. For example, FIG. 6 shows an example in which the main subject indicated by the broken line moves to the right on the shooting screen, and the shooting range of the camera 100 moves to the left. In FIG. 6 and FIG. 7 to be described later, in order to facilitate understanding, images of two frames taken at different times, that is, a reference image and a target image to be subjected to a difference calculation are displayed in a vertical direction. Shown in a staggered manner. In this case, the control device 104 extracts the main subject region for the reference image and other images by the method described above in the first embodiment, and extracts a plurality of pieces extracted from the background region other than the main subject region. The feature points are associated and aligned. In the example shown in FIG. 6, the x mark indicates a plurality of feature points extracted from the background area in each image.

  Then, for example, as illustrated in FIG. 6, the control device 104 calculates a difference image | t0−t4 | by performing a difference operation between the reference image t0 and the image t4 acquired in time series. In this case, the area where the difference calculation between the images t0 and t4 can be performed is limited to the area overlapping between the two images after the alignment by the feature points. For example, the area 6a indicated by hatching in FIG. Cannot perform a difference operation. Here, in the processing in the case where the subject moves in the direction opposite to the shake direction of the camera 100 and the shooting screen as shown in FIG. 6, the right end region of the image t4 in which a part of the main subject is out of frame is the image t0. It is included in the overlap area | region which can perform difference calculation of this, and a difference image can be calculated. Therefore, when the area of the difference area B4 in the difference image | t0−t4 | is smaller than the difference area in the image t3 that is one frame before the image t4, for example, an image in which no frame-out has occurred. The control device 104 can determine that part or all of the main subject is out of frame in the image t4.

  Next, processing when the main subject moves in the same direction on the shooting screen as the camera shake direction will be described. Whether or not the subject is moving in the same direction as the shake direction of the camera 100 is determined based on the shake direction of the camera 100 detected using the output of the shake detection sensor 107 and the motion vector of the main subject obtained from the captured image. It can be determined by comparing the direction of. For example, FIG. 7 shows an example in which the main subject moves to the right on the shooting screen and the shooting range also moves to the right. In this case, the control device 104 extracts the main subject region for the reference image and other images by the method described above in the first embodiment, and extracts a plurality of pieces extracted from the background region other than the main subject region. The feature points are associated and aligned. In the example shown in FIG. 7, the x mark indicates a plurality of feature points extracted from the background area in each image.

  Then, for example, as illustrated in FIG. 7, the control device 104 calculates a difference image | t0−t4 | by performing a difference operation between the reference image t0 and the image t4 acquired in time series. Also in this case, the area where the difference calculation between the image t0 and the image t4 can be performed is limited to the area overlapping between the two images after the alignment by the feature points. For example, the area 7a indicated by hatching in FIG. Cannot perform a difference operation. However, especially when the amount of motion of the background region, that is, the amount of shake of the camera 100, calculated from the motion vector is small compared to the size of the difference region B4, the frame-out determination is performed only with the overlapping region of the two images. Even if it is done, it is considered that there is little possibility of erroneous judgment results. Therefore, when the difference area B4 in FIG. 7 is in contact with the end of the overlapping area and the area is smaller than the difference area in the image t3 one frame before, the control device 104 performs the main operation in the image t4. It is determined that part or all of the subject is out of frame. Furthermore, when the distance between the front end of the main subject in the moving direction and the screen end corresponding to the moving direction is short as described above (for example, smaller than a predetermined threshold), the composition is not preferable.

  FIGS. 11A and 11B show the right ends of the imaged images t0 and t4 after the alignment by the feature points shifted in the vertical direction, and two examples in which the main subject indicated by a black circle exists only in the image t4 (A ) (B). 11A and 11B, the main subject has reached the right edge of the shooting screen in the image t4, FIG. 11B shows the state where the main subject is out of frame, and FIG. 11A shows the case where the main subject is out of frame. It is the figure which showed the mode which is not doing. In this case, the difference calculation cannot be performed because there is no difference calculation target area of the image t0 corresponding to the area P where the main subject exists in the image t4. However, in this case, as shown in FIG. 11A, even if the main subject is not out of frame in the image t4, the difference calculation is performed only in the overlapping region, so that the above unfavorable composition is obtained. An image (an image having a composition such as (A) in which the screen edge is close to the moving direction of the main subject) can be extracted as a shooting failure image.

  According to the second embodiment described above, the following operational effects can be obtained. That is, the control device 104 determines that the main subject has been out of frame when the difference area in the difference image reaches the end of the shooting screen and the area of the difference area is reduced compared to the previous frame. I made it. As a result, even when the subject or the camera moves, it can be determined whether or not the main subject is out of frame, and an image with an unfavorable composition can be extracted.

-Third embodiment-
In the third embodiment, a method for determining whether or not the main subject is out of frame based on the distribution of feature points included in the main subject region will be described. Note that in the thirty-second embodiment, the drawings in FIGS. 1 to 4 are the same as those in the first embodiment, and a description thereof will be omitted.

  As in the first and second embodiments described above, the control device 104 identifies a main subject area from each image captured in time series, and extracts a plurality of feature points in the main subject area. Then, feature points are associated between images taken at different times, and it is determined whether or not the main subject is out of frame based on the distribution situation of feature points that cannot be matched in the main subject region.

  For example, as shown in FIG. 8A, the control device 104 extracts feature points indicated by x from within a main subject region 8a of a certain image, for example, the reference image t0. Similarly, the control device 104 extracts feature points from the main subject region for images taken at different times, for example, the image t4. Then, the control device 104 associates feature points in the main subject region between two different images, and extracts feature points that cannot be matched from them. For example, FIG. 8B shows a result of associating feature points in the main subject region with the reference image t0 and the image t4 as targets. In FIG. 8B, feature points that can be matched are indicated by x, and feature points that cannot be matched are indicated by ◯.

  When the extracted feature points that cannot be dealt with are distributed at positions close to the end points of the main subject region, the control device 104 determines that the main subject is out of frame in the image t4. For example, in the example shown in FIG. 8B, since the feature points 8b and 8c that cannot be matched are distributed at the right end of the main subject area, the control device 104 out-of-frames the main subject in the image t4. Is determined. If the feature points 8b and 8c that cannot be dealt with in this way are distributed at the right end of the main subject region, it can be determined that the main subject has moved out of the frame as a result of moving in the right direction in the shooting screen. .

  Further, in the example shown in FIG. 8C, the feature points 8e and 8f that cannot be matched are distributed at the lower right end of the main subject area, and therefore the control device 104 causes the main subject to be out of frame in the image t4. It is determined that When the feature points 8e and 8f that cannot be dealt with in this way are distributed at the lower right corner of the main subject area, it is assumed that the main subject has moved out of the frame as a result of moving in the lower right direction in the shooting screen. I can judge.

  On the other hand, as shown in FIG. 8D, when the feature points 8g that cannot be matched are distributed near the center of the main subject area, the control device 104 causes the main subject to be framed out in the image t4. Judge that it is not.

  According to the third embodiment described above, the following operational effects can be obtained. In other words, the control device 104, when a feature point set in the main subject region extracted from the reference image cannot be associated with a feature point in another image acquired in time series occurs. When the feature points that cannot be matched are distributed at positions close to the end points of the main subject area, it is determined that the main subject is out of frame. Thereby, it is possible to determine the frame-out with high accuracy according to the correspondence situation of the feature points between the images. In this determination, it is determined whether or not the area of the main subject is within a predetermined distance from the edge of the screen. Alternatively, an image with an unfavorable composition can be extracted with higher accuracy.

  Furthermore, according to this configuration, it is possible to detect an image in which the main subject is occluded behind another subject and extract it as a shooting failure image. At this time, the control device 104 determines whether or not the area of the main subject is within a predetermined distance from the edge of the screen. If it is not within the predetermined distance, for example, as shown in FIGS. In addition, when feature points that cannot be matched are distributed at positions close to the boundary between the main subject region and other regions, it may be determined that occlusion has occurred. In addition, using a plurality of feature points in the main subject area, after positioning between the reference image and an image taken at a different time, a difference image is calculated between the main subject areas, The control device 104 may determine that a frame-out has occurred when the area of the difference image is equal to or greater than a predetermined value.

-Fourth embodiment-
In the fourth embodiment, the control device 104 extracts feature points in each frame of the time-series image in the same manner as in the first to third embodiments, and then associates the feature points in each frame. A method for calculating a motion vector between frames and determining whether or not the main subject has gone out of the frame based on the motion vector calculation result will be described. In particular, the above-mentioned SIFT feature value is robust against rotation, change in size, change in brightness, etc., and the feature point associated with this feature value is suitable for calculating a motion vector. Yes. 1 to 4 are the same as those in the first embodiment, and the description thereof is omitted.

  The control device 104 clusters the start point (or end point) of the motion vector in each frame based on the direction and magnitude of the motion vector calculated as described above, and the adjacency of the position in the screen, so that the main subject region To extract. For example, assume that a motion vector as shown in FIG. 9 is obtained. The control device 104 clusters the motion vectors based on their directions and sizes, and the start points (or end points) of motion vectors having similar motion vector directions and sizes exist continuously (adjacent). The main subject areas 9a to 9c in each image are extracted based on the continuity (adjacent) evaluation result of the in-screen position of the motion vector start point (or end point). FIG. 9A shows the extraction result of the main subject region 9a in the reference image t0 at time t0, and FIG. 9B shows the extraction result of the main subject region 9b in the image t1 at time t1. FIG. 9C shows the extraction result of the main subject region 9c in the image t2 at time t2.

  When the main subject region extracted in this way is in contact with the edge of the screen, or when the area of the main subject region is reduced compared to the previous frame, the control device 104 determines that the main subject region has been out of the frame. judge. For example, in the example shown in FIG. 9, the main subject area 9b is in contact with the screen edge in FIG. 9B, and the area of the main subject area is reduced in FIG. It is determined that the subject is out of frame.

  According to the fourth embodiment described above, the following operational effects can be obtained. That is, the control device 104 extracts the main subject region based on the motion vector calculated between the images and performs the frame-out determination. As a result, the frame-out determination can be performed with a simple configuration.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the first embodiment described above, the control device 104 has been described with respect to an example in which the main subject region is extracted based on the defocus information. A subject area may be extracted. For example, the control device 104 may extract a region having a similar color and texture from each defocus information acquisition area, and specify and extract the main subject region by combining the extracted regions. The main subject area may be based on a recognition result of a specific subject such as face recognition.

(2) In the first to fourth embodiments described above, the example has been described in which the first captured image among the plurality of images input in time series is used as the reference image. However, the reference image may be determined by other methods. For example, while the release button is half-pressed by the user, the acquired image is recorded in the buffer memory, and at the time when the release button is fully pressed, the images taken before and after the full-press are recorded on the memory card. In this case, the image acquired at the timing when the button is fully pressed may be used as the reference image.

  Further, the reference image does not need to be fixed in a series of photographing, and may be changed sequentially with time. For example, the control device 104 may change the reference image every predetermined number of frames. Alternatively, an image at the time when the main subject is framed may be used as the reference image. For example, in the case of fixed focus position shooting in which the focus position is fixed in advance, the defocus amount is detected at the center of the screen in each shooting frame, and a frame in which a defocus amount equal to or less than a predetermined amount is detected. A reference image may be used.

(3) In the above-described first embodiment, the example in which the process illustrated in FIG. 2 is executed assuming that the camera 100 is fixed with a tripod or the like has been described. Since there is a low possibility that positional deviation will occur, the processing in steps S10 to S30 in FIG. 2 may be omitted.

(4) In the first embodiment described above, the difference calculation process is performed between the reference image t0 and the images t1 to t3 taken at a time different from the time when the reference image t0 was taken. The method for calculating the value has been described. However, instead of this method, the difference calculation process may be performed between the closest frames in time, and the absolute value thereof may be calculated. According to such a method, a difference image | t0-t1 |, a difference image | t1-t2 |, and a difference image | t2-t3 | as shown in FIG. It will be. In the example shown in FIG. 10, since the area of the difference area B3 ′ in the difference image | t2-t3 | is decreased from the difference area B2 ′ in the difference image | t1-t2 | It can be determined that it is out. In the above-described embodiment, as an example of the acquisition (photographing) timing of the images t1 to t3, it is described after the time when the reference image t0 is photographed, but before the time when the reference image t0 is photographed. It doesn't matter.

(5) In the above-described second embodiment, it has been described that the difference calculation cannot be performed on the region 7a indicated by the oblique lines in FIG. However, in the case where an area with similar texture or an area with few edges occupies most of the background, such as when shooting a flying airplane with the background as the sky, the reference image t0. Is copied by the amount corresponding to the area Q and pasted on the image end of the reference image t0 (so-called image data in the right end of the screen of the image t0 in the example of FIG. 7). The difference calculation may be performed after the image end of the reference image t0 is extended to the right end of the screen of the image t4. Further, a region having the same area as the region Q is extracted from the right end of the reference image t0, and image data corresponding to the extracted region is added to the right end of the reference image t0, and then a difference calculation with the image t4 is performed. May be.

  In addition, for the region Q where the difference calculation cannot be performed, the difference calculation may be performed using image data of a frame different from the target frame on which the difference calculation is performed. For example, a frame other than the reference image t0 and the image t4 for which the difference calculation is performed, for example, a frame from t1 to t3, or any frame after the image t5 is used, and each of the reference image t0 and any one of these frames is characterized. A background region to be added to the right end of the reference image t0 may be generated by extracting points and aligning two frames using corresponding feature points. As a result, the difference calculation can be performed even when an area where the difference calculation cannot be performed occurs.

(6) In particular, in the processing when the subject moves in the same direction as the camera shake direction in the second embodiment described above, the time series image acquisition time interval is long or the camera moving speed is fast. In this case, there is a possibility that the overlapping area of the two images for which the difference calculation is performed becomes small. In such a case, it is preferable to increase the update frequency of the reference image t0. For example, the update time interval of the time series image, the setting information of the imaging time interval of the camera, the moving speed (motion information) of the camera, the output of the shake detection sensor provided in the camera, the motion vector information obtained from the image data, etc. The reference image t0 may be updated at a determined update frequency using at least one of the above. Thereby, even when the time series image acquisition time interval is long or when the moving speed of the camera is fast, it is possible to prevent the overlapping area of the two images to be subjected to the difference calculation from becoming small. The above processing is performed not only when the subject in the second embodiment moves in the same direction as the camera shake direction, but also when the subject in the second embodiment moves in the direction opposite to the camera shake direction. Or you may apply when the camera in 1st Embodiment is being fixed.

  Further, when the subject moves in the same direction as the camera shake direction and the time series image acquisition time interval is long, or the camera moving speed is fast, the third or fourth implementation described above. The frame-out determination is performed using the method of the form, and if not, the frame-out determination is performed using the method described in the first embodiment or the second embodiment described above. You may make it do. In this way, the processing for determining frame-out may be changed according to the time-series image acquisition time interval and the moving speed of the camera.

(7) The main subject region extracted in the first to fourth embodiments described above may be tracked between images photographed in time series using a technique such as template matching.

(8) In the first to third embodiments described above, an example in which feature points are associated between different images has been described. In this case, when there is a change in the zoom state at the time of acquiring each image, the size of the main subject, or when the camera 100 is rotating, the image is enlarged or reduced based on a plurality of feature point coordinates. Alternatively, it is desirable to perform the affine transformation to associate the positional relationship between the images of the two frames.

(9) In the first to fourth embodiments described above, an example in which an image in which a subject is out of frame is specified as a shooting failure image and such a failure image is not recorded on the memory card has been described. On the other hand, the failure image may be recorded on the memory card, and information indicating the failure image may be recorded in the header information. Further, for example, when a representative image is selected from a plurality of images taken in time series, the failed image may not be selected as the representative image.

(10) In the above-described embodiment, an example of determining whether the main subject has been out of frame has been described. However, the above-described processing can also be used for determination when extracting an image with poor composition. An image with a short distance between the main subject and the photographing screen edge in the main subject moving direction is determined to be undesirable from the viewpoint of composition. In order to make such a determination, an area narrower than the shooting screen (for example, the overlapping area shown in FIGS. 6 and 7 or an area narrower than the overlapping area) is set, and the frame-out determination process described above is performed. What is necessary is just to set it as the structure which performs the same process.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

DESCRIPTION OF SYMBOLS 100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control apparatus, 105 Memory card slot, 106 Monitor, 107 Shake detection sensor

Claims (11)

Extraction means for extracting a main subject area including a main subject from an image taken in time series;
Determining means for determining whether or not at least a part of the main subject region extracted by the extracting means is out of frame from the image;
Specifying means for specifying the image determined to be out-of-frame by the determining means as a frame-out image;
An image evaluation apparatus comprising: an evaluation unit that relatively lowers the evaluation of the frame-out image specified by the specifying unit in the image acquired in time series. The image evaluation apparatus according to claim 1,
The determination unit detects a change in position of the main subject area in the images between a plurality of images taken in time series, the main subject area is located at an end point of the image, and the main subject area An image evaluation apparatus, wherein when the area decreases, it is determined that at least a part of the main subject region is out of frame from the image. The image evaluation apparatus according to claim 2,
The determination means uses an image captured at a certain time as a reference image, and calculates a difference between the reference image and a target image captured at a different time from the reference image, thereby obtaining a plurality of images captured in time series. An image evaluation apparatus for detecting a change in position of the main subject region in an image between the images. The image evaluation apparatus according to claim 3.
The determination means uses the feature points in the reference image and the feature points outside the target image to perform alignment between the reference image and the target image, and calculates the difference after alignment. An image evaluation apparatus. The image evaluation apparatus according to claim 4,
The determination means is acquired at a time different from that of the target image when a corresponding area for calculating a difference from the reference image does not exist in the target image when the alignment is performed. An image evaluation apparatus, wherein the corresponding area is acquired from an image and the difference is calculated. In the image evaluation apparatus according to any one of claims 3 to 5,
The determination unit calculates the difference after normalizing the brightness of the reference image and the brightness of the target image. In the image evaluation apparatus according to any one of claims 3 to 6,
Reference image update means for re-acquiring and updating the reference image at a predetermined update frequency,
The image evaluation apparatus, wherein the reference image update means determines the predetermined update frequency based on at least one of a shooting time interval of images taken in time series and movement information of the imaging apparatus. The image evaluation apparatus according to claim 1,
The extraction means uses an image taken at a certain time as a reference image, extracts the main subject region from the reference image,
The determination unit sets a feature point in the main subject region extracted from the reference image, and corresponds to a feature point in a target image to be photographed later in the feature point set in the reference image An image evaluation apparatus characterized by determining that at least a part of the main subject area in the target image is out of frame from the image when there is something that cannot be attached. The image evaluation apparatus according to claim 8,
The determination means corresponds to the distribution of feature points that cannot be associated with the feature points in the target image among the feature points set in the main subject area or the feature points in the target image. An image evaluation apparatus that determines that at least a part of the main subject area in the target image is out of frame from the image based on a distribution of feature points that can be attached within the main subject. The image evaluation apparatus according to claim 9, wherein
The determination unit further distributes the feature points in the main subject that cannot be associated with the feature points in the target image when the position of the main subject region in the target image satisfies a predetermined condition, or An image evaluation apparatus characterized by determining that occlusion has occurred in a main subject in the target image based on a distribution in the main subject of feature points that can be associated with a feature point in the target image. The image evaluation apparatus according to claim 1,
The image evaluation apparatus characterized in that the extraction means calculates a motion vector between a plurality of images taken in time series, and extracts the main subject region based on the calculated motion vector.

Images