JP2017016592A - Main subject detection device, main subject detection method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a main subject detection device capable of detecting an intended main subject from images.SOLUTION: In an embodiment, an area of an object which frequently appears is detected as a first candidate area from plurality of images prior to specific point of time. A feature amount of the images at a specific point of time is calculated based on a piece of information on a function which uses a detection result of the main subject, and based on the calculated feature amount, a second candidate area is detected. The main subject area is finally detected based on the detected first candidate area and the second candidate area.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for detecting a region of a main subject in an input image.

  Conventionally, a method for detecting a region of a main subject in an input image is known. Patent Document 1 discloses a technique for detecting a stably appearing subject as a main subject such as a long accumulated time existing in images of a plurality of past frames.

  Such main subject area detection processing is performed in an imaging device such as a digital camera, so that autofocus (AF) can be adjusted for the main subject or automatic tracking can be performed for the main subject. become able to.

JP2013-120949A JP 2011-53759 A

L. Itti, C.I. Koch, E .; Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, Inc. 20, no. 11, pp. 1254-1259, Nov 1998. B. K. P. Horn and B.H. G. Schunk, “Determining Optical Flow, Artificial Intelligence”, vol. 17, pp. 185-203, 1981.

  However, as in the technique disclosed in Patent Document 1, it is possible to detect the main subject by mistake of the photographer's intention only by using the main subject as a long and stable image over a plurality of frames. There is sex.

  For example, when a moving object has entered the image (for example, a car has entered) while shooting a non-moving object (for example, a road sign), the conventional patent document 1 uses a road. The sign is detected as the main subject. However, the photographer may have waited for the car to come into the angle of view, trying to photograph the car in a composition in which the road sign is reflected. In this case, it is preferable that the car is detected as the main subject, and the main subject that is contrary to the photographer's intention is detected only by detecting the main subject as a long and stable image. There was a thing. An object of the present invention is to suppress detection of a main subject against the photographer's intention.

  In order to solve the above problems, the present invention is a main subject detection device that detects a region of a main subject of an image at a specific time point, and acquires a plurality of continuous images in time series before the specific time point. On the basis of the information on the object detected by the object detecting means, the object detecting means for detecting an object having a predetermined frequency or more in the plurality of images acquired by the acquiring means, First detection means for detecting a first candidate region of the main subject in the image at the specific time point, and when the main subject region is detected in the image at the specific time point, information on the detected main subject region is obtained. A function acquisition unit that acquires a function to be provided; and a feature amount calculation unit that calculates a feature amount of the image at the specific time according to the function acquired by the function acquisition unit. , Second detection means for detecting a second candidate area of the main subject in the image at the specific time point based on the feature quantity calculated by the feature quantity calculation means, and a first candidate detected by the first detection means And final detection means for detecting a main subject area of the image at the specific time point based on the area and the second candidate area detected by the second detection means.

  According to the above configuration, the present invention can suppress the detection of the main subject against the photographer's intention.

1 is a schematic block diagram illustrating a configuration of a main subject detection device according to a first embodiment. The figure which shows the example of an input image and a remarkable area | region in 1st Embodiment. The figure explaining the detail of the process of the 1st detection part in 1st Embodiment. The figure explaining the detail of the process of the 2nd detection part in 1st Embodiment. 5 is a flowchart illustrating a procedure of main subject detection processing in the first embodiment. FIG. 10 is a diagram for explaining an outline of a main subject detection method according to a second embodiment. 9 is a flowchart showing a procedure of main subject detection processing in a modification of the first embodiment. 9 is a flowchart illustrating a procedure of main subject detection processing in the second embodiment. FIG. 10 is a schematic block diagram illustrating a configuration of a main subject detection device according to a third embodiment. 10 is a flowchart illustrating a procedure of main subject detection processing in the third embodiment.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a schematic block diagram illustrating a configuration of a main subject detection device 100 according to the present embodiment. The main subject detection apparatus 100 includes an image acquisition unit 101, a target object detection unit 102, a first detection unit 103, a function acquisition unit 104, a feature amount calculation unit 105, a second detection unit 106, a final detection unit 107, and a saliency area detection unit. 108, an identical object region specifying unit 109, and an appearance frequency calculating unit 110. The main subject detection apparatus 100 according to the present embodiment is realized using a semiconductor integrated circuit (LSI). For example, a semiconductor integrated circuit provided in an imaging device such as a digital camera may realize the function as the above-described main subject detection device. In this case, the imaging device itself is the main subject detection device of this embodiment. It corresponds to 100.

  The main subject detection device 100 may be configured as a single device. That is, the main subject detection apparatus 100 includes a hardware configuration such as a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). Then, when the CPU executes a program stored in a ROM, an HDD, or the like, for example, each functional configuration and flowchart processing described later are realized. The RAM has a storage area that functions as a work area where the CPU develops and executes the program. The ROM has a storage area for storing programs executed by the CPU. The HDD has a storage area for storing various types of data including various programs necessary for the CPU to execute processing, data on threshold values, and the like.

  Hereinafter, in the description of the present embodiment, a case where the main subject detection device 100 is configured as a digital camera will be described as an example. First, the image acquisition unit 101 acquires input images (moving image frames) sequentially input from an imaging unit (such as an image sensor), and uses the acquired input images (moving image frames) as a target object detection unit 102 and a first detection unit 103. And output to the feature amount calculation unit 105. Note that such sequentially input images are also used for a through display function of a digital camera.

  The target object detection unit 102 detects an object that exists frequently in a plurality of frames (hereinafter referred to as a capture target object). An outline of the processing of the target object detection unit 102 will be briefly described. First, the target object detection unit 102 searches for candidates for capture target objects, and determines whether or not the candidates are reflected for a plurality of frames. At that time, the target object detection unit 102 determines whether or not the candidates searched for in different frames are the same, and if there are candidates determined to be the same over a plurality of frames, the target object detection unit 102 appears to be long. to decide. Then, the target object detection unit 102 outputs information (position, size, color, etc.) related to the capture target object that is determined to be reflected for a long time over a plurality of frames.

  Next, in order to explain the details of the target object detection unit 102, processing for each sub-block constituting the target object detection unit 102 will be described. The saliency detection unit 108 sequentially acquires each moving image frame as an input image and searches for a saliency area from each input image. Various known methods can be used as the salient region searching method. For example, a center-surround difference method or the like described in Non-Patent Document 1 may be used. A small area is set for each pixel of the image, and a feature amount between the small area and the surrounding area is set. A saliency area is detected, for example, by calculating the difference as the saliency.

  FIG. 2 shows an example of the saliency area searched by the input image and the saliency area detection unit 108. FIG. 2A shows an input image 201 acquired by the image acquisition unit 101 and input to the saliency area detection unit 108. In the input image 201, a sign 202 and a car 203 are shown.

  The saliency detection unit 108 calculates the saliency with respect to the input image 201 using a technique as described in Non-Patent Document 1 to obtain a saliency map having the saliency as a pixel value. This saliency map is binarized with a predetermined threshold value, and only a high saliency area having a saliency greater than or equal to the predetermined threshold value is extracted.

  FIG. 2B displays the circumscribed rectangle of the detected high saliency area superimposed on the input image 201. The rectangles 205 and 206 are circumscribed rectangle areas of the high saliency area set for the sign 202 and the car 203. The saliency area detection unit 108 takes the detected highly saliency area as a candidate for a target object and outputs the information to the same object area specification unit 109. The information to be output here is, for example, the image clipped by the circumscribed rectangles 205 and 206 and the position of the circumscribed rectangles 205 and 206 in the input image 201.

  The same object region specifying unit 109 determines whether the candidates for the capture target object detected in different frames (input images) are those obtained by photographing the same object. Whether or not they are the same object is determined by calculating a correlation between candidate areas output from the saliency area detection unit 108 for different frames.

  Returning to FIG. 2, the same object region determination process in the same object region specifying unit 109 will be described. In FIG. 2C, an input image 207 following the input image 201 (continuous in time) is shown. In the input image 207, the sign 208 and the car 209 correspond to the sign 202 and the car 203 in FIG. 2A, and the car 209 has moved slightly from the time of the input image 201. In addition, the input image 207 includes an airplane 210 that has newly entered the frame.

  FIG. 2D shows the circumscribed rectangle of the detected high saliency area superimposed on the input image 207. The rectangles 212, 213, and 214 are circumscribed rectangular areas of the high saliency area set for the sign 208, the car 209, and the airplane 210.

  As described above, the input image 201 at a certain point in time has two rectangles 205 and 206 as acquisition target object candidate regions, and the input image 207 at a different point in time has rectangles 212 and 213 as acquisition target object candidate regions. , 214 exists. Then, the same object region specifying unit 109 correlates candidate regions of frames (input images) at different points in time. In the case of FIG. 2 described above, there are six combinations of “205 and 212”, “205 and 213”, “205 and 214”, “206 and 212”, “206 and 213”, and “206 and 214”. Correlation coefficients are calculated for it. In the method of calculating the correlation coefficient, various known techniques can be used. For example, a normalized correlation may be calculated. When calculating a normalized correlation, it is necessary to align the sizes of candidate areas to be correlated by affine transformation or the like.

  In this way, the same object region specifying unit 109 correlates candidate regions of different frames (input images), and performs threshold processing with a predetermined threshold on the obtained correlation coefficient. Then, candidate regions having correlation coefficients exceeding the threshold are determined as the same object. In the example of FIG. 2, “205 and 212” and “206 and 213” are determined as the same object region. A determination result indicating which candidate regions are determined to be the same object region is output to the appearance frequency calculation unit 110.

  Based on the determination result output from the same object region specifying unit 109, the appearance frequency calculation unit 110 counts how many times each candidate for the capture target object appears in the video frame. In the example of two frames shown in FIG. 2, the appearance frequency of the candidate 205 (212) is twice, the appearance frequency of the candidate 206 (213) is two times, and the appearance frequency of the candidate 214 is one time. Since the candidate (pair) determined to be the same is output from the same object region specifying unit 109, the appearance frequency for the same object can be counted over three frames or more by following this.

  When the appearance frequency calculation unit 110 appears as a target object to be captured when a candidate for the capture target object that has appeared a predetermined number of times or more appears in a predetermined number of frames, the appearance frequency calculation unit 110 sets the candidate as the target object. For example, in the case of FIG. 2, the candidates 206 (213) and 205 ( 212) is an object to be captured.

  As a result of the above processing, the appearance frequency calculation unit 110 outputs information indicating the capture target object (hereinafter referred to as capture target object information) to the first detection unit 103. For example, as the capture target object information of the output result, cut-out images of the capture target objects 212 and 213 and information regarding their positions in the input image 207 are output. As the capture target object information, any of the cropped images may be output as long as the capture target objects 212 and 205 are used, but in this embodiment, the capture target object 212 that is new in time is output. This is because the search for the first candidate region, which will be described later, can be performed with higher accuracy by cutting from a frame that is close in time to the image for which the main subject is to be detected.

  As described above, the target object detection unit 102 detects the capture target object. As a result, an object having a high appearance frequency in the moving image frame is specified as the capture target object. An object with a high appearance frequency in a video frame is likely to be an object that the photographer wants to capture in the captured image, so identifying such an object realizes main subject detection that reflects the photographer's intention It can be said that this is a necessary process.

  It should be noted that such processing for knowing what kind of object the photographer is going to shoot needs to be performed on a moving image frame before the frame at the time of actual main subject detection. is there. Therefore, the processing in the target object detection unit 102 is performed every time a still image constituting a moving image frame is input. In other words, even in the main subject detection apparatus 100 (digital camera) of the present embodiment, the target object detection unit 102 performs the background process every time an image is input regardless of whether or not recording is performed by pressing the shutter. The above process is executed.

  Returning to the description of FIG. 1, the first detection unit 103 uses the capture target object information acquired from the target object detection unit 102 to perform a main subject detection frame input from the image acquisition unit 101 (hereinafter, detection target). Search for where the object to be captured is located. In the present embodiment, the search method is not particularly limited. For example, the capture target object can be searched by template matching using a cut-out image of the capture target object output as capture target object information from the target object detection unit 102 as a template.

  FIG. 3 is a diagram for explaining the processing of the first detection unit 103 in the present embodiment. FIG. 3A shows a detection target frame 301. Template matching is performed on the detection target frame 301 using the capture target objects 212 and 213 as templates, thereby specifying where the capture target objects 212 and 213 exist in the detection target frame 301. FIG. 3B shows a result of processing for specifying an object to be captured by template matching. In the figure, rectangular areas 303 and 304 are specified as locations where the capture target objects 212 and 213 exist. The area identified and detected by the first detection unit 103 is hereinafter referred to as a first candidate area. In the present embodiment, the rectangular areas 303 and 304 are the first candidate areas.

  Alternatively, the capture target object may be detected by a process similar to that of the target object detection unit 102. In this case, the high saliency area is extracted by the same processing as the saliency area detection unit 108, and subsequently, the high saliency area obtained by extracting the correlation calculation as performed in the same object area specifying unit 109 and the cut-out image of the capture target object Between the two. Information about the area (first candidate area) identified and detected by the first detection unit 103 is output from the first detection unit 103 to the final detection unit 107.

  The function acquisition unit 104 acquires information related to functions that can be executed by the device (digital camera) specified by the user. The function here refers to an application that can be realized using the main subject detection result. For example, an autofocus (AF) function, an automatic tracking function, etc., as exemplified in the background art section. It is. The function acquisition unit 104 acquires information regarding which of these functions is to be applied to an image to be taken. For example, if the photographer wants to shoot the main subject using the auto-tracking function, and the camera (digital camera) has been set as such, the function acquisition unit 104 sets the setting. Acquired information (automatic tracking function) is acquired.

  In general, there is a hint for estimating a photographer's intention in terms of which function is selected as a function that uses a main subject detection result (hereinafter referred to as a detection result use function). In other words, the fact that the automatic tracking function has been selected can be inferred that the photographer wants to photograph an object having a motion as the main subject. If the AF function is selected, the photographer may want to photograph the object in the foreground in the image as the main subject.

  The function acquisition unit 104 acquires information about the function set for the shooting target image to be shot, and outputs information indicating the acquired function (usage function information) to the feature amount calculation unit 105. In the present embodiment, as described below, the second candidate area of the main subject is also detected based on what is selected as the detection result utilization function.

  The feature amount calculation unit 105 is set in advance with a feature amount (hereinafter referred to as a calculated feature amount) to be calculated for each detection result using function, and calculates the feature amount corresponding to the input used function information. This is performed on the detection target frame input from the image acquisition unit 101. Here, the type of the calculated feature amount is associated with a feature amount useful for the detection result utilization function. For example, if the detection result utilization function is an automatic tracking function, a feature quantity that detects the movement of an object is associated as a calculated feature quantity, and if it is an AF function, a feature quantity that can separate the foreground and the background Are associated as calculated feature amounts.

  As a feature quantity suitable for detecting the motion of an object, for example, a velocity vector in each pixel such as an optical flow disclosed in Non-Patent Document 2 can be cited. In the case of motion detection, since it is difficult to detect from only one detection target frame, it is necessary to input the frame preceding the detection target frame together to the feature amount calculation unit 105. Alternatively, when the camera is not moving, the feature quantity of the motion can be calculated from the inter-frame difference. Various known techniques can be used as the feature amount that can separate the foreground and the background. For example, a color feature amount calculated by the method disclosed in Patent Document 2 may be used.

  As described above, the feature amount calculation unit 105 calculates a type of feature amount determined for each detection result utilization function, and creates a feature amount map in which the calculated feature amount is a pixel value of each pixel. Then, the created feature amount map is output to the second detection unit 106. If there are a plurality of feature amounts corresponding to the function acquired by the function acquisition unit 104, a plurality of feature amount maps are created and output.

  The second detection unit 106 detects a region useful for the detection result utilization function using the input feature amount map. Since the input feature quantity map is a result of calculating a feature quantity useful for the detection result utilization function, the feature quantity map is subjected to threshold processing with a predetermined threshold, and an area exceeding the threshold is set as the second candidate for the main subject. Detect as a region.

  FIG. 4 is a diagram for explaining details of the processing of the second detection unit 106. Here, a case where the automatic tracking function is set as the detection result utilization function will be described. In this case, the feature amount calculation unit 105 creates and outputs a feature amount map having a high value for a region with motion. For the detection target frame 401 shown in FIG. 4A, a feature amount suitable for detecting the movement of the object is calculated, and threshold processing is performed. It can be specified whether a large area exists. Note that since a feature quantity related to motion cannot be calculated with only one frame, the feature quantity of motion is calculated using not only the detection target frame 401 but also the previous frame (not shown). Specifically, for example, a difference is calculated between the detection target frame 401 and the previous frame, and threshold processing is performed. FIG. 4B shows areas 403 and 404 identified as areas with large motion in the detection target frame 401. The second detection unit 106 searches for such a region having a large movement and identifies these regions (403, 404) as second candidate regions.

  The final detection unit 107 determines the final main subject as a main subject detection result based on the first candidate region input from the first detection unit 103 and the second candidate region input from the second detection unit 106. Detect areas. In the present embodiment, a region that exists in common with the first candidate region and the second candidate region is detected as the final main subject region. The final detection unit 107 outputs, as main subject information, the position and size of the detected main subject region, and also the detection region score (main subjectness). The main subject detection score is calculated by weighting each piece of information such as the position and size of the detection area, the correlation value obtained when detecting the first candidate area, and the feature value obtained when detecting the second candidate area. What is necessary is just to calculate by performing calculations, such as addition. The weight value for weighted addition can be determined by learning from a large number of images (samples) by a known learning method.

  Here, details of the processing of the final detection unit 107 will be described with reference to FIGS. 3 and 4. The detection target frame 301 in FIG. 3 and the detection target frame 401 in FIG. 4 are the same frame (input image). At this time, the areas 303 and 304 are specified as the first candidate areas, and the areas 403 and 403 are specified as the second candidate areas. 404 is identified. In the example of the present embodiment, an area that is common to both candidate areas is detected as the final main subject detection area, so the final main subject detection area is the area 304 (or area 403).

  In the present embodiment, if there is no common area between the first candidate area and the second candidate area, the final detection unit 107 determines that there is no main subject. The fact that there is no common area between the two candidate areas means that there is no reliable area as the main subject, so in that case, outputting no main subject will cause false detection of the main subject in the function used later. Can be suppressed. If the main subject area is erroneously detected, the photographer may feel uncomfortable or uncomfortable, so if there is no common area, the possibility is reduced by eliminating the main subject. .

  Next, details of main subject detection processing by the main subject detection apparatus 100 will be described. FIG. 5 is a flowchart showing a processing procedure of main subject detection processing according to the present embodiment. Here, a process for detecting a main subject region for a detection target frame (input image) input at a certain timing (a specific time) will be described.

  In the flowchart shown in FIG. 5, first, in step S501, the function acquisition unit 104 acquires information related to the detection result use function set. In step S <b> 502, the feature amount calculation unit 105 determines a feature amount to be calculated according to the detection result use function acquired by the function acquisition unit 104. The processes in steps S501 and S502 can be performed at an arbitrary timing as long as the process is performed before the timing (specific time point) at which the detection target frame is acquired.

  In step S503, the image acquisition unit 101 determines input of a moving image frame (input image). If an image is input, the input image is passed to the target object detection unit 102, and the process proceeds to step S504. In step S504, the saliency area detection unit 108 of the target object detection unit 102 detects a high saliency area as a candidate for the capture target object. In the subsequent step S505, the same object region specifying unit 109 determines whether the candidates for the capture target objects detected in different frames (input images) are the same object, and specifies the same object region. . Then, the process proceeds to step S506, and the appearance frequency calculation unit 110 counts the number of appearances of each capture target object candidate based on the determination result of the same object region specifying unit 109. The acquisition target object is specified by the processes of steps S504 to S506. The acquisition target object is specified in advance using a plurality of frames that are sequentially acquired at a timing earlier than the detection target frame input at a certain timing (specific time). .

  Then, a detection target frame is input at a certain timing (a specific time), and in step S507, the main subject detection apparatus 100 determines whether or not to perform main subject detection processing on the input image to be processed. If it is determined that the main subject detection process is to be performed on the input image, the process proceeds to step S508 and subsequent steps. First, in step S508, the first detection unit 103 searches for a capture target object from the detection target frame input from the image acquisition unit 101 using a method such as template matching, and detects a first candidate region.

  Next, in step S509, the feature amount calculation unit 105 calculates the type of feature amount determined in step S502 from the detection target frame and, if necessary, the previous frame, and creates a feature amount map. In step S510, the second detection unit 106 detects a second candidate region based on the feature amount map calculated by the feature amount calculation unit. In the flowchart shown in FIG. 5, the processes of steps S508 to S510 are performed after the processes of steps S504 to S506. However, in the case where both the processes of steps S504 to S506 and the processes of steps S508 to S510 are performed on the same image, either may be performed first, or the processes may be performed simultaneously in parallel.

  Subsequently, in step S511, the final detection unit 107 determines whether or not there is a common area between the first candidate area and the second candidate area. If the result of determination in step S511 is that there is a common area, the process proceeds to step S512, and the final detection unit 107 outputs information on the common area as the final candidate area for main subject detection. On the other hand, if there is no common area, the process proceeds to step S513, and the final detection unit 107 determines that there is no main subject in the detection target frame, and outputs the result.

  As described above, the main subject detection apparatus 100 according to the present embodiment detects a capture target object that is to be captured in an image captured by the photographer from a group of frames (a plurality of images) before a certain specific time (first). Detection of one candidate area). At the same time, an object area that is meaningful for an application provided with the result of main subject detection is detected (detection of a second candidate area). Since it is the photographer that determines the angle of view of the imaging device (camera), checking what is frequently present in the angle of view is the photographer's view of the image to be photographed at a particular point in time. This leads to estimation of intention. Also, since it is the photographer who sets what function (application) the result of the main subject detection is used for, it is also possible to know what the main subject is to be estimated for at a certain point in time. It can be said that this leads to estimation of the photographer's intention in the image to be attempted.

  In this embodiment, the photographer's intention is estimated from two different viewpoints as described above to detect a candidate area, and specify the final main subject candidate area from both candidate areas. As described above, by estimating the photographer's intention in multiple layers, in this embodiment, it is possible to suppress detection of the main subject against the photographer's intention. Therefore, it is possible to reduce discomfort and discomfort associated with erroneous detection of the main subject detection result for the photographer who operates the application using the main subject detection result.

  For example, as in the technique of Patent Document 1, when the main subject is estimated only by examining what is frequently present in the angle of view (searching for the first candidate area), in the example of FIG. Alternatively, 304 is specified as the main subject candidate area. Specifically, both the areas 303 and 304 are determined as the main subject area, or the area 303 that is a more stable (not moving) subject is determined as the main subject area. On the other hand, in the main subject detection method of the present embodiment, information on what the main subject detection result is used for (for example, for an automatic tracking function) is also used as a clue. As a result, this embodiment can detect the region 304 (or the region 403) as the main subject region, reflecting the photographer's intention that he / she wants to photograph a moving object as the main subject. It becomes. Thereby, this embodiment can suppress the detection of the main subject contrary to the photographer's intention.

[Modification of First Embodiment]
In the first embodiment described above, if there is no common area between the first candidate area and the second candidate area, the final detection unit 107 determines that there is no main subject. Therefore, as a modification of the first embodiment, when the final detection unit 107 determines that there is no main subject, information on the area specified as the second candidate area at that timing is used as a subsequent frame (temporarily subsequent frame). The form used for the processing in will be described. In addition, about the structure already demonstrated by 1st Embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

  First, the outline of the main subject detection method of this modification will be described with reference to FIG. FIG. 6A shows the input images sequentially acquired by the image acquisition unit 101 arranged in time series. FIGS. 6B and 6C show the first candidate area and the second candidate area detected for each input image in FIG. Here, as in the first embodiment described above, the automatic tracking function is set as the detection result utilization function, and the frame 601 (and previous frames) has a “marker” in the frame. It shows the case.

  In such a case, when the frame 602 is input as a detection target frame, in the first embodiment, “signpost” is detected as the first candidate region and “airplane” is detected as the second candidate region. The final main subject detection result is determined to have no main subject because there is no common area between the first candidate area and the second candidate area. In this modification, in such a case, the object included in the region detected as the second candidate region is added to the capture target object in the main subject detection process in the detection target frame at the next timing. .

  In the case of this modification shown in FIG. 6, the region detected as the second candidate region at the timing of the frame 602 is added to the capture target object in the main subject detection process executed for the next detection target frame 603, It is processed. Therefore, the first candidate area detected for the detection target frame 603 is “signpost” and “airplane”, the second candidate area is “airplane”, and “airplane” is output as the final main subject detection area. It will be. The same output result is obtained for the next detection target frame 604.

  The reason why this modification performs the processing in this way is that the photographer has selected the automatic tracking function as the detection result utilization function, because it is assumed that he / she wants to photograph a moving object as the main subject. is there. In other words, if an object that frequently exists in the angle of view (capture target object) is an object that does not move, it is determined that this is not the main subject, and in such a composition ("composition with a sign"), a moving object This is because it is assumed that it is waiting for ("Airplane") to enter the angle of view. This means that the present modification particularly emphasizes the estimation of which function (application) the main subject detection result is to be used. In order to realize main subject detection based on such estimation, the above-described process is performed in the present modification.

  FIG. 7 is a flowchart showing a processing procedure of main subject detection processing in the present modification. The difference from the first embodiment is that in step S511, if it is determined that there is no common area in the first candidate area and the second candidate area, and if it is determined that there is no main subject, the process proceeds to step S701. is there. In step S701, the final detection unit 107 adds the area detected as the second candidate area in the detection target frame to the area of the capture target object in the processing of the detection target frame at the next timing. The added capture target object region is used in the detection of the first candidate region for the next detection target frame.

  Note that, depending on the moving image frame, an area that is not detected as the first candidate area after some time (eg, the airplane 602 in FIG. 6) without being added as an object to be captured in step S701 after some time. It may be detected as an object to be captured. However, the effect of being able to follow early can be expected when the object is clearly added as a capture target object as in step S701.

[Second Embodiment]
Next, as a second embodiment of the present invention, a mode in which the first candidate region and the second candidate region do not have a common region is classified into three cases and different detection results are output for each case will be described. In addition, about the structure already demonstrated by 1st Embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

  When there is no common area between the first candidate area and the second candidate area, there are the following three types. That is, “1. When there is a first candidate area and a second candidate area but no common area”, “2. When there is a first candidate area but no second candidate area”, “3. This is a case where there is no first candidate area but a second candidate area exists ”.

  First, “1. When both the first candidate area and the second candidate area exist but there is no common area” is as follows. That is, there is no region that satisfies both the result (first candidate region) detected using the capture target object as a clue and the result detected using the detection result utilization function (second candidate region). That's what it means. Therefore, in this case, in this embodiment, it is determined that there is no main subject for the detection target frame.

  Next, “2. The first candidate area exists but the second candidate area does not exist” is a case where an area based on the detection result use function information is not detected. In such a case, for example, when automatic tracking is set as the detection result utilization function, a scene in which a stationary object (animal or the like) is waiting to move is assumed. Since there is no movement while the object is still, the second candidate area is not detected. However, in such a case, there is no problem even if the still animal is used as the main subject, and if it starts to move after that, it may be automatically tracked. Therefore, in the second case, the result of the first candidate area is output as the main subject detection result.

  Next, “3. When there is no first candidate area but there is a second candidate area” is a case where an area corresponding to the capture target object is not detected. In such a case, for example, when automatic tracking is set as a detection result utilization function, a scene in which an airplane enters the screen when shooting a blue sky in fine weather is assumed. Since there is no significant area in the blue sky in fine weather, the object to be captured in the image is not detected. However, in such a case, there is no problem even if the airplane that has entered the frame is automatically tracked. Therefore, in the third case, the result of the second candidate area is output as the main subject detection result.

  FIG. 8 is a flowchart showing a processing procedure of main subject detection processing in the present embodiment. The difference from the first embodiment is that the processes in steps S801 to S806 are added instead of the process in step S513. First, in step S801, the final detection unit 107 determines whether or not there is a first candidate area. If there is a first candidate area, the process proceeds to step S802. If there is no first candidate area, the process proceeds to step S802. The process proceeds to S803. In steps S802 and S803, the final detection unit 107 determines whether or not there is a second candidate area. If there is no common area in the first candidate area and the second candidate area by this three-step determination process, the process is classified into three cases, and an output result corresponding to the above-mentioned case classification in each of steps S804 to S806. Is made.

  As described above, according to the present embodiment, when there is no common area in the first candidate area and the second candidate area, the case is further divided into three cases, and different detection results are output for each, thereby providing a more detailed main subject area. Detection processing can be realized.

[Third Embodiment]
Next, as a third embodiment of the present invention, a mode in which a main subject region is detected using a change in posture of an imaging device (camera) such as a pan / tilt operation or a change in angle of view by a photographer will be described. In addition, about the structure already demonstrated by 1st, 2nd embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

  In general, whether or not the imaging device (camera) is performing the pan / tilt operation is controlled by the camera itself, so that the camera can easily detect the state change. Further, regarding the change in the angle of view by the photographer, it is possible to detect whether or not the camera is moving based on the output of a motion sensor such as a gyro sensor mounted on the camera. In this way, the main subject detection apparatus 100 can acquire information regarding the movement of the apparatus itself (camera).

Next, the relationship between the camera posture change and the main subject area estimated at that time will be described.
First, when the target object detection unit 102 detects the capture target object when there is no movement of the camera, it is a case assumed in the above embodiments. In this case, there is a possibility that it cannot be clearly distinguished whether it is "I want to shoot the object to be captured" or "I just happened to be an object to be captured that entered the angle of view". For this purpose, it is preferable to detect the second candidate area using the detection result utilization function information as in each of the above-described embodiments and use the second candidate area to distinguish between the two.

  On the other hand, when the capture target object is detected by the target object detection unit 102 when the camera moves, the photographer moves the camera (changes the angle of view) so that the photographer selects the capture target object. It can be judged that the intention to manage the angle of view is working. In such a case, it is estimated that the capture target object that is going to be within the angle of view even when the camera is moved is highly likely to be the main subject area. In this embodiment, in such a case, the first candidate area is preferentially output as the final candidate area (main subject area).

  FIG. 9 shows a schematic block diagram of the main subject detection apparatus 100 in the present embodiment. In FIG. 9A, the motion detection unit 901 is configured by a motion sensor such as a gyro sensor provided outside the main subject detection device 100, detects the motion of the camera (the main subject detection device 100 itself), and the information thereof. Is output. Then, the final detection unit 107 acquires camera motion information and specifies the final candidate area (final main subject area) using the information. Specifically, when it is detected from the camera motion information that the camera is moving, the final detection unit 107 prioritizes the first candidate area and identifies the final candidate area (main subject area). For example, when both the first candidate area and the second candidate area exist, but there is no common area, in the present embodiment, the first candidate area is output as the main subject detection result.

  Note that the process here may be performed not based on the presence or absence of camera movement but as a trigger whether or not the magnitude of camera movement exceeds a predetermined threshold. In addition, as a method for measuring the camera posture change, it is possible to directly measure the movement of the camera itself with a sensor or the like, but indirectly the movement of the camera from changes in multiple images taken from the camera imaging unit. It is also possible to estimate it. A known technique called Structure from Motion is known as a method for estimating the movement of a camera from changes in a plurality of images taken from an imaging unit of the camera. In Structure from Motion, the posture change of the camera and the geometric structure of the object photographed by the camera are calculated by using the three-dimensional transition of many feature points in the image. With this method, it is possible to estimate the change in the posture of the camera from only the captured image without using an external sensor or the like. FIG. 9B shows a schematic block diagram of the main subject detection apparatus 100 that estimates the movement of the camera based on such a change in the captured image. The motion detection unit 902 detects and outputs motion information of the camera (main subject detection device 100) using the captured moving image frame. When the motion detection unit 902 estimates the motion of the camera from the moving image frame, the above-described Structure from Motion is used.

  FIG. 10 is a flowchart showing a processing procedure of main subject detection processing in the present embodiment. The difference from the first embodiment is that in step S1001 before step S511, the final detection unit 107 moves in the camera (main subject detection apparatus 100) based on the detection result of the motion detection unit 901 or the motion detection unit 902. There is a step of determining whether or not. If it is determined in this step that there is no movement, the process proceeds to step S511, and the same process as in the first embodiment is performed. On the other hand, if it is determined in step S1001 that the camera is moving, the process proceeds to step S1002, and the final detection unit 107 outputs the first candidate area as the main subject detection result.

  As described above, according to the present embodiment, the main subject region can be detected with high accuracy by detecting the main subject region using the change in the movement of the imaging device (camera), which is contrary to the intention of the photographer. The detection of the main subject can be further suppressed.

[Other Embodiments]
In the above description, as a method for specifying the same object in the same object region specifying unit 109, a method based on correlation calculation has been described as an example, but other methods such as comparing color histograms may be used.

  Further, the process related to the search for the target object to be captured by the target object detection unit 102 is to search for an object that exists frequently over a plurality of frames. However, as long as it is a method for estimating an object that the photographer wants to capture in the image, the capture target object may be searched by another method. For example, the target object detection unit 102 may simply take a prominent object and specify it as a target object. Furthermore, not only is the image captured for a long time across multiple frames, but also how much the object is prominent, or the position in the image where the object exists, etc. A technique may be used.

  In addition, the present invention supplies software (program) for realizing the functions of the above-described embodiments to a system or apparatus via a network or various storage media, and the computer of the system or apparatus (or CPU, MPU, etc.) programs Is read and executed. Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. is not. That is, the present invention includes all the combinations of the above-described embodiments and modifications thereof.

DESCRIPTION OF SYMBOLS 100 Main subject detection apparatus 101 Image acquisition part 102 Target object detection part 103 1st detection part 104 Function acquisition part 105 Feature-value calculation part 106 2nd detection part 107 Final detection part 108 Notable area | region detection part 109 Same object area | region identification part 110 Appearance Frequency calculator

Claims (17)

A main subject detection device for detecting a region of a main subject of an image at a specific time,
Image acquisition means for acquiring a plurality of images in chronological order before the specific time point;
Object detection means for detecting an object having a predetermined frequency or more in a plurality of images acquired by the acquisition means;
First detection means for detecting a first candidate region of a main subject in the image at the specific time point based on information on the object detected by the object detection means;
Function acquisition means for acquiring a function provided with information of the detected main subject area when the main subject area is detected in the image at the specific time;
A feature amount calculating unit that calculates a feature amount of the image at the specific time point according to the function acquired by the function acquiring unit;
Second detection means for detecting a second candidate area of the main subject in the image at the specific time point based on the feature quantity calculated by the feature quantity calculation means;
Final detection means for detecting a main subject area of the image at the specific time point based on the first candidate area detected by the first detection means and the second candidate area detected by the second detection means;
A main subject detection apparatus comprising:   The object detection means calculates the saliency in each pixel of each of the plurality of images acquired by the image acquisition means, and detects an area where the saliency is a predetermined threshold or more as an area where the object exists. The main subject detection device according to claim 1.   The object detection unit calculates the saliency based on a feature amount difference between a small region set in each pixel of each of the plurality of images acquired by the image acquisition unit and a region around the small region. The main subject detection device according to claim 2, wherein:   4. The main subject detection apparatus according to claim 1, wherein the function acquisition unit acquires at least one of autofocus and automatic tracking as the function. 5.   5. The main subject detection apparatus according to claim 4, wherein when the function acquisition unit acquires autofocus as the function, the feature amount calculation unit calculates a color feature amount of the image at the specific time point.   6. The main subject detection device according to claim 4, wherein when the function acquisition unit acquires automatic tracking as the function, the feature amount calculation unit calculates a velocity vector of the image at the specific time point. .   The said last detection means detects the area | region of the main to-be-photographed object of the image in the said specific time for the area | region common to the said 1st candidate area | region and the said 2nd candidate area | region. The main subject detection device according to the item.   The final detection means determines that there is no main subject area in the image at the specific time when there is no area common to the first candidate area and the second candidate area. The main subject detection device according to claim 7.   The final detection means determines the second candidate area when the first candidate area is not detected by the first detection means and there is no area common to the first candidate area and the second candidate area. The main subject detection device according to claim 1, wherein the main subject detection device detects the main subject region of the image at a specific time.   The final detection means, when the second candidate area is not detected by the second detection means, and there is no area common to the first candidate area and the second candidate area, the first candidate area The main subject detection device according to claim 1, wherein the main subject detection device detects the main subject region of the image at a specific time.   11. The information according to claim 1, wherein when the final detection unit determines that there is no main subject region in the image at the specific time, the final detection unit outputs information indicating that there is no main subject region. The main subject detection device according to claim 1.   When the final detection unit fails to detect the main subject region in the image at the specific time point, the final detection unit detects the object included in the second candidate region detected by the second detection unit as the main object of the image after the specific time point. The main subject detection device according to claim 1, wherein when detecting a region of a subject, the main subject detection device adds to the object detected by the object detection means. An image capturing unit for capturing an image;
The main subject detection device according to claim 1, wherein an image captured by the imaging unit is acquired. Motion detection means for detecting a change in motion of the main subject detection device;
14. The main subject detection apparatus according to claim 13, wherein the final detection unit detects a main subject region of the image at the specific time point based on the change in the motion detected by the motion detection unit. .   The main subject detection apparatus according to claim 14, wherein the motion detection unit detects a change in the motion based on a plurality of images captured from the imaging unit. A main subject detection method for detecting a region of a main subject of an image at a specific time,
Acquiring a plurality of continuous images in time series before the specific time point;
Detecting an object having a predetermined number of times or more in the plurality of acquired images;
Detecting a first candidate area of a main subject in the image at the specific time point based on the detected object information;
Obtaining a function provided with information of the detected main subject area when the main subject area is detected in the image at the specific time;
Calculating a feature amount of the image at the specific time according to the acquired function;
Detecting a second candidate region of the main subject in the image at the specific time point based on the calculated feature amount;
Detecting a region of a main subject of the image at the specific time point based on the first candidate region and the second candidate region;
A main subject detection method characterized by comprising:   A program for causing a computer to function as the main subject detection device according to any one of claims 1 to 12.

Images