JP2015106849A - Main subject estimation device, program, and information processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate a main subject in accordance with various imaging situations.SOLUTION: A control section 1 includes: an image acquisition part 21 for acquiring temporally continuous captured images; a subject detection part 23 for detecting subjects from the captured images, respectively; a subject information detection part 24 for detecting subject information (such as a size and a position of a subject within the captured image) from the detected subject; a main subject estimation part 31 for estimating a main subject of the captured image from the subject information; a score function memory 22 for storing a score function S; and a determination condition memory 26 for storing a cancellation condition Jrequired for canceling a subject that is defined as the main subject, from the main subject and a setting condition Jrequired for setting a new subject as the main subject. The score function S is a function that gives a probability for a subject that gives subject information, to be the main subject. The main subject estimation part 31 estimates the main subject within the captured image on the basis of the score function S, the cancellation condition Jand the setting condition J.

Description

  The present invention relates to a main subject estimation device, a program, and an information processing device.

A technique is known in which a photographer's intention to change a main subject is estimated based on a panning or tilting operation of the camera body detected by a gyro sensor or the like, and the main subject is determined from the estimation result (for example, Patent Document 1) below.
JP 2011-223174 A

  However, in the technique of Patent Document 1, since it is necessary to detect a panning or tilting operation in order to determine the main subject, there is a limitation in setting the main subject according to various shooting situations.

  The present invention has been made in view of the above-described facts, and its object is to provide a main subject estimation device, a program, and an information processing device that can estimate a main subject according to various shooting situations. To do.

  In order to solve the above problems, the main subject estimation device of the present invention includes an image acquisition unit that acquires a plurality of images, a subject detection unit that detects a subject from captured images acquired by the image acquisition unit, and a subject detection unit A subject information detection unit that detects subject information related to the size of the captured subject in a captured image, and a main subject estimation unit that estimates a main subject from the plurality of captured images based on the subject information, The subject estimation unit does not satisfy the first condition in the second image acquired after the first image, in which the subject information of the first subject that is the main subject in the first image is satisfied. When the subject information of the second subject different from the subject satisfies the second condition different from the first condition, the main subject is changed to the second subject. Here, the plurality of captured images may be arranged in the order in which the images are actually captured, or may be arranged in the order reversed in time. Examples of the captured image include a still image captured by a digital still camera and a moving image captured by a digital video camera.

The main subject estimation unit according to one preferable aspect of the present invention has a value related to the size or size of the first subject that is equal to or less than the first predetermined value, and a value related to the size or size of the second subject. When the second predetermined value or more is reached, the main subject is changed to the second subject when the second subject is within a predetermined range in the captured image. Here, as the value related to the size of the subject in the captured image, for example, a value obtained by normalizing the size of the subject in the captured image with the size of the first main subject in time, a predetermined number of images Examples include an average value of the size of the subject in the captured image over the image, a value obtained by removing temporally abrupt fluctuation components from time-series data of the size of the subject in the captured image, and the like.
The more preferable main subject estimation unit of this aspect sets the predetermined range to be smaller when the angle of view when the captured image is captured is larger than before the angle of view is increased, and captures the captured image. When the angle of view is reduced, the predetermined range is set to be larger than before the angle of view is reduced.

  Preferably, the subject information includes at least one of a value related to the size, a position of the subject in the captured image, and a value related to the position in addition to the size of the subject in the captured image. The unit uses a function of subject information, and the value of the function reflects the probability that the subject providing the subject information is the main subject or a value related to the probability. Here, the value related to the probability is a value related to the probability that a certain subject will be the main subject, for example, a score value that increases or decreases as the probability increases.

  Preferably, when the angle of view when the captured image is captured is increased, the main subject estimation unit determines the ratio of the second predetermined value to the first predetermined value or the first value from the second predetermined value. The difference obtained by subtracting the predetermined value is set larger than before the angle of view is increased, and when the angle of view when the captured image is captured is decreased, the second predetermined value with respect to the first predetermined value is set. The ratio of the values or the difference obtained by subtracting the first predetermined value from the second predetermined value may be set smaller than before the angle of view decreases.

  A digital camera according to the present invention includes the above-described main subject estimation device.

The program according to the present invention includes an image acquisition step for acquiring a plurality of images, a subject detection step for detecting a subject from the captured image acquired in the image acquisition step, and a captured image of the subject detected in the subject detection step. A subject information detecting step for detecting subject information relating to the size of the subject, a main subject estimating step for estimating a main subject from a plurality of captured images based on the subject information,
In the main subject estimation step, the subject information of the first subject that is the main subject in the first image is acquired after the first image. When the subject information of the second subject that does not satisfy the first condition and the second subject different from the first subject satisfies the second condition different from the first condition, It is characterized by changing to a subject.
In the main subject estimation step of the program according to one preferred aspect of the present invention, the value relating to the size or size of the first subject is less than or equal to the first predetermined value and the size or size relating to the second subject. When the value is equal to or greater than the second predetermined value, the main subject is changed to the second subject when the second subject is within a predetermined range in the captured image. In the main subject estimation step of the program of a more preferable aspect, when the angle of view when the captured image is captured is increased, the predetermined range is set smaller than before the angle of view is increased, and the captured image is When the angle of view at the time of image pickup becomes small, the predetermined range is set larger than before the angle of view becomes small.

  Preferably, the subject information includes at least one of a value related to the size, a position of the subject in the captured image, and a value related to the position in addition to the size of the subject in the captured image. In the process, a function of subject information is used, and the value of the function reflects the probability that the subject giving the subject information is the main subject or a value related to the probability.

  Preferably, in the main subject estimation step, when the angle of view when the captured image is captured increases, the ratio of the second predetermined value to the first predetermined value or the second predetermined value The difference obtained by subtracting the first predetermined value is set larger than before the angle of view increases, and when the angle of view when the captured image is captured becomes smaller, The ratio of the predetermined value of 2 or the difference obtained by subtracting the first predetermined value from the second predetermined value is set smaller than that before the angle of view decreases.

  The information processing apparatus of the present invention estimates the main subject in the captured image according to the above-described program. The information processing apparatus may include the program, or may estimate the main subject according to a program downloaded via the Internet or the like. Alternatively, the information processing apparatus may be instructed to estimate the main subject according to a program stored in another external apparatus. As the information processing apparatus of the present invention, any apparatus capable of processing a captured image, for example, a computer, a portable information communication terminal such as a mobile phone or a tablet terminal, a dedicated image analysis apparatus, or the like can be used.

  According to the present invention, it is possible to estimate a main subject according to various shooting situations.

FIG. 1 is a schematic configuration diagram of a digital camera to which a main subject estimation device and a program according to each embodiment of the present invention are applied. FIG. 2 is a circuit block diagram of the digital camera of FIG. FIG. 3 is a functional block diagram when the control unit of the digital camera of FIG. 1 functions as the main subject estimation apparatus according to the first embodiment of the present invention. FIG. 4 is a functional block diagram when the control unit of the digital camera of FIG. 1 functions as a main subject estimation device according to the first and second embodiments of the present invention. FIG. 5 is a functional block diagram when the control unit of the digital camera of FIG. 1 functions as a main subject estimation apparatus according to the fifth embodiment of the present invention. FIG. 6 is a flowchart showing a process flow of the main subject estimation apparatus according to the first embodiment of the present invention. FIG. 7 is a flowchart showing a detailed processing flow of the main subject estimation step in the flowchart of FIG. FIG. 8 is a diagram for explaining a time-series image to be processed by the main subject estimation device according to each embodiment of the present invention. FIG. 9 is a flowchart showing a processing flow of the main subject estimation apparatus according to the second to fifth embodiments of the present invention. FIG. 10 is a flowchart showing a detailed processing flow of the main subject estimation step according to the second embodiment using the size of the subject as a criterion for determining the main subject. FIG. 11 is a flowchart showing a detailed processing flow of the main subject estimation step according to the fourth embodiment using the position of the subject as a determination criterion for the main subject. FIG. 12 is a flowchart showing a detailed processing flow of the main subject estimation step according to the fourth embodiment using the position of the subject as a determination criterion for the main subject. FIG. 13 is a diagram for explaining a difference in angle of view caused by different focal lengths (wide angle and telephoto). FIG. 14 is a diagram for explaining a change in the size of a subject when images are taken from different shooting distances with cameras having different focal lengths (wide angle and telephoto). FIG. A state in which the objects are arranged at the same shooting distance, (B) shows a state in which the subject 2 has advanced forward from the subject 1. FIG. 15 is a diagram for explaining a first example in the first embodiment of the present invention. FIG. 16 is a diagram for explaining a second example in the first embodiment of the present invention. FIG. 17 is a diagram for explaining a third example in the first embodiment of the present invention. FIG. 18 is a diagram for explaining a fourth example in the first embodiment of the present invention. FIG. 19 is a diagram for explaining a fifth example in the first embodiment of the present invention. FIG. 20 is a diagram for explaining a sixth example in the first embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic configuration diagram of a digital camera to which a main subject estimation device and a program according to each embodiment of the present invention are applied.

A digital camera 10 shown in FIG. 1 includes a camera body 11 and a photographing lens 12 that is replaceably attached to the camera body 11. The photographic lens 12 may be fixed to the camera body 11, but in the present embodiment, the photographic lens 12 will be described as a replaceable lens.
The camera body 11 includes a control unit 1 that controls the entire digital camera 10, an image processing unit 2, an image sensor 5 composed of a CCD, a CMOS, and the like, an image captured by the image sensor 5, shooting information, and a shooting menu. Etc., and a wireless communication unit 9.
The digital camera 10 is configured to be capable of an automatic focusing (AF) operation that automatically performs focus adjustment on a subject. In order to perform this automatic focusing operation, the taking lens 12 includes a focusing lens group 15. As the focusing method, for example, contrast AF, phase difference detection AF, or a focusing method using both is employed. Contrast AF is executed based on the output of the image sensor 5. The phase difference detection AF is executed using an AF sensor module (not shown) or a phase difference detection AF sensor embedded in the image sensor 5.

  FIG. 2 is a circuit block diagram showing a connection mode of each component of the digital camera 10. As shown in the figure, the control unit 1, the image processing unit 2, the image sensor 5, the display 6, the wireless communication unit 9, and the imaging lens 12 described above are connected to the system bus 3 so as to be capable of bidirectional communication. . The digital camera 10 (camera body 11) further includes an operation interface unit 7 and a memory card access unit 8. These components are also connected to the system bus 3 so as to be capable of bidirectional communication with the above-described components.

The control unit 1 includes at least a CPU 1a, a RAM 1b, a ROM 1c, and a flash memory 1d. These components are connected to be able to communicate bidirectionally via the internal bus 1e. The internal bus 1e is connected to the system bus 3. The CPU 1a executes predetermined processing, for example, subject tracking described later, based on a program stored in advance in the ROM 1c or the flash memory 1d. The CPU 1a controls the other components by transmitting commands to other components via the internal bus 1e and the system bus 3. Further, data from the other components, for example, image data transmitted from the image processing unit 2 is sent to the RAM 1b or the flash memory 1d via the system bus 3 and the internal bus 1e, and is sent to the RAM 1b or the flash memory 1d. Stored. The CPU 1a can also perform arithmetic processing on the image data stored in the RAM 1b or the flash memory 1d. The ROM 1c also stores information about the digital camera 10, for example.
The image processing unit 2 performs predetermined image processing such as white balance processing, gamma correction processing, lens aberration correction processing, noise reduction processing, and dynamic range expansion on image data corresponding to the image captured by the image sensor 5. Processing, resize processing of image data, conversion processing from raw data to compressed data such as JPEG, and the like are executed.
In addition to still image shooting, the image sensor 5 can perform continuous shooting and moving image shooting at a predetermined frame rate. The captured image data is transferred to the image processing unit 2 and the display 6 via the system bus 3.
The display 6 can display captured still images or moving images in real time, and can also reproduce and display still images or moving images stored in a storage medium such as a memory card described later. Furthermore, the display 6 may be provided with a touch panel sensor, and the photographer may be able to designate a portion to be focused only by touching a part on the screen displayed on the display 6.
The operation interface unit 7 transmits to the control unit 1 via the system bus 3 operation information of a photographer by operating an operation member (not shown) such as a dial, a button, and a touch panel provided in the digital camera 10. The CPU 1 a of the control unit 1 controls each component according to the operation information transmitted from the operation interface unit 7. The operation interface unit 7 transmits the position and size of the crop area designated by the photographer to the CPU 1a via the system bus 3 and the internal bus 1e.
The memory card access unit 8 reads image data, programs, firmware, and the like stored in the memory card 13 inserted in the digital camera 10, or is imaged by the image sensor 5 and image-processed by the image processing unit 2. A process for writing data to the memory card 13 is executed. The memory card access unit 8 can also transfer a program or firmware stored in the memory card 13 to the RAM 1b or the flash memory 1d of the control unit 1 via the system bus 3 in accordance with a command from the control unit 1. . The CPU 1a executes a predetermined process, for example, a main subject estimation process to be described later, with the updated program or firmware.
The wireless communication unit 9 includes, for example, a WiFi (registered trademark) function, and this function enables the digital camera 10 to connect to the Internet wirelessly. For example, the wireless communication unit 9 may execute communication when downloading a program, firmware, image data, or the like provided on the Internet according to a command from the control unit 1. The downloaded program or the like is transferred to the control unit 1 via the system bus 3. The CPU 1a of the control unit 1 writes the transferred program or the like in the flash memory 1d. The CPU 1a executes a predetermined process, for example, a main subject estimation process to be described later, with the updated program or firmware. Alternatively, the CPU 1a may store the downloaded image data as a captured image in the RAM 1b, display it on the display 6, or write it on the memory card 13. Conversely, the CPU 1a can also upload a captured image captured by the digital camera 10 to a predetermined site or blog on the Internet.
The wireless communication unit 9 controls bidirectional communication with a mobile communication terminal such as a smartphone or a tablet terminal. For example, the control unit 1 transmits image data processed by the image processing unit 2 to the mobile communication terminal via the wireless communication unit 9 or receives image data from the mobile communication terminal to receive the image processing unit 2 or It can be transferred to the display 6. The wireless communication unit 9 receives a command or program sent from the mobile communication terminal and transfers it to the control unit 1. The control unit 1 can also control the digital camera 10 based on commands and programs sent from the mobile communication terminal.

  The photographic lens 12 includes a lens CPU 12a that controls each component described later of the photographic lens 12 according to a command from the control unit 1, a focal length of the photographic lens 12 (when the photographic lens 12 is a single focus lens), and an open F. A lens ROM 12b that stores lens information such as value and aberration information, and an internal bus 12k to which each component of the photographing lens 12 is connected are provided. The lens information stored in the lens ROM 12b is sent to the control unit 1 and the image processing unit 2 via the internal bus 12k and the system bus 3. The control unit 1 executes predetermined processing, for example, subject tracking processing described later, based on information on the focal length of the photographing lens 12. Further, the image processing unit 2 uses the lens information for predetermined image processing and stores it in the image data shooting information area for each frame.

Further, the photographing lens 12 includes a focusing motor 12c and an anti-vibration driving unit 12d for driving the anti-vibration lens group 16 (FIG. 1). The lens CPU 12a controls the focusing motor 12c so as to move the focusing lens group 15 (FIG. 1) to the focusing position in accordance with a command from the control unit 1.
Further, the photographic lens 12 detects a distance encoder 12e for detecting the distance from the image sensor 5 to the subject, and the position of the zoom ring corresponding to the focal length of the photographic lens 12 when the photographic lens 12 is a zoom lens. And a focal length encoder 12f. Information about the detected distance to the subject and the focal length is sent to the control unit 1 via the internal bus 12k and the system bus 3. The control unit 1 executes predetermined processing based on the detected distance to the subject and the focal length, for example, main subject estimation processing described later.

The digital camera 10 may have at least one of an optical camera shake correction function and an electronic hand touch correction. In this case, the photographing lens 12 includes a camera shake detection sensor 12g for detecting a camera shake amount of the photographer. Further, when the optical hand touch correction function is provided, the photographing lens 12 includes an anti-vibration lens group 16 that is driven and controlled so as to prevent camera shake.
A signal indicating the amount of camera shake detected by the camera shake detection sensor 12g is sent to the control unit 1 via the internal bus 12k and the system bus 3. The control unit 1 calculates a driving amount of the image stabilizing lens group 16 for canceling the detected camera shake amount, and transmits the driving amount to the lens CPU 12a. The lens CPU 12a controls the image stabilization drive unit 12d to drive the image stabilization lens group 16 so as to cancel out the camera shake amount.

  The camera shake detection sensor 12g includes angular velocity sensors 17a and 17b (FIG. 1) that detect the angular velocity of the digital camera 10 as camera shake (angle shake). The camera shake detection sensor 12g may also include acceleration sensors 18a and 18b (FIG. 1) for preventing linear shake (so-called “shift shake”).

The digital camera to which the embodiment of the present invention is applied is not limited to the mirrorless camera as shown in FIG. 1, and can be similarly applied to a single-lens reflex camera or a video camera. Further, the present invention can be applied to a digital camera or a video camera provided in a mobile phone, a smartphone, a multi-function information communication terminal, or the like.
(First embodiment)
FIG. 3 shows a functional block diagram when the control unit 1 of the digital camera 10 functions as the main subject estimation device according to the first embodiment.

As illustrated in FIG. 3, the control unit 1 includes an image acquisition unit 21 that acquires temporally continuous captured images transferred from the image processing unit 2, and a subject that detects a subject from the plurality of captured images. And a detection unit 23. In the present embodiment, the image acquisition unit 21 includes, for example, the RAM 1b of FIG. 2 that temporarily stores the captured image transferred from the image processing unit 2. The subject detection unit 23 detects a subject (subject region) by reading a captured image stored in the RAM 1b and extracting a feature (for example, a feature of a person's face) from the image of the subject in the read captured image. The area of the subject to be detected may be the entire subject to be photographed or a part thereof (for example, the “face” portion of a person).
The control unit 1 also includes a subject information detection unit 24 for detecting subject information (size, position, etc. of the subject in the captured image) from the detected subject, and a captured image from the detected subject information. And a main subject estimation unit 31 that estimates a main subject.
The control unit 1 further includes a score function memory 22 that stores the score function S and a determination condition memory 26. The score function memory 22 and the determination condition memory 26 can be realized by at least one of the RAM 1b, the ROM 1c, and the flash memory 1d in FIG. The score function S is a function that uses subject information as a variable and gives a high probability that the subject to which the subject information is given is the main subject. In the determination conditions stored in the determination condition memory 26, at least a cancellation condition J1 necessary for releasing the main subject from the main subject and a certain subject in the image are set as new main subjects. Therefore, the necessary setting condition J ₂ is included. Main subject estimating unit 31, based on the score function S, the release condition J ₁ and setting condition J _2, to estimate the main subject in the captured image.
Further, the control unit 1 includes a tracking control unit 33. The tracking control unit 33 generates a tracking control signal including main subject information in order to track the main subject estimated by the main subject estimation unit 31. For example, the control unit 1 causes the AF lens to be superimposed on the area of the main subject on the display 6 or to drive the focusing motor 12c in accordance with the tracking control signal, so that the photographing lens 12 can adjust the focus on the main subject. Perform a focusing operation. Alternatively, the control unit 1 can control the shutter speed of the digital camera, the aperture of the photographing lens 12, and the ISO sensitivity in order to adjust the exposure to the main subject based on the information on the main subject. The tracking control unit 33 may not generate the tracking control signal, and the tracking control unit 33 itself may perform the control. The main subject information is not limited to the use of this example, and can be used for other purposes, such as cropping image data of a region including the main subject.
The subject detection unit 23, the subject information detection unit 24, the main subject estimation unit 31, and the tracking control unit 33 are realized by the CPU 1a executing a program recorded in the ROM 1c, the flash memory 1d, or the like of the control unit 1. can do.
Here, the score function S stored in the score function memory 22 will be described. In the following, the value of the score function relates to the subject O _i (subscript i is a number assigned to each subject, i = 1, 2,... N (n is the number of subjects)). When it is desired to indicate that the score S is expressed as S (O _i ). When the score function is to be shown as a function of the size L (O _i ) of the subject O _i in the captured image, the score function is expressed as S (L (O _i )), and not only the size L but also the subject O _i. When it is desired to show the score function as a function of the position P (O _i ), the score function is expressed as S (L (O _i ), P (O _i )). When there is no need to specify the subject Oi, it is simply abbreviated as S (L) or S (L, P).

The score function S is a function of at least the size L in the captured image of the subject O _i or a value related to the size. In this embodiment, the score function S is a function that increases as the probability that the subject is the main subject increases. Is set. In general, the larger the subject size L in the captured image, the higher the probability that the subject is the main subject. Therefore, the score function S (O _i ) has a score value that increases as the subject L _i size L increases. It is set to be large. In this case, the score function S (O _i ) is a monotonically increasing function of the size L of the subject O _i . Alternatively, the score function S may be an increase function of the size L within the range of the normal size of the subject. In the other range, the score function S may take a constant value or a decrease function of the size L (a subject that is too large). Exclusion).
Next, the cancellation condition J ₁ and the setting condition J ₂ stored in the determination condition memory 26 will be described. The cancellation condition J ₁ is a necessary condition (but not a sufficient condition) for determining that the subject that is the main subject satisfies the condition J ₁ and that the probability of the subject's likelihood of being a main subject is reduced and is not the main subject. ) Is satisfied. On the other hand, the setting condition J ₂ indicates that when a certain subject satisfies this condition J ₂ , the probability of the subject being the main subject is increased and the condition for determining that the subject is a new main subject is satisfied. It is a condition to show. Here, the setting condition J ₂ is set as a condition that is more severe than a condition in which the cancellation condition J ₁ is not satisfied (that is, the object O ₁ is maintained as the main object). In other words, the set of subject information (subject size, position, or score S) that satisfies the setting condition J ₂ is included in the set of subject information that satisfies the condition that the release condition J ₁ is not satisfied.
When a one-dimensional score function is used as in the present embodiment in order to estimate whether the subject is a main subject, the cancellation condition J ₁ and the setting condition J ₂ are indicated by threshold values or numerical ranges. When using the function S also increases as the score value the probability that the main subject is increased, setting condition J ₂ is a release condition J ₁ is severe conditions than the conditions not satisfied, the threshold setting condition J ₂ which means that greater than the threshold of release conditions J _1. In other words, the set (S> ThB) score values established setting condition J ₂ are included in the set condition is satisfied score value which the cancellation condition J ₁ is not satisfied (S> ThA) (ThA < ThB More obvious).
Conversely, when a score function is used in which the score value decreases as the probability of being a main subject increases, the setting condition J ₂ is more severe than the condition in which the cancellation condition J ₁ is not satisfied. This means that the threshold value of J ₂ is smaller than the threshold value of release condition J ₁ .
As described above, a tendency to continue to maintain the first subject set as the main subject as the main subject rather than newly setting the second subject that was not set as the main subject as the main subject. Therefore, it can be said that the main subject estimation process of the present invention has hysteresis characteristics.

  Next, the processing flow of the main subject estimation apparatus according to the first embodiment of the present invention will be described with reference to FIGS. Among these, FIGS. 6 to 7 are flowcharts showing an example of the processing procedure of the control unit 1. The processing procedure programs shown in FIGS. 6 to 7 are recorded in the RAM 1b, ROM 1c, or flash memory 1d shown in FIG.

As shown in FIG. 6, the image acquisition unit 21 acquires a captured image f _k (k = 1) that is input first in time, that is, a captured image f ₁ (step 100). Then, subject detection unit 23 detects the object Oi in the captured image _{f 1 (i = 1,2, ·} ( step 101). For example, as shown in FIG. 8, the person 1 in the captured image f _1, 2 and 3, the subject detection unit 23 detects the face areas of the persons ₁ , ₂ , and ₃ as subjects O ₁ , O ₂ , and O ₃ , at which time the subject information detection unit 24 captures the captured image. The subject information (subject size L ₁ (O _i )) of the subjects O ₁ , O ₂ , and O ₃ in f ₁ is detected, and the subject information detector 24 sets the subject size L ₁ (O _i ). In addition, only the position P ₁ (O _i ) or the position P ₁ (O _i ) of the subject may be detected.

Returning to FIG. 6, the control unit 1 designates a specific subject (in this example, O ₁ (first subject)) as a main subject from the subjects detected in the captured image f ₁ (step 102). . Hereinafter, the main subject is represented as O _m (currently, O _m = O ₁ ). Regarding the main subject designation method, for example, the main subject estimation unit 31 may automatically estimate the largest subject among the detected subjects as the main subject. Alternatively, the control unit 1 can set the subject designated by the photographer as the main subject. For example, the control unit 1 may recognize the subject designated by the photographer touching an operation member (not shown) or the display 6 as the first main subject.

Next, the image acquisition unit 21 increments the image number k by 1, and acquires the next captured image f _k (step 104). Then, subject detection unit 23 continuously detects the object Oi in the captured image _{f k (i = 1,2, ·} ( step 106). For example, as shown in FIG. 8, in the captured image f _k The subjects O ₁ , O ₂ , and O ₃ that are estimated to be the same person as the persons 1, 2, and 3 detected in the captured image f ₁ are detected. The control unit 1 searches a region around the position corresponding to the position of the subject in the previous captured image in the current captured image, and similar images by matching calculation with the image data of the subject This is possible by detecting a subject having characteristics.

Next, the subject information detection unit 24 detects subject information of the subject O _i (i = 1, 2,...) In the captured image f _k (step 108). The subject information detection unit 24 detects at least a size L _k (O _i ) or a value related to the size of the subject in the captured image as subject information. The subject information detection unit 24 may detect a position P _k (O _i ) of the subject in the captured image or a value related to the position other than the size of the subject or instead of the size of the subject. Note that the size L _k (O _i ) of the subject in the captured image is, for example, the length of one side or diagonal line of the region detected as the subject in the captured image or the area of the region, as shown in FIG. Can be used. As the position P _k (O _i ) of the subject in the captured image, the center position of the area detected as the subject in the captured image, the center of gravity position, or the like can be used.

Next, the main subject estimation unit 31 captures an image based on the value of the score function S (O _i ) regarding at least one of L _k (O _i ) and P _k (O _i ) (i = 1, 2,. The main subject O _m in f _k is estimated (step 110) After estimating the main subject O _m in the captured image f _k , the control unit 1 returns to step 104 and the next input captured image f _{k is} calculated. By executing the same processing, the main subject O _m in the captured image f _k is estimated.

  Next, the process flow of step 110 will be described with reference to FIG.

As shown in FIG. 7, the main subject estimating unit 31 calls the score function S from the score function S memory 22, reads the setting condition J ₂ release conditions J ₁ and the main subject of the main object determination condition memory 26 (step 150).
Next, the main subject estimation unit 31 substitutes the subject information of the current main subject O _m (O _m = O _{1 in the first} loop) into the score function, so that the score value S (O _m ). Main subject estimating unit 31 determines whether the score value S _{(O m)} satisfies the cancellation condition _{J 1} (step 152).
When the score value S (O _m ) does not satisfy the release condition J ₁ (negative determination in step 152), the main subject estimation unit 31 determines that the main subject is maintained (O _m = O ₁ ), and this routine To return to the main routine of FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.

On the other hand, when the score value S (O _m ) satisfies the cancellation condition J ₁ (affirmative determination in step 152), the main subject estimation unit 31 determines the score values S (O _j ) of all subjects other than O _m. searching for a second object _{O n} giving the maximum score value S _max among (step 154). Here obtaining a score value of the second object S (O _n) by substituting the subject information of the object O _n the score function.
It is determined whether or not the score value S (O _n ) satisfies the setting condition J ₂ (step 156). When the score value S (O _n ) does not satisfy the setting condition J ₂ (negative determination in step 156), the main subject estimation unit 31 determines that the main subject should be maintained (O _m = O ₁ ), This routine is returned to return to the main routine of FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.

On the other hand, if the score value S _{(O n)} satisfies the setting condition _{J 2} (affirmative determination in step 156), the main subject estimating unit 31, the second object _{O n} the main subject (step 158) . That is, the main subject number m = n. Thereafter, the main subject estimation unit 31 returns this routine and returns to the main routine of FIG. In this case, the tracking control unit 33 tracks as the main subject different object O _n and the time of the previous captured image. In the main routine of FIG. 6, the control unit 1 repeats the same processing for the next input captured image.

According to the first embodiment, previously designated as a main subject (or estimated) by the value of the score function S of the first object satisfies the cancellation condition J ₁ was, and the main object other than the first object If the value of the score function S of the second object which probability is considered to be high is satisfies a setting condition J _2, the control unit 1, the second object is estimated to be the main subject. For setting condition J ₂ are severe than the conditions for maintaining the first object does not hold the release condition J ₁ to the main subject, the first object which is a main subject earlier, estimated (or specified as a main subject ) And the second subject is less likely to be estimated as the main subject than the first subject. As a result, even if the score value of the subject fluctuates abruptly for some reason that the main subject should not be switched (for example, when a very large second subject accidentally enters the captured image), it is stable. Thus, it is possible to continue to estimate the main subject. Furthermore, according to the first embodiment, since the main subject is estimated based on at least the size of the subject in the captured image, even if the photographer does not pan or tilt the digital camera 10, or the operation thereof Even in the middle of this, the main subject can be accurately estimated.
In the above example, the score function S is a function of the size L (O _i ) of the subject O _i in the captured image, but a value related to the size of the subject is used as a variable of the score function S. May be. As values related to the size, for example, the size L ₁ , L ₂ , L ₃ , average value L _{ave of} the same subject over a predetermined number of temporally continuous frames, their variance values L _v , changes in L ΔL (Changes in the size of the same subject in captured images that are separated in time). These values, combinations of these values, and the like can also be used as variables of the score function. For example, S (L _ave ), S (L, L _v ), S (L _ave , L _v ), S (L, ΔL), S (L _ave , L _v , ΔL), etc. are used as the score function. be able to.
Further, as a variable of the score function S (O _i), of the subject _{O i} not only the size L (O _i), the value relating to the position P (O _i) or position of the object _{O i} in the captured image, alone Or in combination with the size L. When the size L and the position P are used, the score function is S (L, P). At this time, S (L, P) is set so that the score value increases as the position P (O _i ) has a higher probability that the main subject exists in the captured image. However, since the score function S is also a function of the size L, the score value does not increase if the size L of the subject is small even if the subject is located at a position where the existence probability of the main subject is high. The reverse is also true.
Further, as the value related to the position, for example, the positions P ₁ , P ₂ , P ₃ , and the average value P _{ave of} the same subject over a predetermined number of temporally continuous frames, and changes in their variance values P _v , P ΔP (change in position of the same subject in a captured image separated in time), a distance | P−P ₀ |, a position P ₀ (center position of the captured image) that is considered to have the highest probability that the main subject exists An approach tendency to the position P ₀ can be mentioned. As an example of the approach tendency, if the position of the subject is approaching a predetermined position on the screen (for example, the center position of the captured image) as a result of evaluating the subject position over a predetermined number of frames, the score value is increased. If it is far away, the score value is decreased. These values, combinations of these values, and the like can also be used as variables of the score function. For example, as a score function, S (L, P _ave ), S (L, P _v ), S (L _ave , P _v ), S (L, ΔP), S (L _ave , | P−P ₀ |) , S (L, Δ (P−P ₀ )) and the like can be used.
From the above consideration, in the first embodiment, for example, the score function S can be generally expressed as the following equation.

S (x ₁ , x ₂ , x ₃ ,... J = w ₁ .x ₁ + w ₂ .x ₂ + w ₃ .x ₃ +. (1)
Here, the variables x ₁ , x ₂ , x ₃ . w ₁ , w ₂ , w ₃ ,. . . , Each variable _{x 1, x 2, x 3} , a-off weight coefficients, these weighting coefficients _{w 1, w} 2. w ₃ ,... can take positive and negative values and 0 values. For example, the weighting factor of the size L is positive. That is, as the subject size L increases, the score value (probability) that is the main subject increases. Conversely, the weighting coefficient of the distance | P−P ₀ | with respect to the position P ₀ (the center position of the captured image) is negative. That is, as the distance increases, the score value (probability) that is the main subject decreases. In particular, for a variable that is not used for main subject estimation, the weighting coefficient may be set to zero.
Depending on various shooting situations and scenes, variables x ₁ , x ₂ , x ₃ ,... Of the main subject are analyzed, and an appropriate weighting factor w _{1 is set} so that the score function value of the main subject becomes larger. , W ₂ . w ₃ ,. . . Are determined in advance by selection or learning. In this way, by appropriately adjusting the weighting factor, it is possible to estimate the main subject in various shooting situations. Of course, the expression of the score function is not limited to the expression (1).
(Second Embodiment)
Next, a second embodiment will be described. FIG. 4 shows a functional block diagram when the control unit 1 of the digital camera 10 functions as the main subject estimation apparatus according to the second embodiment.

As illustrated in FIG. 4, the control unit 1 includes a predetermined range setting unit 25 and a determination condition setting unit 27 instead of the score function memory 22 and the determination condition memory 26 illustrated in FIG. 3. The predetermined range setting unit 25 sets a predetermined range that is a specific area in the captured image. The determination condition setting unit 27 sets a determination condition for determining whether or not the subject of interest is the main subject from the threshold values stored in the threshold memory 30. In FIG. 4, components having the same functions as those in the first embodiment are given the same reference numerals as those in FIG. 3, and detailed descriptions thereof are omitted. The predetermined range setting unit 25 and the determination condition setting unit 27 can be realized by the CPU 1a executing a program recorded in the ROM 1c, the flash memory 1d, or the like of the control unit 1.
The predetermined range set by the predetermined range setting unit 25 is a specific area in a captured image that is expected to have a high probability that a main subject exists (as an example, a predetermined range 51a in a captured image 55a, b in FIG. b). The predetermined range is set, for example, as a region of a predetermined shape having a predetermined size around a fixed position (for example, the center position of the captured image). In the case of this example, the predetermined range setting unit 25 includes, for example, the position information of the fixed position and the information on the shape and size of the predetermined range stored in the specific memory area of the ROM 1c (FIG. 2) that stores the predetermined range. A predetermined range is set based on this. In addition, the center position of the predetermined range may be adjustable so that the center position of the predetermined range matches the position of the subject that is the main subject first or the position specified by the photographer. Further, the size of the predetermined range may be adjusted by an operation command from the photographer. In addition to the rectangles such as the predetermined ranges 51a and 51b in FIG. 13, the predetermined range can have other shapes that can represent a region where the main subject exists at a high probability, such as a circle or an ellipse.
The main subject estimation unit 31 of the second embodiment estimates a main subject from subjects existing in the captured image based on a predetermined range and determination conditions.

  Next, a processing flow of the main subject estimation apparatus according to the second embodiment of the present invention will be described with reference to FIGS. 9 and 10. 9 and 10 are flowcharts illustrating an example of a processing procedure of the control unit 1. The program of the processing procedure is recorded in the RAM 1b, ROM 1c, or flash memory 1d shown in FIG.

As shown in FIG. 9, the image acquisition unit 21 acquires a captured image f _k (k = 1) that is input first in time, that is, a captured image f ₁ (step 200). Then, subject detection unit 23 detects the object Oi in the captured image _{f 1 (i = 1,2, ·} ( step 201).

Next, the control unit 1 designates a specific subject (in this example, O ₁ is a main subject) as a main subject from the subjects detected in the captured image f ₁ (step 202). The method for specifying the main subject is the same as in the first embodiment.

Next, the image acquisition unit 21 increments the image number k by 1, and acquires the next captured image f _k (step 204). Next, the subject detection unit 23 continuously detects the subject Oi in the captured image f _k (i = 1, 2, (step 206). Regarding a method for continuously detecting the subject in the captured image, This is the same as in the first embodiment.

Next, the subject information detection unit 24 detects subject information of the subject O _i (i = 1, 2,...) In the captured image f _k (step 208). The subject information detection unit 24 detects at least a size L ₁ (O _i ) or a value related to the size as subject information. The subject information detection unit 24 detects a position P ₁ (O _i ) or a value related to the position other than the size of the subject or instead of the size.

Next, the main subject estimation unit 31 estimates the main subject O _m in the captured image f _k based on L _k (O _i ) or position P ₁ (O _i ) (i = 1, 2,... (Step 210). After estimating the main subject O _m in the captured image f _k , the control unit 1 returns to Step 204 and performs the same processing on the next input captured image f _{k + 1} to thereby capture the captured image f _k. Estimate the main subject O _m at ₊₁ .

  Next, the process flow of step 210 will be described with reference to FIG.

  As shown in FIG. 10, the determination condition setting unit 27 reads threshold values ThA and ThB (ThA <ThB) from the threshold memory 30 (step 250). The determination condition setting unit 27 transmits the threshold values ThA and ThB to the main subject estimation unit 31.

Next, the main subject estimation unit 31 has the size L (O _m ) of the subject O _m (O _m = O _{1 in the first} loop) that is assumed to be the main subject at present is equal to or less than the threshold ThA. Whether or not (step 252). If L (O _m ) is larger than the threshold ThA (negative determination in step 252), the main subject estimation unit 31 determines that the main subject should be maintained (O _m = O ₁ ), and returns this routine. Then, the process returns to the main routine of FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.
On the other hand, when L (O _m ) is equal to or smaller than the threshold value ThA (positive determination in step 252), the main subject estimation unit 31 has the largest size among the sizes L (O _j ) of all subjects other than O _m. searching for a second object _{O n} giving L _max (step 254).

When the second object O _n is retrieved, the main subject estimation unit 31, whether or not the position P of the second object O _n (O _n) is within a predetermined range set by a predetermined range setting unit 25 Is determined (step 255). When the position P of the second object O _n (O _n) is not within a predetermined range (negative determination in step 255), the main subject estimating unit 31, regarded as the second object O _n is not the main subject, the The routine is returned to return to the main routine of FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.
On the other hand, when the position P of the second object _{O n (O n)} is within a predetermined range (affirmative determination in Step 255), the main subject estimating unit 31, the size of the second object L _{(O n)} is It is determined whether it is equal to or greater than ThB (step 256). When L (O _n ) is smaller than ThB (negative determination at step 256), the main subject estimation unit 31 determines that the main subject should be maintained (O _m = O ₁ ), and returns this routine. Then, the process returns to the main routine of FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.

If L _{(O n)} is equal to or greater than ThB (affirmative determination in step 256), the main subject estimating unit 31, the second object _{O n} the main subject (step 258). That is, the main subject number m = n. Thereafter, the main subject estimation unit 31 returns this routine and returns to the main routine of FIG. In this case, the tracking control unit 33 tracks as the main subject different object O _n and the time of the previous captured image. In the main routine of FIG. 9, the control unit 1 repeats the same processing for the next captured image.
According to the second embodiment, when the size L of the first subject previously designated (or estimated) as the main subject is equal to or less than ThA, and there is a high probability of being the main subject other than the first subject. When a possible second subject exists within a predetermined range in the imaging screen and the size L is equal to or greater than ThB, the control unit 1 estimates that the second subject is the main subject. Since the threshold ThB is larger than the threshold ThA, the first subject previously estimated as the main subject can easily maintain the estimation as the main subject, and the second subject is the main subject compared to the first subject. It becomes difficult to be estimated. That is, the main subject estimation has a hysteresis characteristic with respect to the evaluation of the subject size.
As a result, even if the score value of the subject fluctuates abruptly for some reason that the main subject should not be switched (for example, when a very large second subject accidentally enters the captured image), it is stable. Thus, it is possible to continue to estimate the main subject.

In the example of FIG. 8, the subject O ₁ is designated or automatically detected as the main subject in the captured image f ₁ . Thereafter, the captured image _{f k,} as compared with the captured image _{f 1,} size _L k _{(O 1)} is reduced in the subject _{O 1,} size _L k _{(O 2)} of the object _{O 2} is increased, the object O ₃ of size _L k _{(O 3)} is further reduced. Here, when L _k (O ₁ ) is equal to or less than ThA and L _k (O ₂ ) is equal to or greater than ThB, it is determined that the main subject has been switched from O ₁ to O ₂ .
Further, according to the second embodiment, since the main subject is estimated based on the size of the subject in the captured image, the photographer does not pan or tilt the digital camera 10, or the operation thereof Even in the middle of this, the main subject can be accurately estimated.

  Further, in the second embodiment, it is also determined whether or not the second subject is within a predetermined range (step 255 in FIG. 10). As described above, in the second embodiment, not only the size of the subject but also the information on the position of the subject can be used for the main subject estimation, so that the main subject can be estimated more accurately.

  In the second embodiment, a mode in which the predetermined range setting unit 25 is omitted is also possible. In this case, step 255 in FIG. 10 is also omitted.

In the second embodiment, an aspect in which the predetermined range setting unit 25 is omitted corresponds to an aspect in which the score function S is a function of only the subject size L in the first embodiment. An example of this mode is that the score function S (L) = w · L (w is a positive constant), the release condition J ₁ is L ≦ ThA, and the setting condition J ₂ is L ≧ ThB. .

In the second embodiment, an aspect in which the predetermined range setting unit 25 is added corresponds to the score function S as a function of the subject size L and the position P in the first embodiment. An example of this aspect is that the score function S (L, P) = w · L (w is a positive constant) when the position P is within the predetermined range, and the score function S (L (L) when the position P is outside the predetermined range. , P) = 0.
(Third embodiment)
Next, a third embodiment will be described. The second embodiment estimates the main subject based on the size of the subject, whereas the third embodiment estimates the main subject based on the position of the subject. In the functional block diagram of the third embodiment, in FIG. 4, the predetermined range setting unit 25 is omitted, and the threshold memory 30 is replaced with threshold areas (first area RA, second area), which will be described later, instead of the threshold values ThA and ThB. ) Information.

  Next, the processing flow of the main subject estimation apparatus according to the third embodiment of the present invention will be described with reference to FIGS. 9 and 11 are flowcharts illustrating an example of a processing procedure of the control unit 1. The program of the processing procedure is recorded in the RAM 1b, ROM 1c, or flash memory 1d shown in FIG.

In the third embodiment, steps 200 to 206 shown in FIG. 9 are performed as described above with respect to the second embodiment. Next, at step 208, the position P ₁ (O _i ) of the subject O _i is detected as subject information of the subject O _i (i = 1, 2,...) In the captured image f _k . Next, in step 210, the main subject estimation unit 31 estimates the main subject O _m in the captured image f _k based on the position P ₁ (O _i ) (i = 1, 2, _... ) Of the subject O _i .

  Next, the process flow of step 210 will be described with reference to FIG.

As shown in FIG. 11, the determination condition setting unit 27 reads information necessary for demarcating the first area RA and the second area RB in the captured image from the threshold memory 30 (step 300). Here, the first region RA includes the second region RB (see FIG. 12 described later). Further, in step 300, the determination condition setting unit 27, the region RA, may read the coordinate values of the representative position _{P 0} in the RB. The position P ₀ is a position with a high probability that the main subject exists, and is the center position of the captured image or the position designated as the position of the main subject first (see FIG. 12 described later). The regions RA and RB may be defined around this position. The determination condition setting unit 27 transmits information on the areas RA, RB, and position P0 to the main subject estimation unit 31.

Next, the main subject estimation unit 31 determines that the position P (O _m ) of the subject O _m (O _m = O _{1 in the first} loop) that is currently regarded as the main subject is within the first area RA. It is determined whether or not the position is (step 302). When P (O _m ) is a position in the first region RA (Yes in Step 302), the main subject estimation unit 31 determines that the main subject should be maintained (O _m = O ₁ ). This routine is then returned to return to the main routine of FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.
On the other hand, when the position P (O _m ) is not a position in the first region RA (No determination in step 302), the main subject estimation unit 31 sets the positions P (O _j ) of all subjects other than O _m . searching for a second object _{O n} in closest distance among the position _{P 0} (step 304).

When the second object O _n is retrieved, the main subject estimating unit 31, the position P of the second object O _n (O _n) determines whether the position of the second region RB ( Step 306). When the position P of the second object _{O n (O n)} is not located in the second region RB (negative determination in step 306), the main subject estimating unit 31, the second object _{O n} is the main subject The routine is returned to the main routine shown in FIG. In this case, the tracking controller 33 tracks the same object as in the immediately preceding captured image as the main subject O _m.
On the other hand, when the position P of the second object _{O n (O n)} is a position in the second region RB (affirmative determination in Step 306), the main subject estimating unit 31, the main the second object _{O n} A subject is set (step 308). That is, the main subject number m = n. Thereafter, the main subject estimation unit 31 returns this routine and returns to the main routine of FIG. In this case, the tracking control unit 33 tracks as the main subject different object O _n and the time of the previous captured image. In the main routine of FIG. 9, the control unit 1 repeats the same processing for the next captured image.
Next, a third embodiment will be described with reference to FIG.
In FIG. 12A, in the captured image 55, the first subject 81 exists in the first region RA where the main subject is likely to exist, and the second subject 82 is the region of the first region RA. It shows the situation that exists outside. At this time, the first subject 81 is recognized as the main subject. Next, FIG. 12B shows a situation in which the first subject 81 moves out of the first region RA and the second subject 82 moves into the first region RA. In this case, the second subject 82 is the main subject in spite of the fact that the first subject exists outside the first region RA and the second subject 82 exists in the first region RA. The first subject is still the main subject, not the subject. Finally, FIG. 12C shows a situation in which the second subject 82 moves into the second region RB while the first subject 81 remains outside the region of the first region RA. In this case, the second subject 82 is estimated as the main subject.
According to the third embodiment, the position of the first subject 81 previously designated (or estimated) as the main subject is a position outside the first RA area, and the main subject is other than the first subject. When the position of the second subject 82, which is considered to have a high probability, is the position in the second region RB included in the first region RA, the control unit 1 selects the second subject. Estimated to be the main subject. Since the second region RB is included in the first region RA, the first subject previously estimated as the main subject can easily maintain the estimation as the main subject, and the second subject is the first subject. Is less likely to be estimated as the main subject. That is, the main subject estimation has a hysteresis characteristic.
As a result, in a situation where the score value of the subject fluctuates rapidly due to some reason that the main subject should not be switched (for example, when a certain subject accidentally enters a position with a high probability of being the main subject in the captured image). Even in this case, it is possible to stably estimate the main subject.
(Fourth embodiment)
Next, a fourth embodiment will be described. The size of the subject in the captured image varies depending on the actual size of the subject, the shooting situation, the type and shape of the subject, the shooting distance from the digital camera to the subject, and the focal length of the shooting lens. For example, a person's face and a vehicle such as an airplane may have different sizes detected in the captured image due to differences in actual size and shape. In the same person shooting, the size of the face area to be detected is larger in the latter than in the former when shooting a person with a wide background and when the person is greatly close-up. On the other hand, the threshold value used in the second embodiment is a fixed value stored in the memory.
The fourth embodiment of the present invention normalizes the threshold value as a fixed value in accordance with the size of the subject that is initially set as the main subject in order to cope with variations in the size of the subject in the captured image. is there. The configuration of the fourth embodiment is substantially the same as the configuration of the second embodiment shown in FIG. The processing flow of the fourth embodiment is substantially the same as that of the second embodiment shown in FIGS. In the following description of the fourth embodiment, detailed description of the same components and processes as those of the second embodiment will be omitted.
In the fourth embodiment of the present invention, the determination condition setting unit 27 in FIG. 4 normalizes the thresholds ThA and ThB used in steps 252 and 256 in FIG. 10 according to the size of the main subject. For example, the normalization method is as follows.
ThA _normal (normalized threshold ThA) → ThA · L _q (O _m )
ThB _normal (normalized threshold ThB) → ThB · L _q (O _m )
Here, L _q is the size of the subject when the current main subject is estimated as the main subject for the first time. ThA and ThB stored in the threshold memory 30 are given as dimensionless values, not as values having a dimension of magnitude. As a result of the normalization, ThA _normal and ThB _normal have dimensions that can be compared with the size of the subject.
According to the normalization method of the fourth embodiment, in step 252 of FIG. 10, the main subject estimation unit 31 determines that the current main subject size is the threshold value ThA normalized according to the initial size. If it is determined that the following is true, the process proceeds to step 254. In step 256 of FIG. 10, the main subject estimation unit 31 determines that the size of the second subject that is a candidate for the main subject is equal to or larger than the threshold ThB normalized according to the initial size of the current main subject. If it is determined, the process proceeds to step 258.
In another example of the fourth embodiment, instead of normalizing the threshold value, the main subject estimation unit 31 may normalize the size of the detected subject with L _q (O _m ) as follows. .
L (O _m ) _normal (normalized first subject size) → L (O _m ) / L _q (O _m )
L (O _n ) _normal (normalized second subject size) → L (O _n ) / L _q (O _m )
The ThA and ThB stored in the threshold memory 30 are given as dimensionless values, not as values having a dimension of magnitude, and L (O _m ) _normal and L (O _n ) _normal compared with these values. Is also a dimensionless quantity.
According to this normalization method of the fourth embodiment, at step 252 in FIG. 10, the main subject estimation unit 31 monitors how much the current main subject has become smaller from its initial size, If it is determined that the ratio is equal to or less than ThA, the process proceeds to step 254. In step 256 of FIG. 10, the main subject estimation unit 31 monitors how much the second subject, which is a candidate for the main subject, is larger than the initial size of the current main subject, If it is determined that the ratio is equal to or greater than ThB, the process proceeds to step 258.
For example, in the first shooting, it is assumed that the main subject is estimated from a plurality of persons with the background widely included in the shooting range. At this time, since the size of the person is small, the threshold value normalized as described above according to the size of the main subject is set to be relatively small. In the present embodiment, since a relatively small threshold is used in a state where each person is small, main subject estimation can be performed accurately. Next, when it is desired to focus on a small number of people from a plurality of persons, the focal length is increased or the person is approached. In this case, the size of the person in the captured image increases. In this embodiment, since the threshold value is greatly normalized according to the size of the enlarged subject, the main subject can be accurately estimated even when the subject is large.
In addition, the size of the subject varies depending not only on the focal length and shooting distance but also on the type of subject. For example, when shooting a vehicle such as an airplane, the size of the subject can be larger than the face area of the person, depending on the algorithm for extracting the contour of the subject. Even in this case, the present embodiment normalizes the threshold according to the size thereof, so that it is possible to accurately estimate the main subject regardless of the type of subject.
In the above example, the threshold value or the size of the subject is normalized by L _q (O _m ), but the present invention is not limited to this example. For example, it is possible to use the size of the subject at another time point.
According to the above as the fourth embodiment, the magnitude of the threshold or object, the actual size of the object, shooting distance from the digital camera to the subject will vary depending on the focal length of the taking lens L _q ( Since normalization is performed using O _m ), more accurate main subject estimation is possible.
(Fifth embodiment)
Next, a fifth embodiment will be described. FIG. 5 is a functional block diagram when the control unit 1 of the digital camera 10 functions as a main subject estimation device according to the fifth embodiment. In FIG. 5, components having the same functions as those in the second and fourth embodiments are given the same reference numerals as those in FIG. 4, and detailed descriptions thereof are omitted. Since the processing flow of the fifth embodiment is substantially the same as that of the second embodiment shown in FIGS. 9 and 10, the second embodiment will be described in the following description of the fifth embodiment. Regarding the same processing as the embodiment, detailed description is omitted, and only different portions will be described.

When the focal length of the taking lens 12 or the size of the crop region changes, the angle of view of the obtained image also changes in accordance with this change. When the angle of view is wide and when it is telephoto, the moving distance within the imaging angle of view differs when the position or angle of the subject changes relative to the digital camera.
In addition, when the field angle is wide and telephoto, even if the subject is a certain size in the captured image, the shooting distance from the digital camera to the subject is different, so the change in the subject size is It is different between wide angle and telephoto. That is, when the shooting distance from the digital camera to the subject changes by a certain distance, the change in the size of the subject when the angle of view is wide is greater than when the telephoto is wide. In this way, the perspective is emphasized at the wide angle, and the perspective is compressed at the telephoto.

  In view of the above fact, the fifth embodiment is different from the second embodiment and the fourth embodiment in the difference in the change in the moving distance in the captured image of the subject due to the difference in the angle of view and the difference in the angle of view. This compensates for differences in the size of the subject.

  As shown in FIG. 5, the control unit 1 further includes an angle-of-view acquisition unit 28 that acquires an angle of view when a captured image is captured, in addition to the components illustrated in FIG. 4. The angle-of-view obtaining unit 28 specifies the focal length information sent from the photographic lens 12 and / or the designation operation of the size of the imaging region transmitted by the photographer via the operation interface unit 7 (for example, 35 mm full). The angle of view when the captured image is captured is acquired based on the size, the APS-C size, the switching operation of 1.2 times, 1.3 times, or the like. The angle of view acquisition unit 28 transmits the acquired information of the angle of view to the predetermined range setting unit 25 and the determination condition setting unit 27. When the digital camera 10 does not have a crop function, or when the crop function is not used, the angle-of-view acquisition unit 28 uses the predetermined range setting unit 25 and the determination condition setting unit as the focal length information from the photographing lens 12. 27 directly.

  The operation of the predetermined range setting unit 25 according to the fifth embodiment will be described with reference to FIG.

FIG. 13 shows the difference in the angle of view between when the focal length of the photographic lens is wide and when it is telephoto. Subject 50 is assumed to be relatively moved in the same movement amount in the direction of arrow 58 at the same photographing distance D ₁ (the length of the arrow 58 denote the amount of movement) (for the digital camera 10 is subject This also includes moving the subject around the screen by changing the direction).
When taken with a wide-angle lens, for imaging range 60 is wider in the photographing distance D ₁ by wide angle, the ratio of subject movement distance with respect to the imaging range 60 is smaller than that of the telephoto. When taken with a telephoto lens contrast, since the imaging range 62 in the photographing distance D ₁ narrowed by a narrow angle, the ratio of subject movement distance with respect to the imaging range 62 is larger than that of the wide angle. This situation is also applicable to the case where the telephoto effect is obtained by cropping a part of the photographing region even when the photographing lens has the same focal length.
Accordingly, in the captured image, an area where the main subject is highly likely to be present is wider when taken at a telephoto position than when taken at a wide angle. In order to cope with this situation, the predetermined range setting unit 25 sets the predetermined range to be smaller (larger) as the angle of view becomes larger (smaller). For example, as shown in FIG. 13, when the predetermined range 51a is set for the captured image 55a when shooting at a wide angle, the predetermined range wider than the predetermined range 51a is set for the captured image 55b when shooting at a telephoto position. 51b is set. As a result, the determination accuracy in step 255 in FIG. 10 can be improved. Note that the area change according to the angle of view can also be applied to the first area RA and the second area RB of the third embodiment.
In FIG. 13, when the shooting distance to the subject 50 is D ₂ (<D ₁ ), the shooting ranges 61 and 63 are smaller than the shooting ranges 60 and 62, respectively. Therefore, when the shooting distance to the subject 50 can be detected, the predetermined range setting unit 25 may increase (decrease) the predetermined range as the shooting distance is shorter (larger).

Next, the operation of the determination condition setting unit 27 according to the fifth embodiment will be described with reference to FIG.
In FIG. 14A, the camera 91 photographs the subjects O ₁ and O ₂ from the same photographing distance l ₁ with a wide-angle lens, and the camera 92 uses the telephoto lens to photograph the subjects O ₁ and O ₂ at the same photographing distance l _2. The shooting situation is shown. Here, it is assumed that the subject O ₁ is the main subject in both the cameras 91 and 92. For the sake of simplicity, the focal length of the telephoto lens of the camera 92 is l ₂ / l ₁ times the focal length of the wide-angle lens of the camera 91, and the actual sizes of the subjects O ₁ and O ₂ are the same. Assuming that the sizes of the subjects O ₁ and O ₂ in the captured images of the cameras 91 and 92 are equal.
FIG. 14B shows a situation in which the subject O ₂ approaches the cameras 91 and 92 by a distance l ₃ from the photographing situation of FIG. In the case of FIG. 14B, the size L _1k (O ₂ ) of the subject O ₂ in the captured image by the camera 91 and the size L _2k (O ₂ ) of the subject O _{2 in} the captured image by the camera 92 are When the size of the subject O _{2 in} the situation of FIG. 14A is L ₁ (O ₂ ), the following is obtained.
Camera 91: L _1k (O ₂ ) = (l ₁ / (l ₁ −l ₃ )) · L ₁ (O ₂ )
Camera 92: L _2k (O ₂ ) = (l ₂ / (l ₂ −l ₃ )) · L ₁ (O ₂ )
Since l ₂ > l ₁ , L _1k (O ₂ )> L _2k (O ₂ ). That is, in the situation of FIG. 14B, the size of the subject O ₂ at the camera 91 is larger than the size of the subject O ₂ at the camera 92.
Assume that the size L ₁ (O ₂ ) of the subject O ₁ is equal to or smaller than ThA, and L _2k (O ₂ )>ThB> L _1k (O ₂ ). In this case, the camera 91 sets the subject O ₂ as the main subject, while the camera 92 does not set the subject O ₂ as the main subject even though the same situation as in FIG. 14B is taken. Can happen. In the fifth embodiment, since the detection rate of the main subject is the same or close to the wide angle and the telephoto, the determination condition setting unit 27 increases the threshold angle ratio ThB / ThA as the field angle at the time of shooting increases (the focal length decreases). (Or the difference ThB-ThA) is increased. In addition, the threshold ratio ThB / ThA (or the difference ThB-ThA) is reduced as the angle of view at the time of shooting decreases (the focal length increases). In this way, by changing the hysteresis characteristics of main subject detection according to the angle of view, it is possible to reduce variations in estimation accuracy of the main subject due to differences in the angle of view at the time of shooting.

For example, when shooting a subject O ₁ from a distance of l ₂ = 10 m with a camera 92 having a lens with a focal length of 100 mm, and shooting the same subject O ₁ from a distance of 5 m with a camera 91 having a lens with a focal length of 50 mm. In any camera, the size of the subject O ₁ in the imaging screen is the same. Here, assuming a situation where the object _{O 2,} which was present in the same shooting distance as the subject _{O 1} approaches the camera 91 and 92 the object _{O 1} unplug 1 m, the object _{O 2} is 9m from the camera 92, the camera 91 The distance is 4 m. Assuming that the actual sizes of the subjects O ₁ and O ₂ are the same, the subject O ₂ is 10/9 = about 1.1 times the subject O ₁ in the imaging screen of the camera 92, and in the imaging screen of the camera 91, The subject O ₂ is 5/4 = 1.25 times the subject O ₁ . Accordingly, the determination condition setting unit 27 increases ThB (or decreases ThA) as the angle of view at the time of imaging increases, and decreases ThB as the angle of view at the time of imaging increases (or / And ThA are increased), in any of the cameras 91 and 92, the subject O ₂ can be estimated as the main subject.

When the shooting distances to the subjects O ₁ and O ₂ can be detected, the determination condition setting unit 27 performs control to increase ThB as the shooting distance decreases and decrease ThB as the shooting distance increases. You can also.
The embodiments of the present invention have been described above. However, the present invention is not limited to the above-described examples, and can be arbitrarily modified within the scope of the present invention.

  For example, the main subject estimation apparatus of the present invention has been described using the digital camera 10 as an example, but the present invention is not limited to application to a digital camera. For example, it can be widely applied to information processing devices such as computers, dedicated image analysis devices, and portable information processing terminals (including mobile phones, smartphones, tablet terminals, etc.) regardless of the presence or absence of a camera function. is there. The present invention includes all devices that can execute estimation of a main subject by inputting a plurality of consecutive captured images even when the camera function is not used or the camera function is not used.

The present invention may be provided in the form of a program for causing the image information processing apparatus to execute estimation of a main subject. The program is included in the scope of the present invention in any form of electronic data downloaded from the Internet or the like, a form stored in a storage medium, or a form described in source code.
The control unit 1 may use not only data that is continuously shot or captured as a still image by the digital camera 10 and sent from the image processing unit 2, but also a plurality of shot images stored or downloaded in a medium. Good. Further, the control unit 1 may use a captured image resized to an image size smaller than that at the time of shooting, in order to save the computation of later processing. The captured images f ₁ , f ₂ ,. . . fk,. . . Are either in the oldest order (i.e. those taken at a later time as the frame number is larger) or in the newer order (i.e. those taken at a later time as the frame number is larger). They may be arranged in order. In the latter case, the captured image has been captured, and the main subject is estimated retroactively in time.

The processing flow in the flowcharts of FIGS. 6, 7, 9 and 10 can be arbitrarily changed. For example, in step 154 in FIG. 7 and step 254 in FIG. 10, after the determinations in steps 152 and 252 are established, the subject that gives the largest or largest score value other than the first subject that is the main subject is selected. A second subject was used. However, the present invention is not limited to this example. The subject that gives the largest or largest score value other than the main subject in the _first captured image f ₁ is set as the second subject, and the subject is also in the subsequent captured image. You may track as a 2nd object continuously as it is.

  The “view angle” acquired by the view angle acquisition unit 28 in FIG. 5 can be represented by “diagonal view angle”, but “horizontal view angle” or “vertical view angle” may be used depending on the situation. it can. When cropping is not performed, since the size of the image sensor is constant, the “focal length value” of the photographing lens itself can be used as the angle of view (in this case, “the angle of view increases” Corresponds to “smaller” and “smaller angle of view” corresponds to “larger focal length”). When cropping, use the value (converted focal length) obtained by multiplying the focal length of the photographic lens by the square root of the value obtained by dividing the original shooting area without cropping by the area of the cropping area (magnification ratio by cropping). (In this case, “increasing the angle of view” corresponds to “decreasing the converted focal length”, and “decreasing the angle of view” corresponds to “increasing the converted focal length”) To do).

  In the first embodiment, in the equation (1), as the variable of the score function S, it is possible to use various combinations of the value related to the size or size of the subject and the value related to the location or position of the subject. Showed that. Hereinafter, examples of such combinations will be described as first to sixth embodiments with reference to FIGS. 15 to 20.

Each of FIG. 15 to FIG. 20 shows a first subject (displayed in a circle) and a second subject (displayed in a square) that have been estimated as the main subject in the captured image. Yes. Moreover, the arrows in the figure indicate the transition of the size or position of the first and second subjects.
(First embodiment)
First, the first embodiment will be described. In the first embodiment, main subject estimation is performed using the size L of the subject and the stability of the size as variables of the score function S. FIG. 15 is a diagram illustrating a shooting situation to which the first embodiment is applied. The shooting situation in FIG. 15 corresponds to the case where the photographer zooms the shooting lens, approaches the subject, or moves away from the subject so that the main subject has almost the same size. For this reason, in the first embodiment, the stability of the size is used in addition to the size L of the subject in the captured image as a variable of the score function. The larger the size L, the more stable the size L. The score function S is increased as the degree increases.
As the stability of the size, an average value L _ave of the size L (average value of the size L of the subject over a predetermined frame) or a deviation or variance value Lv of the size L calculated over the predetermined frame is used. be able to. The deviation of the magnitude L or the variance Lv is considered to be less stable because the larger the value, the greater the variation in the magnitude L. Therefore, the weight coefficient W of Lv becomes negative, and the score S decreases as Lv increases.

As the score function S, S (L, Lv), S (L, L _ave ), S (L, Lv, L _ave ), or the like can be used.

In the example of FIG. 15, L (O ₁ ) hardly changes in the first subject indicated by a circle in FIG. On the other hand, in the second subject indicated by □ in the figure, L (O ₂ ) changes greatly. In the first stage, the first subject and the second subject have almost the same size, but in the next stage, the second subject becomes abruptly larger and larger than the first subject. However, in the third stage, the second subject becomes abruptly smaller and smaller than the first subject. Therefore, in the score function of the second subject, L _ave of the second subject cannot contribute to the score as compared to L _{ave of} the first subject. Alternatively, since the variance value Lv of the second subject is larger than the variance value Lv of the first subject, the score of the second subject is reduced compared to the score of the first subject by the negative weighting factor. . As a result, when focusing only on the size L, in the second stage, L (O ₁ ) is equal to or less than ThA, and even when L (O ₂ ) is equal to or greater than ThB, the main subject to which stability of size L is added. In this estimation, it is possible to maintain the main subject at O ₁ as in the example of FIG. Of course, the main subject may be changed to O ₂ depending on the variation of the size.

Note that the average value L _ave of the magnitude L is not limited to a simple average of magnitudes, but may be obtained, for example, by blocking a sudden change in magnitude within a short time.
(Second embodiment)
Next, a second embodiment will be described. In the second embodiment, main subject estimation is performed using a subject size L and a change in the size as a variable of the score function S. FIG. 16 shows a shooting situation to which the second embodiment is applied. The shooting situation in FIG. 16 is a situation in which, for example, during a race, the size of the first subject, which was previously estimated as the main subject, is overtaken by the second subject and reversed in size. Show.

As a variable of the score function S, the magnitude L and the change ΔL of the magnitude L can be used, and S (L, ΔL) or the like can be used as the score function. ΔL can be a value obtained by subtracting the size of the subject in the captured image input earlier in time from the size of the subject in the captured image input later in time. Alternatively, ΔL can be an average rate of change of the size L obtained by the least square method or the like from the size distribution of the subject in the captured image over a predetermined frame.
ΔL has a negative value when the size of the subject decreases, and has a positive value when the size of the subject increases. Therefore, the score function S decreases more rapidly when the size of the subject decreases, and increases more rapidly when the size of the subject increases. In the shooting situation of FIG. 16, even if the score function S (L) that is a function of only the size L is used, the second subject is estimated as the main subject. However, in the second embodiment, since S (L, ΔL) is used and the sign of the main subject (ΔL> 0) is reflected in the score, compared with the case where the score function S (L) is used, It becomes possible to estimate the main subject more quickly.
(Third embodiment)
Next, a third embodiment will be described. In the third embodiment, main subject estimation is performed using the subject size L and the subject position stability as variables of the score function S. FIG. 17 is a diagram illustrating a shooting situation to which the third embodiment is applied. The shooting situation in FIG. 17 corresponds to a case where the photographer frames the digital camera and follows the main subject so that the main subject maintains the substantially same position in the captured image. For this reason, in the third embodiment, the stability of the position of the subject is used in addition to the size L of the subject in the captured image as a variable of the score function, and S (L, Pv) or the like is used as the score function S. Is set. The value of the score function S is set so as to increase as the subject size L increases and as the stability of the subject position L increases. Here, as a reflection of the stability of the position L, the variance value Pv of the position of the subject is used, and its weight coefficient is set to be negative.

In the example of FIG. 17, the variance value Pv of the position of the first subject indicated by ◯ in the figure is larger than the variance value Pv of the position of the second subject indicated by □ in the figure. For this reason, the score of the first subject is reduced, and the main subject is changed to the second subject. Instead of Pv or in addition to Pv, ΔP that is a change in the position of the subject over a predetermined frame may be used.
(Fourth embodiment)
Next, a fourth embodiment will be described.
FIG. 18 shows a fourth embodiment. In the fourth embodiment, main subject estimation is performed using a subject size L and a distance from a predetermined position of the subject as variables of the score function S. FIG. 18 is a diagram illustrating a shooting situation to which the fourth embodiment is applied. The shooting situation in FIG. 18 corresponds to a case where the photographer frames the digital camera and follows the main subject so that the main subject is maintained in the vicinity of the predetermined position P ₀ in the captured image. Here, the predetermined position P ₀ is a position that is assumed to have a high probability that the main subject exists, such as the center position of the captured image and the position of the AF frame specified by the photographer.
In the fourth embodiment, as a variable of the score function, a distance | P−P ₀ | between the current position P of the subject and the predetermined position P ₀ is used in addition to the size L of the subject in the captured image. Therefore, S (L, | P−P ₀ |) or the like is set as the score function S. Score function S is, the larger the size L of the object, also subject is set to be larger closer to the predetermined position P _0. Therefore, in the score function S, the weighting factor of | P−P ₀ | is set to be negative.

In the example of FIG. 18, the size of the first subject indicated by ◯ in the figure is small, and the size of the second subject indicated by □ in the figure is large. However, the first subject is at a position near the predetermined position P ₀ , but the second subject is still at a position away from the predetermined position P ₀ . As a result, the score of the second subject does not increase so much despite the increase in the size L, and the main subject is maintained as the first subject.
(Fifth embodiment)
Next, a fifth embodiment will be described.
FIG. 19 shows a fifth embodiment. In the fifth embodiment, main subject estimation is performed using the size L of the subject and the tendency of the subject to approach a predetermined position as variables of the score function S. FIG. 19 is a diagram illustrating a shooting situation to which the fifth embodiment is applied. The shooting situation in FIG. 19 corresponds to the case where the photographer frames the digital camera and follows the main subject so that the main subject approaches the predetermined position P ₀ in the captured image. Here, the predetermined position P ₀ is a position that is assumed to have a high probability that the main subject exists, such as the center position of the captured image and the position of the AF frame specified by the photographer.
In the fifth embodiment, as a variable of the score function, in addition to the size L of the subject in the captured image, a time of a distance | P−P ₀ | between the current position P of the subject and the predetermined position P ₀ Since Δ | P−P ₀ | indicating the amount of change is used, S (L, Δ | P−P ₀ |) or the like is set as the score function S. Score function S is set such as the size L is large subject, and the position of the object becomes larger as changes to be close to the predetermined position P _0.

In the example of FIG. 19, the size of the first subject indicated by a circle in FIG. 19 does not change, but its position changes so as to move away from the predetermined position P ₀ , so the score of the first subject is Will be reduced. In contrast, the second object indicated in FIG □, along with their size L is increased, since its position to approach the predetermined position P ₀ is changed, the score of the second subject Will be increased. As a result, the main subject is changed from the first subject to the second subject.
(Sixth embodiment)
Next, a sixth embodiment will be described. In the sixth embodiment, the main subject is estimated using the position of the subject and the stability of the position as a variable of the score function S. FIG. 20 is a diagram for explaining a shooting situation to which the sixth embodiment is applied. The shooting situation in FIG. 20 corresponds to a case where the photographer frames the digital camera and follows the main subject so that the main subject is maintained at substantially the same position (for example, an AF frame) in the captured image. For this reason, in the sixth embodiment, the subject in the captured image, the predetermined position P ₀ (for example, the above-described AF frame or the image center position), and the stability of the position of the subject are used as variables of the score function. As the score function S, S (| P−P ₀ |, Pv) or the like is set. The value of the score function S is subject is set closer to the predetermined position P _0, and as the larger stability of the position of the subject is large. Here, as a reflection of the stability of the position L, the variance value Pv of the position of the subject is used, and its weight coefficient is set to be negative.

In the example of FIG. 20, the first subject indicated by a circle in the figure is initially closer to the predetermined position P ₀ than the second subject indicated by a square in the figure, and is estimated to be the main subject first. It was done. However, the first subject then moved away from the predetermined position P ₀ rapidly and finally approached the position P ₀ again. In contrast, the second object indicated in FIG □ approached stably in position P _0. For this reason, with respect to the position variance value Pv, the first subject is larger than the second subject, and the score S of the first subject is reduced. As a result, in the example of FIG. 20, the main subject is changed to the second subject.

  The above is an example of the first embodiment, but it goes without saying that there are various combinations of variables in addition to the above example. Further, the present invention includes a combination of the above-described embodiments and examples.

DESCRIPTION OF SYMBOLS 1 Control part 2 Image processing unit 5 Image pick-up element 6 Display 10 Digital camera 12 Shooting lens 12a Lens CPU
DESCRIPTION OF SYMBOLS 15 Focusing lens group 21 Image acquisition part 22 Score function memory 23 Subject detection part 24 Subject information detection part 25 Predetermined range setting part 26 Judgment condition memory 27 Judgment condition setting part 28 Angle of view acquisition part 30 Threshold memory 31 Main subject estimation part 50 Subject 55a, 55b Captured image 51a, 51b Predetermined range

Claims (12)

An image acquisition unit for acquiring a plurality of images;
A subject detection unit for detecting a subject from a captured image acquired by the image acquisition unit;
A subject information detection unit for detecting subject information related to the size of the subject detected by the subject detection unit in the captured image;
A main subject estimation unit that estimates a main subject from the plurality of captured images based on the subject information,
The subject estimation unit does not satisfy the first condition in the second image acquired after the first image, the subject information of the first subject that is the main subject in the first image; and The main subject is changed to the second subject when the subject information of the second subject different from the first subject satisfies a second condition different from the first condition. Estimating device. The main subject estimation unit includes:
When a value related to the size or size of the first subject is equal to or less than a first predetermined value, and a value related to the size or size of the second subject is equal to or greater than a second predetermined value, 2. The main subject estimation apparatus according to claim 1, further comprising: changing the main subject to the second subject when the second subject exists within a predetermined range in the captured image. The main subject estimation unit includes:
When the angle of view when the captured image is captured increases, the predetermined range is set smaller than before the angle of view increases,
The main subject according to claim 2, wherein when the angle of view when the captured image is captured becomes smaller, the predetermined range is set larger than before the angle of view is decreased. Estimating device.   The subject information includes at least one of a value related to the size, a position of the subject in the captured image, and a value related to the position in addition to the size of the subject in the captured image, and the main subject estimation The main subject estimation apparatus according to claim 1, wherein the unit uses a function of the subject information, and the value of the function reflects a probability that the subject to which the subject information is provided is a main subject or a value related to the probability. The main subject estimation unit includes:
When the angle of view when the captured image is captured increases, the ratio of the second predetermined value to the first predetermined value, or the first predetermined value from the second predetermined value. Set the subtracted difference larger than before the angle of view increases,
When the angle of view when the captured image is captured becomes small, the ratio of the second predetermined value to the first predetermined value, or the first predetermined value from the second predetermined value. The main subject estimation apparatus according to claim 2, wherein the difference obtained by setting is set to be smaller than that before the angle of view is reduced.   A digital camera comprising the main subject estimation device according to any one of claims 1 to 5. An image acquisition step of acquiring a plurality of images;
A subject detection step of detecting a subject from the captured image acquired in the image acquisition step;
Subject information detection step of detecting subject information relating to the size of the subject detected in the subject detection step in the captured image;
A main subject estimation step of estimating a main subject from the plurality of captured images based on the subject information;
Is a program for causing the information processing apparatus to execute
In the main subject estimation step,
The subject information of the first subject that is the main subject in the first image does not satisfy the first condition in the second image acquired after the first image, and the first subject A program for changing a main subject to the second subject when the subject information of a second subject different from the first subject satisfies a second condition different from the first condition. In the main subject estimation step,
When a value related to the size or size of the first subject is equal to or less than a first predetermined value, and a value related to the size or size of the second subject is equal to or greater than a second predetermined value, The program according to claim 7, further comprising: changing a main subject to the second subject when the second subject is within a predetermined range in the captured image. In the main subject estimation step,
When the angle of view when the captured image is captured increases, the predetermined range is set smaller than before the angle of view increases,
9. The program according to claim 8, wherein when the angle of view when the captured image is captured becomes smaller, the predetermined range is set larger than before the angle of view becomes smaller.   The subject information includes at least one of a value related to the size, a position of the subject in the captured image, and a value related to the position in addition to the size of the subject in the captured image. The program according to claim 7, wherein in the estimation step, a function of the subject information is used, and a value of the function reflects a probability that the subject to which the subject information is provided is a main subject or a value related to the probability. In the main subject estimation step,
When the angle of view when the captured image is captured increases, the ratio of the second predetermined value to the first predetermined value, or the first predetermined value from the second predetermined value. Set the subtracted difference larger than before the angle of view increases,
When the angle of view when the captured image is captured becomes small, the ratio of the second predetermined value to the first predetermined value, or the first predetermined value from the second predetermined value. The program according to claim 8 or 9, wherein the subtracted difference is set smaller than before the angle of view becomes smaller.   An information processing apparatus that estimates a main subject in a captured image according to the program according to claim 7.

Images