JP6274272B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present disclosure relates to an image processing apparatus that determines a main subject in an image, an image processing method, and a program that executes a process of determining a main subject in an image.

JP 2011-166305 A JP 2011-146826 A JP 2011-146827 A JP 2011-160379 A

  Recent digital still cameras and digital video cameras have a face detection function as standard, and are equipped with functions that optimally adjust various camera parameters (focus, brightness, etc.) according to the face position and area. Yes.

On the other hand, Patent Document 1 describes a method in which a user designates a “main subject” as a subject tracking target in a captured image by designating the user.
For example, if the techniques disclosed in Patent Documents 2, 3, and 4 are used, subject tracking surrounding a whole body frame of an arbitrary subject can be realized.
In addition, autofocus / automatic exposure has a function of detecting and tracking a desired area in a captured image and controlling an optical system or the like so that the area is most suitable.
As described above, techniques such as tracking an image designated by the user as a main subject in the captured image, for example, an image area such as a face, and focusing on the face area are known.

By the way, the desired region for tracking and focusing in the captured image, that is, the “main subject” is currently selected by the photographer himself from “a plurality of candidate regions” obtained from various detectors. This is determined by selecting one candidate.
For example, an arbitrary face is selected by a touch panel from a plurality of faces that are displayed on a through image displayed on the screen (a monitoring image of a subject that is displayed at a time other than the shutter operation time) while holding the camera. The main subject is selected by the action. Alternatively, a subject existing in a predetermined area at a user-specified timing (half-press of the shutter or the like) is set as a main subject.

However, considering this user interface in an actual use case, the act of “selection of a main subject by a photographer” is often difficult.
For example, it is difficult for the user to select a finger while holding the camera and aiming at the subject while he wants to use this function to keep focusing on the moving subject.
For example, the designation may be difficult due to the response speed of the user to the change (movement) of the subject. For example, it is a case where a moving subject cannot be specified well on a live view screen.
In the first place, it is difficult for the user to select the main subject with his / her finger on the screen in the situation where the user holds the camera and selects the subject toward the subject.
Also, depending on the resolution of the display screen provided with the touch panel, it may be difficult for the user to select the subject.
Further, depending on the subject size on the display screen provided with the touch panel and the size (thickness) of the user's finger, a desired subject may not be appropriately specified.
In addition, a time lag on the camera system, for example, a time lag between an actual scene and a through image of a captured image, may make it difficult for the user to appropriately specify a subject.

  Furthermore, when this operation is performed during moving image capturing / recording, image shake due to the selection of the main subject is recorded as it is, or re-selection when tracking is lost (failure) due to frame-out or temporary occlusion. It is supposed to be forced to act.

As described above, in a handheld type camera, the act of selecting a main subject itself is difficult in many use cases that require it, and there is a problem in that it gives stress to the photographer.
Therefore, an object of the present disclosure is to realize a technique for determining a target subject desired by a user and setting it as a main subject without a user such as a photographer intentionally selecting a subject.

  An image processing apparatus according to the present disclosure includes a candidate detection unit that detects a candidate image that is a candidate for a main subject for each of a plurality of frames of image data, and a candidate image detected by the candidate detection unit. And a main subject determination unit that determines a main subject in the candidate image using the stable presence level.

  The image processing method of the present disclosure detects a candidate image that is a candidate for a main subject for each of a plurality of frames of image data, obtains a stable presence level in the image data over a plurality of frames for the detected candidate images, This is an image processing method for determining a main subject among candidate images using the degree of stable presence.

  The program according to the present disclosure is determined by a position state determination process for determining a position state in an angle-of-view space for a candidate image detected as a main subject candidate for each of a plurality of frames of image data, and the position state determination process. From the position of the candidate image in each frame, the stable presence degree calculation process for obtaining the stable presence degree in the image data over a plurality of frames for the candidate image, and the stability obtained by the above stable presence degree calculation process This is a program for causing an arithmetic processing device to execute main subject determination processing for determining a main subject among candidate images using the presence level.

  According to these techniques of the present disclosure, the degree of stable presence in a plurality of frames is obtained from the extracted candidate images. The stable presence level is a value indicating the frequency (temporal existence accuracy) that a certain image is in a predetermined position in the field of view. A subject with high probability that the photographer holding the camera seems to aim as a target has a high degree of stable presence. In other words, the subject that the photographer is mainly aiming for is positioned as much as possible in the point or area that the photographer considers the center as the position in the captured image, and further, the photographer aims at the subject and naturally takes a long time. Come into the captured image. Accordingly, a subject (a subject having a high degree of stable presence) that exists in the captured image with a predetermined position in terms of position with a high frequency in time can be estimated as a main subject targeted by the photographer.

  According to the present disclosure, since the main subject is automatically determined in the captured image, it is not necessary for a user such as a photographer to select the main subject. Therefore, user operability and convenience can be improved. For example, product additions such as improved operability when holding an imaging device equipped with the image processing device of the present disclosure in hand, reduction of user stress, and realization of various functions according to automatic main subject determination Increase value.

It is a block diagram of an example of composition of an image processing device of an embodiment of this indication. 6 is a flowchart of main subject determination processing of the image processing apparatus according to the embodiment. It is a block diagram of the imaging device of an embodiment. It is a flowchart of main subject determination processing type I of the embodiment. It is a flowchart of main subject determination processing type II of the embodiment. It is explanatory drawing of the candidate image frame and determination reference point of embodiment. It is explanatory drawing of the distance of the candidate image frame and determination reference point of embodiment. It is explanatory drawing of the stable presence determination by the position state of embodiment. It is a flowchart of the process example 1 of an embodiment. It is a flowchart of the example 2 of processing of an embodiment. It is a flowchart of the process example 3 of an embodiment. It is a flowchart of the process example 4 of an embodiment. It is a flowchart of the process example 5 of an embodiment. It is a flowchart of the process example 6 of an embodiment. It is explanatory drawing of the candidate image frame and determination reference | standard area | region of embodiment. It is explanatory drawing of the stable presence determination by the position state of embodiment. It is a flowchart of the process example 7 of an embodiment. It is a flowchart of the process example 8 of an embodiment. It is a flowchart of the process example 9 of an embodiment. It is a flowchart of the process example 10 of an embodiment. It is a flowchart of the process example 11 of an embodiment. It is a flowchart of the process example 12 of an embodiment. It is a block diagram in the case of application in the computer apparatus of embodiment.

Hereinafter, embodiments will be described in the following order.
<1. Configuration of Image Processing Device>
<2. Configuration of Imaging Device>
<3. Overview of main subject determination process>
[3-1: Processing type I]
[3-2: Processing type II]
[3-3: Opportunity for determining main subject / purpose, etc.]
<4. Specific processing example>
[4-1: Processing Example 1 (Type I; Judgment Reference Point; First Arrival Determination; Cumulative Existence)]
[4-2: Processing Example 2 (Type I; Judgment Reference Point; First Arrival Determination; Continuing Existence)]
[4-3: Processing Example 3 (Type I; Judgment Reference Point; First Arrival Determination; Cumulative Existence; Extended Example)]
[4-4: Process example 4 (Type I; determination reference point; first-come determination; cumulative existence; with condition determination)]
[4-5: Process example 5 (Type I; determination reference point; determination for a certain period of time; cumulative existence)]
[4-6: Processing Example 6 (Type I; Determination Reference Point; Determination for a Fixed Time; Continuing Existence)]
[4-7: Processing Example 7 (Type I; Judgment Reference Area; First Arrival Determination; Cumulative / Continuous Existence)]
[4-8: Process Example 8 (Type I; Determination Criteria Area; Determination for a Fixed Time; Cumulative / Continuous Existence)]
[4-9: Treatment Example 9 (Type II; Judgment Reference Point; Average Presence)]
[4-10: Processing Example 10 (Type II; Judgment Reference Point; Cumulative Existence)]
[4-11: Processing Example 11 (Type II; Judgment Reference Point; Continuing Existence)]
[4-12: Treatment Example 12 (Type II; Judgment Reference Area; Average Presence)]
<5. Application to Program and Computer Device>
<6. Modification>

In addition, the meaning of each term used is as follows.
“Field of view space” means a space that appears in a captured image. Although it mainly has a meaning as a two-dimensional space as a screen plane in a captured image, it is also used as a three-dimensional space including not only this but also a relative distance of a subject to a camera position at the time of imaging. Note that the term “subject distance” is used to describe the relative distance of the subject with respect to the camera position at the time of imaging for the captured image.
In the embodiment, the “viewing angle space” is mainly used to represent a two-dimensional space (an xy coordinate space in the captured image) of the screen plane. In the technology of the present disclosure, the “viewing angle space” is the screen plane. However, when considering “subject distance” in addition to the screen plane, “view angle space” means a three-dimensional space (xyz coordinate space). Become.
Furthermore, as will be described later as a modification, the “view angle space” is only the horizontal position (x coordinate) on the screen plane, only the horizontal position (y coordinate) on the screen plane, or the subject distance (z coordinate). It is also possible to think of it as a one-dimensional space that considers only.

“Position state” is a general term for the situation of the absolute or relative position of the candidate image in the captured image data in the angle of view space.
As a specific example of "position state",
-Relative distance to the determination reference point described in the embodiment-Relative positional relationship and relative distance to the determination reference area described in the embodiment-Position of the candidate image in the two-dimensional plane of the captured image-Subject distance- There is a relative positional relationship between the subject distance and the determination reference point or the determination reference area.

The “stable presence level” is a value indicating the frequency with which a certain subject is in a predetermined position in the field of view. For example, it is an index value that can determine whether or not the image is in a predetermined state with high temporal accuracy. Speaking in accordance with an embodiment to be described later, the candidate image is a value indicating the accumulated time or duration existing in a certain predetermined position state as the position state in the angle-of-view space, or the average presence, etc. For example, an image calculated as the “stable presence”, for example, an image having a high cumulative time or a long duration, can be estimated as a subject mainly aimed by the photographer.

<1. Configuration of Image Processing Device>

FIG. 1 shows a configuration example of an image processing apparatus according to an embodiment.
The image processing apparatus 1 includes a main subject determination unit 2 and a candidate detection unit 3.

The candidate detection unit 3 detects candidate images that are candidates for the main subject for each of the plurality of frames of the input image data Dg, and outputs candidate image information to the main subject determination unit 2.
That is, face image detection, human body image detection, and the like are performed for each frame (or every intermittent frame) of the image data Dg that is continuously input on the time axis, and images that are candidates for the main subject are extracted. .
In addition, face detection, human body detection, etc. can be done by pattern matching method in image analysis for captured image data, but other detectors can be realized in principle by replacing the dictionary used for pattern matching. Is possible. For example, candidate images of main subjects may be extracted as (specific type) dog face detection / cat face detection.
Further, for example, a moving object detection method using a frame difference may be used to detect a moving object, and the moving object may be used as a candidate image. Alternatively, a gaze area extraction method called “Saliency” may be used.
The candidate detection unit 3 then displays information indicating the extracted candidate image, for example, position information (x, y coordinate values) in the two-dimensional direction within the screen of the candidate image, subject distance, image size (number of pixels), and the like as the candidate image. Output as information.

The main subject determination unit 2 obtains a stable presence degree in the image data over a plurality of frames for the candidate image indicated by the candidate image information supplied from the candidate detection unit 3, and uses the obtained stable existence degree to obtain a candidate image. The main subject is determined. Then, main subject information Dm is output.
For this reason, the main subject determination unit 2 is provided with a position state determination unit 2a, a stable presence degree calculation unit 2b, and a setting processing unit 2c as arithmetic processing functions realized by, for example, a software program.

The position state determination unit 2a determines the position state in the angle of view space for the candidate image indicated by the candidate image information.
The stable presence degree calculation unit 2b obtains the stable presence degree in the image data over a plurality of frames for each candidate image from the position state of the candidate image in each frame determined by the position state determination unit 2a.
The setting processing unit 2c performs a process of determining a main subject from each candidate image using the stable presence level obtained by the stable presence level calculating unit 2b and setting a certain candidate image as the main subject. Information indicating an image set as a main subject is output as main subject information Dm and transferred to other application software, a processing circuit unit, or the like.

The candidate detection unit 3 can be implemented as a video processing DSP (Digital Signal Processor) or CPU (Central Processing Unit) as an image analysis device.
The main subject determination unit 2 can be realized by a CPU or DSP as an arithmetic processing unit.
In addition, a configuration in which the CPU or the like configuring the main subject determination unit 2 executes the operation as the candidate detection unit 3 is also assumed. That is, the candidate detection unit 3 that performs image analysis may be configured as one functional block in the arithmetic processing device that serves as the main subject determination unit 2.

The flow of processing by the image processing apparatus 1 is as shown in FIG. 2A.
In step F1000, input of image data Dg is started. The image data Dg is input by an imaging device unit (not shown) or a receiving unit that receives image data transmitted from a separate imaging device. Alternatively, moving image data captured by an imaging device and stored in a recording medium may be reproduced and input.

When the image processing apparatus 1 executes main subject determination, the process proceeds from step F1001 to F1002.
In step F1002, the candidate detection unit 3 performs image analysis, frame difference detection, gaze region detection, and the like for each frame of the sequentially input image data Dg, and starts processing for detecting a predetermined candidate image. .
The candidate detection unit 3 may extract candidate images for all input frames, or may extract candidate images for intermittent frames such as every other frame or every other frame. That is, the candidate detection unit 3 may perform candidate image extraction processing for at least a plurality of frames in time series during the main subject determination period. There are various kinds of images as candidate images depending on the setting. For example, a face image, a human body image, a dog image, a cat image, and the like are conceivable.
Then, the candidate detection unit 3 outputs candidate image information indicating the detected candidate image for each frame to the main subject determination unit 2.

In step F1003, the main subject determination unit 2 performs main subject determination processing.
This main subject determination process is as shown in FIG. 2B.
In step F1, the position state determination unit 2a takes in candidate image information that is sequentially input, and determines the position state of each candidate image. There may be one or more candidate images per frame, and there may be no candidate images in the frame. When a plurality of candidate images exist in a certain frame, the position state is determined for each candidate image.

  In step F2, the stable presence calculation unit 2b calculates the stable presence in the image data over a plurality of frames from the position state of the candidate image in each frame determined in the position state determination process. To do. For example, the stable presence degree is calculated as a value indicating the frequency at which the position state is close to the center of the image.

  In step F3, the setting processing unit 2c determines a main subject in the candidate images using the degree of stable presence for each candidate image. For example, the candidate image having the highest stable presence level (or the value that reaches the predetermined value earliest in time) as a value indicating the frequency at which the position state is close to the center of the image is determined as the main subject. .

In step F1003 in FIG. 2A, the main subject determination unit 2 performs steps F1 to F3 in FIG. 2B as described above to set, for example, one candidate image as the main subject.
In step F1004 in FIG. 2A, the main subject determination unit 2 passes the main subject information Dm to an application program or the like.
In the application program or the like, processing is performed in response to the image set as the main subject being specified. For example, focus control, tracking processing, image effect processing, and the like.

As described above, the main subject determination process by the image processing apparatus 1 obtains a stable presence level in a plurality of frames in the extracted candidate images. In other words, an index value for determining whether or not the image is present in the image is obtained with a high frequency in terms of position and stability.
For example, an object with high probability that an imager holding a camera seems to aim as a target has a high degree of stable presence. In other words, the subject that the photographer is mainly aiming for is positioned as much as possible in the point or area that the photographer considers the center as the position in the captured image, and further, the photographer aims at the subject and naturally takes a long time. Come into the captured image. Therefore, a subject that is stable in position and that exists in the captured image at a high frequency in time (a subject having a high degree of stable presence) can be estimated as a main subject aimed by the photographer.
The main subject is determined based on the degree of stable presence. As a result, the main subject is automatically determined without a specific operation by a user such as a photographer, and the operability of the user is greatly improved in various electronic devices that perform operations according to the main subject setting.

<2. Configuration of Imaging Device>

In the following, the main subject determination operation will be described in detail by taking the imaging apparatus 10 incorporating the image processing apparatus as described above as an example.
FIG. 3 shows a configuration example of the imaging apparatus 10 according to the embodiment. The imaging device 10 is a so-called digital still camera or digital video camera, and is a device that captures / records still images and moving images, and incorporates an image processing device as defined in the claims.
A configuration corresponding to the main subject determination unit 2 in the image processing apparatus is implemented by software in the control unit 30 in the imaging apparatus 10. The control part 30 performs the operation | movement as an image processing method as used in a claim by performing the process based on the program as used in a claim.

  As shown in FIG. 3, the imaging apparatus 10 includes an optical system 11, an imager 12, an optical system driving unit 13, a sensor unit 14, a recording unit 15, a communication unit 16, a digital signal processing unit 20, a control unit 30, a user interface controller ( Hereinafter, “UI controller”) 32 and user interface 33 are included.

The optical system 11 includes a lens such as a cover lens, a zoom lens, and a focus lens, and a diaphragm mechanism. The optical system 11 collects light from the subject on the imager 12.
The imager 12 includes, for example, an image sensor such as a charge coupled device (CCD) type or a complementary metal oxide semiconductor (CMOS) type.
The imager 12 executes, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and the like on the electrical signal obtained by photoelectric conversion in the image sensor, and further performs A / D (Analog / Digital) conversion processing. I do. Then, the imaging signal as digital data is output to the digital signal processing unit 20 at the subsequent stage.

  The optical system drive unit 13 drives the focus lens in the optical system 11 based on the control of the control unit 30 and executes a focus operation. Further, the optical system driving unit 13 drives an aperture mechanism in the optical system 11 based on the control of the control unit 30, and performs exposure adjustment. Furthermore, the optical system drive unit 13 drives the zoom lens in the optical system 11 based on the control of the control unit 30 and executes a zoom operation.

The digital signal processor 20 is configured as an image processor by a DSP or the like, for example. The digital signal processing unit 20 performs various signal processes on the digital signal (captured image signal) from the imager 12.
For example, the digital signal processing unit 20 includes a preprocessing unit 21, a synchronization unit 22, a YC generation unit 23, a resolution conversion unit 24, a codec unit 25, and a candidate detection unit 26.

The pre-processing unit 21 performs clamp processing for clamping the R, G, and B black levels to a predetermined level on the captured image signal from the imager 12, correction processing between the R, G, and B color channels, and the like. Apply.
The synchronization unit 22 performs demosaic processing so that the image data for each pixel has all the R, G, and B color components.
The YC generation unit 23 generates (separates) a luminance (Y) signal and a color (C) signal from R, G, and B image data.
The resolution conversion unit 24 performs resolution conversion processing on image data that has been subjected to various types of signal processing.
The codec unit 25 performs, for example, a recording or communication encoding process on the resolution-converted image data.

The candidate detection unit 26 corresponds to the candidate detection unit 3 described with reference to FIG. 1. For example, image analysis in units of frames is performed on the captured image signal (luminance signal / color signal) obtained by the YC generation unit 23. Processing is performed to extract candidate images. For example, as an example, a face image is detected, and an area where the face image exists is extracted as a candidate image frame. For the extracted candidate image, the position information of the candidate image frame (information on x, y coordinate values, subject distance, etc. on the screen) and size information (for example, the width, height, number of pixels, etc. of the candidate image frame) To the main subject determination unit 31 of the control unit 30 as candidate image information. Here, since it is information indicating the frame of the image area to be a candidate image, the candidate image information is also referred to as “candidate image frame information”.
The candidate image frame information may further include attribute information of the candidate image (type of face, human body, dog, cat, etc., individual (individual) identification information, and further image data itself).
As described above, the candidate detection unit 26 may extract a specific image to be a candidate by the pattern matching method. For example, the candidate detection unit 26 detects a moving object by a method of moving object detection using a frame difference, and selects the moving object as a candidate. It can also be considered as an image. The method of extracting and selecting candidate images is not limited to the above, and various methods can be considered.
The candidate detection unit 26 may generate candidate image frame information by performing a process such as a smoothing process and an outlier removal process on the image.

The control unit 30 is configured by a microcomputer (arithmetic processing unit) including a CPU, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like.
When the CPU executes a program stored in a ROM, a flash memory, or the like, the entire imaging apparatus 10 is comprehensively controlled.
The RAM is used as a work area for various data processing of the CPU for temporary storage of data and programs.
ROM and flash memory (non-volatile memory) are used to store OS (Operating System) for the CPU to control each unit, content files such as image files, application programs for various operations, firmware, and the like. Used. For example, a program for executing main subject determination processing, which will be described later in this example, and an application program that uses the main subject determination result are stored.

Such a control unit 30 includes various signal processing instructions in the digital signal processing unit 20, an imaging operation and a recording operation according to a user operation, a reproduction operation of a recorded image file, a camera operation such as zoom, focus, and exposure adjustment. Controls the operation of each necessary unit for user interface operations and the like.
Further, in the case of the present embodiment, the control unit 30 has a function as the main subject determination unit 31 and executes main subject determination processing described later.
The main subject determination unit 31 is a function corresponding to the main subject determination unit 2 in FIG. 1 and executes the processing as the above-described position state determination unit 2a, stable presence degree calculation unit 2b, and setting processing unit 2c.

  The user interface 33 executes display output and audio output for the user, and accepts user operation input. For this reason, a display device, an operation device, a speaker device, a microphone device, and the like are included. Here, a display unit 34 and an operation unit 35 are shown.

The display unit 34 is a display unit that performs various displays for a user (imager or the like). For example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display formed on the housing of the imaging device 10. The display device is formed. It may be formed using an LCD, an organic EL display, or the like in the form of a so-called viewfinder.
The display unit 34 includes the display device described above and a display driver that causes the display device to perform display. The display driver executes various displays on the display device based on instructions from the control unit 30. For example, the display driver reproduces and displays still images and moving images that have been captured and recorded on a recording medium, or a through image (subject monitoring image) as a moving image based on captured image data of each frame that is captured during a release (shutter operation) standby. ) On the display device screen. Further, various operation menus, icons, messages, etc., that is, display as a GUI (Graphical User Interface) is executed on the screen. In the case of the present embodiment, for example, a display that allows the user to know the determination result of the main subject determination is also executed on a through image or a reproduction image.

The operation unit 35 has an input function for inputting a user operation, and sends a signal corresponding to the input operation to the control unit 30.
The operation unit 35 is realized as, for example, various operators provided on the housing of the imaging device 10 or a touch panel formed on the display unit 34.
As controls on the housing, a playback menu activation button, a determination button, a cross key, a cancel button, a zoom key, a slide key, a shutter button (release button), and the like are provided.
Various operations may be performed by a touch panel operation using icons, menus, and the like displayed on the touch panel and the display unit 34.

  Operations of the display unit 34 and the like of the user interface 33 are controlled by the UI controller 32 in accordance with instructions from the control unit 30. Further, operation information from the operation unit 35 is transmitted to the control unit 30 by the UI controller 32.

The recording unit 15 includes, for example, a non-volatile memory, and functions as a storage area for storing image files (content files) such as still image data and moving image data, image file attribute information, thumbnail images, and the like.
The image file is stored in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or GIF (Graphics Interchange Format).
Various forms of the recording unit 15 can be considered. For example, the recording unit 15 may be a flash memory built in the imaging device 10, or a memory card (for example, a portable flash memory) that can be attached to and detached from the imaging device 10 and a card recording / reproducing that performs recording / reproducing access to the memory card. The form by a part may be sufficient. Further, as a form incorporated in the imaging device 10, it may be realized as an HDD (Hard Disk Drive) or the like.
In addition, a program for executing main subject determination processing, which will be described later in this example, may be stored in the recording unit 15.

The communication unit 16 performs data communication and network communication with an external device in a wired or wireless manner.
For example, captured image data (still image file or moving image file) is communicated between an external display device, a recording device, a playback device, and the like.
In addition, as a network communication unit, for example, communication via various networks such as the Internet, home network, and LAN (Local Area Network) may be performed, and various types of data may be transmitted and received between servers and terminals on the network. .

The sensor unit 14 comprehensively shows various sensors. For example, a gyro sensor for detecting camera shake, an acceleration sensor for detecting the posture of the imaging device 10, and the like are provided. Furthermore, an angular velocity sensor that detects the attitude and movement of the imaging apparatus 10, an illuminance sensor that detects external illuminance for exposure adjustment, and a distance measurement sensor that measures the subject distance may be provided.
Various sensors of the sensor unit 14 transmit detected information to the control unit 30. The control unit 30 can perform various controls using information detected by the sensor unit 14.

<3. Overview of main subject determination process>

In the imaging apparatus 10 having the above configuration, main subject determination processing executed by the control unit 30 (main subject determination unit 31) will be described below.
Later, as various specific processing examples 1 to 12, various main subject determination processing examples will be described. The main subject determination processing is a stable presence in image data over a plurality of frames for a candidate image indicated by candidate image information. This is a process of determining the main subject in the candidate image using the degree of stable presence. As such main subject determination processing, processing type I and processing type II are exemplified below. However, the main subject determination process of the present disclosure is not limited to only two types, and it means that processing examples 1 to 12 described later can be roughly classified into type I and type II.

The processing type I is a processing mode in which determination is executed while capturing candidate image information during a main subject determination period.
On the other hand, processing type II captures (buffers) candidate image information during a certain period. And when the said period passes, it is a processing mode which performs main subject determination using the taken in candidate image information.

[3-1: Processing type I]

FIG. 4 shows the flow of processing type I. In FIG. 4, for each of steps F10 to F15, the correspondence relationship of the processes corresponding to steps F1 (capture / position state determination), F2 (stable presence calculation), and F3 (main subject setting) in FIG. 2B is also shown. deep.
The following is processing by the main subject determination unit 31 of the control unit 30.

In step F <b> 10, the control unit 30 takes in candidate image frame information for a certain frame from the candidate detection unit 26.
In step F11, the control unit 30 determines the position state by calculating the position in the angle-of-view space for each of one or more candidate image frames indicated by the captured candidate image frame information.
In this case, the distance of the candidate image with respect to the determination reference point set in the view angle space is determined as the position state. Alternatively, as the position state, the positional relationship of the candidate image with respect to the determination reference region set in the view angle space is determined.

In step F12, the control unit 30 calculates a stable presence level for each candidate image frame. In this case, the control unit 30 calculates the accumulated time information that the position state satisfies the predetermined condition as the degree of stable presence. Alternatively, the control unit 30 calculates duration information that the position state continues and satisfies a predetermined condition as the degree of stable presence.
Note that the position information of the candidate image in the angle-of-view space or the size information of the candidate image may be used as the calculation condition in calculating the stable presence level.

In step F13, the control unit 30 determines the main subject using the degree of stable presence.
Here, the determination in step F13 is processing for determining, as the main subject, a candidate image whose stability presence level has reached the predetermined value earliest from the start of main subject determination. Alternatively, in the main subject determination period, the candidate image having the highest stable presence value is determined as the main subject.
In addition to the value of the stable presence level, the position information of the candidate image in the angle of view space or the size information of the candidate image may be used for determining the main subject.

  A candidate image having the highest stable presence value in the period when the candidate image whose stable presence value has reached the predetermined value does not yet exist or the predetermined main subject determination period has not elapsed. At the point when it cannot be selected yet, it cannot be determined as the main subject in the process of step F13. In that case, the control unit 30 returns from Step F14 to F10 and repeats each process. That is, the candidate image frame information of the frame processed next by the candidate detection unit 26 is fetched, and the same processing is performed.

When a candidate image whose stable presence value has reached a predetermined value at a certain time point is found, or after a predetermined main subject determination period, a candidate image having the highest stable presence value in that period Can be selected, the control unit 30 proceeds from step F14 to step F15. Then, the candidate image determined in step F13 is set as the main subject.

[3-2: Processing type II]

Next, FIG. 5 shows a processing flow of the processing type II. Also in FIG. 5, for each of steps F20 to F24, correspondence relationships of processes corresponding to steps F1 (capture / position state determination), F2 (stability presence calculation), and F3 (main subject setting) in FIG. 2B are also shown. .
The following is processing by the main subject determination unit 31 of the control unit 30.

In step F <b> 20, the control unit 30 takes in candidate image frame information for a certain frame from the candidate detection unit 26.
In step F <b> 21, the control unit 30 determines the position state by calculating the position in the angle-of-view space for each of one or more candidate image frames indicated by the captured candidate image frame information.
For example, the distance of the candidate image with respect to the determination reference point set in the view angle space is determined as the position state. Alternatively, as the position state, the positional relationship of the candidate image with respect to the determination reference region set in the view angle space is determined.
The control unit 30 accumulates (buffers) information on the calculated position state (distance and positional relationship) in an internal RAM or the like.

In step F22, the control unit 30 determines whether or not the accumulation of the position state information has been completed for a predetermined period (the length of time or the number of frames as the predetermined main subject determination period). If not, the process returns to step F20.
That is, it is determined whether or not the predetermined period has elapsed or the processing of steps F20 and F21 has been performed for a predetermined number of frames.

When the accumulation is completed, the control unit 30 proceeds to step F23.
In step F23, the control unit 30 calculates the degree of stable presence using, for example, the position state information for a predetermined period that has been buffered for each candidate image frame.
In this case, the control unit 30 calculates, as the stable presence degree, cumulative time information in which the position state satisfies the predetermined condition, or duration information in which the position state continues and satisfies the predetermined condition.
Note that the position information of the candidate image in the angle-of-view space or the size information of the candidate image may be used as the calculation condition in calculating the stable presence level.

In step F24, the control unit 30 determines the main subject using the degree of stable presence.
For example, during the main subject determination period in which buffering is performed, the candidate image having the highest stable presence value is determined as the main subject, and the main subject is set.
Note that the main subject may be determined by using position information of a candidate image in the angle-of-view space or size information of the candidate image in addition to the value of the stable presence level.

As an example of the processing type II in FIG. 5, an example may be considered in which step F21 is performed after it is determined in step F22 that accumulation has been completed. That is, this is a method of accumulating candidate image frame information for a predetermined period and obtaining position state information for each candidate image frame information after the period has elapsed.

[3-3: Opportunity for determining main subject / purpose, etc.]

In the present embodiment, main subject determination is performed by the above processing types I and II. Opportunities and purposes for performing main subject determination in the imaging apparatus 10 will be described.

First, an example of using the main subject determination result will be described.
The main subject determination is executed, for example, when the user (photographer) is aiming for the shutter timing (release timing). The control unit 30 automatically determines the main subject and then performs the following processing. be able to.

-Tracking process The main subject set in each frame to be imaged is tracked. For example, the main subject is clearly shown to the user on the through image display, and is used for the user's angle of view adjustment (for example, subject determination in a state where the camera is held in hand).
In addition, as a presentation of the main subject, it is conceivable that the frame of the main subject is highlighted on the through image display on the display unit 34. The highlighting or the like may be performed for a certain period immediately after the determination, or may be executed as long as the main subject exists in the through image.

• Focus adjustment Auto focus control is performed on the main subject. In addition to the tracking process, even when the main subject moves around, the focus is adjusted by tracking the main subject.
-Exposure adjustment Automatic exposure adjustment is performed based on the brightness (luminance) of the main subject.
Directivity adjustment When sound is collected by a microphone together with imaging (for example, moving image capturing), directivity adjustment is performed according to the direction of the main subject in the angle of view space.

It can also be used for various signal processing on captured image signals.
Image effect processing Image processing such as image quality adjustment, noise reduction, and skin color adjustment is applied only to the area of the main subject in each imaged frame.
Alternatively, it is also possible to add image effects such as mosaic processing, blurring processing, and painting processing in addition to the main subject area.

-Image editing processing Edit processing such as framing and cropping is applied to the captured image or recorded image.
For example, it is possible to perform processing such as cutting out or enlarging a partial area in a frame including the main subject.
Further, in the captured image data, composition adjustment can be performed by cutting the periphery of the image so that the main subject is arranged at the center of the image.

  These are merely examples, but other than these, various processes in which the application program and the automatic adjustment function in the imaging apparatus use the set main subject can be considered.

Next, various times may be considered as to when the main subject determination process is executed.
For example, when the imaging apparatus 10 is turned on and imaging is performed (a period during which a through image is displayed on the display unit 34), the main subject determination process may be performed at all times.
Further, when the main subject is determined, when the tracking process is performed, the main subject determination process may be performed again when the tracking is lost.
Further, the main subject determination process may be started by a user operation.
Further, it may be executed at all times when the user selects the determination execution mode, or may be executed when tracking is lost.
It is also conceivable that the main subject determination process is automatically activated regardless of the user operation.

The effects of performing the main subject determination process are as follows.
As described above, when the photographer is aiming at the subject with the imaging device 10, an operation for designating the main subject is difficult in the first place. Also, the act of specifying it over and over is troublesome. By automatically executing the main subject determination, such a situation is eliminated and an effect of reducing user stress can be obtained.
In addition, as the imaging apparatus 10 that is carried and used by the user, such as a digital still camera that is normally used by the user and a camera with a built-in mobile phone, the display unit 34 is also small, and the user performs an operation of specifying a main subject on the screen. Even so, it is difficult to do accurately. Automatic determination as in this embodiment eliminates erroneous designation.

In addition, when the imaging device 10 automatically performs main subject determination, for the user, the main subject determination is executed only by holding the imaging device 10 aiming at the subject, and the user is in the device. The effect of improving the sense of intelligent feeling and improving added value can be obtained.
In addition, since the imaging device 10 can be used as if it is naturally held, it can be used as if it is a main role, so that it is possible to increase imaging opportunities and use cases, and to provide a user with an easy-to-use camera.

From the above, the imaging apparatus 10 of the present embodiment that automatically performs main subject determination is particularly suitable as a handheld camera.

<4. Specific processing example>
[4-1: Processing Example 1 (Type I; Judgment Reference Point; First Arrival Determination; Cumulative Existence)]

Hereinafter, Process Examples 1 to 12 will be sequentially described as specific examples of the main subject determination process executed by the control unit 30.
Process example 1 is a specific example of process type I described above.
Then, as the position state of the candidate image frame, a distance from the set determination reference point is obtained.
Further, as the degree of stable presence for each candidate image frame, cumulative time information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is calculated.
In addition, a candidate image whose stable presence level reaches the predetermined value earliest from the start of main subject determination is determined as a main subject.

  Prior to the description of the specific processing, each of the candidate image frame, the distance to the determination reference point, and the stable presence level will be described with reference to FIGS. 6, 7, and 8.

FIG. 6 schematically shows a candidate image frame extraction operation performed by the candidate detection unit 26.
In the figure, each frame FR1, FR2, FR3... Of the captured image signal input to the digital signal processing unit 20 by the operation of the optical system 11 and the imager 12 of the imaging apparatus 10 is shown. The candidate detection unit 26 detects a candidate image for each successive frame (or each intermittent frame) that is sequentially input.
For example, as shown in the figure, when there are three people in the frame FR1, each face image portion is extracted as a candidate image, and candidate image frame information for the candidate image frames E1, E2, and E3 is output. To do. For example, the candidate image frame information of the candidate image frame E1 includes, for example, position information (x, y position information, subject distance information) in the image of the candidate image frame E1, size information (frame width, height, number of pixels). , Attribute information and the like.
The candidate detection unit 26 also generates such candidate image frame information for the candidate image frames E2 and E3, and passes them to the control unit 30 (main subject determination unit 31).

  Similarly, for the subsequent frames FR2, FR3,..., The candidate detection unit 26 extracts candidate images, generates candidate image frame information for each candidate image frame, and transfers the candidate image frame information to the control unit 30.

Each time the candidate image frame information of each frame is captured, the control unit 30 calculates the distance from the determination reference point as the position state for each candidate image frame.
FIG. 7A shows an example of the determination reference point SP. This is an example in which the center of the image is the determination reference point SP. Let the xy coordinate value of the determination reference point SP be (Cx, Cy).
For example, when the candidate image frame information of each of the candidate image frames E1, E2, and E3 is captured at the time of the frame FR1, the control unit 30 determines the determination criterion from the center of gravity G of each of the candidate image frames E1, E2, and E3 illustrated. The distances Diff1, Diff2, and Diff3 of the point SP are calculated.

The determination reference point SP is an example of the center of the screen.
For example, as shown in FIG. 7B, the determination reference point SP may be set at a position slightly upper left from the center. For example, when considering the composition of a still image, there are many cases where the main subject may be arranged at a position other than the center.
In any case, as shown in FIG. 7B, the distance between each candidate image frame (for example, E4, E5) and the determination reference point SP (for example, Diff4, Diff5) is calculated.

The determination reference point SP may be fixedly set at a position as shown in FIGS. 7A and 7B, for example, or may be arbitrarily designated by the user by a touch operation on the display screen. Further, some candidate points for determination reference points may be presented to the user on the screen of the display unit 34 so that the user can select them. Furthermore, it is also conceivable that the control unit 30 determines the optimum position in consideration of the composition and the like according to the image content, the image analysis result, etc., and automatically sets it.
That is, for the determination reference point SP,
A predetermined fixed position is set as the center position of the image or the position shifted from the center.
・ Specified arbitrarily by the user.
-Some candidate points are presented to the user and set by the user selecting.
The control unit 30 determines the optimum position according to the image content and automatically variably sets it.
And so on.

The distance Diff (n) as the position state for the candidate image frame E (n) is obtained at the time of each frame.
FIG. 8 shows how the calculated distances Diff1, Diff2, and Diff3 vary, assuming that the candidate image frames E1, E2, and E3 continue to exist in the frame (FR1, FR2,...) For a certain period. Is shown.
For example, it is assumed that the photographer has captured three subjects with the imaging device 10 for a certain period of time. The calculated distances Diff1, Diff2, and Diff3 fluctuate on the time axis because each of the three people is moving, or the photographer is moving the subject direction of the imaging apparatus 10 or there is a camera shake. doing.

In Processing Example 1, the stable presence level is an accumulated time in a state close to the determination reference point SP. Therefore, the distance threshold Thr-diff is used for the determination of “near / not close”.
The lower part of FIG. 8 shows the determination result as to whether or not the distance is within the distance threshold Thr-diff at each time of distances Diff1, Diff2, and Diff3. If the distance Diff (n) is equal to or smaller than the distance threshold Thr-diff, it is assumed that close = “1”.
The result of cumulative addition of the determination result “1” at each time point is the stable presence degree in the processing example 1.

The period from the determination start to the determination end varies depending on the processing example. In the processing example 1, the cumulative addition value of the determination result “1” that is equal to or smaller than the distance threshold Thr-diff is a cumulative time that indicates the degree of stable presence up to that point, but the candidate image whose cumulative time has reached a predetermined value The point in time when is found is the end timing of the determination.
For example, in the example of FIG. 8, the candidate image frame E3 is continuously determined to be “1 (= close to the determination reference point SP)”, but the determination is performed when the accumulated addition value reaches a certain predetermined value. Is completed, and the candidate image frame E3 is determined to be the main subject.

  In processing example 1, continuity is not questioned. For example, although the distance Diff2 of the candidate image frame E2 in FIG. 8 is determined to be “1” or “0” depending on the time point, since it is an accumulated time, there are many situations of “1”, which is earlier than other candidate image frames. If the accumulated time reaches a predetermined value, the candidate image frame E2 may be determined as the main subject.

The main subject determination process of the control unit 30 as Process Example 1 will be described with reference to FIG.
When the main subject determination process is started, the control unit 30 first sets a variable TSF = 0 and a count value Cnt (n) = 0 in step F100.
The variable TSF is a flag indicating whether or not the main subject has been set. TSF = “0” indicates that the main subject has not been determined.
The count value Cnt (n) is a value of a counter that adds the value of the comparison determination result with the distance threshold Thr-diff of the above-described distance Diff.

Note that “n” represents a natural number 1, 2, 3,..., And the count value Cnt (n) is applied to each detected candidate image frame E (n) like the candidate image frames E1, E2, E3. The corresponding count value is used. For example, when three candidate image frames E1, E2, and E3 are detected, Cnt1, Cnt2, and Cnt3 are used as count values. In the description of the flowchart, it should be understood that the process for the count value Cnt (n) indicates a process for each of Cnt1, Cnt2, and Cnt3, for example.
Similarly, the distance Diff (n) collectively represents the distances Diff1, Diff2, and Diff3 from the determination reference point SP of the three candidate image frames E1, E2, and E3. This processing is used to mean processing for each of the distances Diff1, Diff2, and Diff3.

  Moreover, although the candidate image frame E (n) represents candidate image frames E1, E2, E3,... For example, in the example in which the candidate detection unit 26 extracts a face, when the person A, the person B, and the person C are subjects, the face image portion of the person A is the candidate image frame E1, the person in common in each frame. The face image portion of B is set as a candidate image frame E2, and the face image portion of the person C is set as a candidate image frame E3. Even if only the person D enters the subject in a certain frame on the way, the face image portion of the person D is set as a candidate image frame E4. Therefore, the candidate detection unit 26 may not only detect the “face” but also perform individual (individual) discrimination.

In step F <b> 101, the control unit 30 takes in candidate image frame information for a certain frame from the candidate detection unit 26. For example, for each candidate image frame E (n), as position information, the x value and y value as the two-dimensional (xy) coordinate values of the image data, and the width w and height h of the candidate image frame as size information. Get information.
The candidate image frame information includes subject distance (relative distance of the subject to the camera position: z value represented by a value in the z-axis direction orthogonal to the two-dimensional (xy) coordinate plane) and the number of pixels. Etc. may also be included.

In step F102, the control unit 30 calculates the coordinates of the center of gravity G for each candidate image frame E (n).
For example, depending on the candidate image frame information, the coordinate value of the upper left vertex of the rectangular candidate image frame is given as the x and y coordinate values of the candidate image frame. The x and y coordinate values are (E (n) _x, E (n) _y). Further, as shown in FIG. 7, the x, y coordinates are set to the origin O (x, y coordinate value is (0, 0)) at the upper left of the screen plane.
The width w of the candidate image frame E (n) is E (n) _w, and the height h is E (n) _h.
If the coordinate value of the centroid G of the candidate image frame E (n) is (E (n) _cx, E (n) _cy), the coordinate value of the centroid G is
E (n) _cx = E (n) _cx + E (n) _w / 2
E (n) _cy = E (n) _cy + E (n) _h / 2
As required.

In step F103, the control unit 30 calculates a distance Diff (n) from the center of gravity G of each candidate image frame E (n) to the determination reference point SP. As coordinate values (Cx, Cy) of the determination reference point SP,
Diff (n) = √ {(E (n) _cx−Cx) ² + (E (n) _cy−Cy) ² }
Is required.

In step F104, the control unit 30 confirms the variable TSF. If variable TSF = 0, the process proceeds to step F105.
Note that if the processing is to be continued all the time after the main subject determination processing is started, the processing in step F104 is skipped when the determination is unnecessary (when the variable TSF = 1). Step F104 may be unnecessary when the main subject determination process of FIG. 9 is executed when necessary by user operation or automatic activation determination. The same applies to processing example 2 and later described later.

In steps F105, F106, and F107, the control unit 30 confirms whether each candidate image frame E (n) satisfies a predetermined condition regarding the distance Diff (n).
That is, it is determined using the distance threshold Thr-diff whether the distance Diff (n) to the determination reference point SP is close to or not close to the determination reference point SP.
Therefore, the control unit 30 compares the distance Diff (n) to the determination reference point SP of each candidate image frame E (n) with the distance threshold Thr-diff in Step F105, and if Diff (n) <Thr-diff. In step F106, the flag Flg (n) = 1 (close) is set. If Diff (n) <Thr-diff is not satisfied, the flag Flg (n) = 0 (not close) is set in step F107.

Next, in steps F108 and F109, the control unit 30 calculates the degree of stable presence of each candidate image frame E (n). In this case, in step F108, it is checked whether or not the flag Flg (n) = 1 for each candidate image frame E (n). If the flag Flg (n) = 1, the count value Cnt is determined in step F109. (N) is incremented (+1 addition). When the flag Flg (n) = 0, the count value Cnt (n) is not changed.
The count value Cnt (n) is a value of the stable presence as the cumulative addition value described above. That is, the candidate image frame E (n) is a value representing the frequency of the state “close” to the determination reference point SP.

Next, in Steps F111, F112, and F113, the control unit 30 determines a main subject using the degree of stable presence of each candidate image frame E (n).
In step F111, the control unit 30 checks whether the count value Cnt (n) of each candidate image frame E (n) has reached the count threshold value CTthr.

  If Cnt (n) ≧ CTthr is not satisfied, that is, if any of the count values Cnt (n) of each candidate image frame E (n) has not reached the count threshold value CTthr, the variable TSF = 0 is maintained in step F113. In step F114, the determination is not completed, and the process returns to step F101. In this case, based on the candidate image frame information input for the next frame, the processing after step F101 is executed in the same manner as described above.

In step F114, if the variable TSF = 0, it is determined that the determination of the main subject has not been completed, and the determination process is continued. If the variable TSF = 1, the determination of the main subject is completed.
If the variable TSF = 1 is detected in step F104 described above, the determination is ended as it is.
Although detailed description is omitted, in parallel with the automatic main subject determination of this example, for example, the user touches the main subject on the screen of the display unit 34 or the subject is placed on the screen at a predetermined position and the shutter is moved. The main subject may be selected as an operation such as half-pressing the button. When the user performs such a designation operation during the execution of the process of FIG. 9, it is preferable to give priority to the user operation. Therefore, when the main subject is set as such a manual operation, the variable TSF = 1 is set. In this case, the processing in FIG. 9 may be terminated (end of interruption) based on the determination in steps F104 and F114.

  The main subject determination based on the stable presence level is determined with a certain length of time. Therefore, if the processing on the candidate image frame information for a certain amount of time (number of frames) is not performed, the determination ends in step F114 as described above. Without returning to step F101, the process is repeated.

Here, for example, as shown in FIG. 8, a certain candidate image frame E <b> 3 may be discontinuous, but exists at a position close to the determination reference point SP on the captured image at a high frequency among a plurality of frames. Suppose there is a situation. Then, as time progresses, the opportunity for incrementing the count value Cnt3 of the candidate image frame E3 in step F109 occurs, and the count value Cnt3 advances earlier than the count values Cnt1 and Cnt2.
Then, at a certain time, the count value Cnt3 first reaches the count threshold value CTthr.
In such a case, the control unit 30 advances the process from step F111 to step F112.
In step F112, the control unit 30 determines that the candidate image frame E (n) whose count value Cnt (n) has reached the count threshold value CTthr is a main subject, and performs main subject setting. The variable TSF = 1 is set.

In this case, the determination is ended in step F114. That is, for example, the candidate image frame E3 is set as the main subject, and the main subject determination process in FIG. 9 is completed.
This processing example 1 is continued until the variable TSF = 1, but in practice, it is appropriate to set a predetermined time limit. That is, if the main subject cannot be determined even after a predetermined time has elapsed since the processing start time of FIG. 9, the processing is terminated with no main subject.

According to the above processing example 1, the imaging apparatus 10 is held (following the subject) so that the subject that the user mainly wants to image is as close to the determination reference point SP as possible in the center of the image. It will be automatically determined as the main subject.
In particular, in Processing Example 1, the stable presence level is determined based on the accumulated time in a state “close” to the determination reference point SP. In the case of a moving subject or a fast-moving subject such as an animal, it may be difficult for the photographer to continuously capture the main subject to be captured in the center of the image for a certain amount of time (for example, even for several seconds). Also, depending on the imaging skill of the photographer, it may be difficult to maintain the subject that is desired to be taken at the center of the image due to severe camera shake. Even in such a case, the main subject can be determined relatively quickly by using the accumulated time.
Therefore, it is suitable for a fast moving subject or a relatively unaccustomed user.

In addition, the main subject determination process is not necessarily performed for a certain period of time, and is terminated when the main subject is determined. Therefore, depending on the subject and the skill of the photographer, the main subject determination can be performed quickly. There are also advantages.

[4-2: Processing Example 2 (Type I; Judgment Reference Point; First Arrival Determination; Continuing Existence)]

Subsequently, process example 2 is also a specific example of process type I.
Then, as the position state of the candidate image frame, a distance from the set determination reference point is obtained.
Further, as the degree of stable presence for each candidate image frame, duration information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is continuously calculated.
In addition, a candidate image whose stable presence level reaches the predetermined value earliest from the start of main subject determination is determined as a main subject.
The difference from Processing Example 1 is that the stable presence level is set as the duration of the state close to the determination reference point SP. A candidate image frame that has been in a state of being close to the determination reference point SP for a predetermined time (predetermined frame) or longer is determined as a main subject.

FIG. 10 shows processing example 2. The same processing as that in FIG. 10 is assigned the same step number to avoid redundant description.
In addition, in processing examples 3 to 12 described later, in each flowchart, the already-described processing is denoted by the same step number and description thereof is omitted.
FIG. 10 is an example in which step F110 is added to the processing of FIG.

In steps F108, F109, and F110, the control unit 30 calculates the degree of stable presence of each candidate image frame E (n). In this case, in step F108, it is checked whether or not the flag Flg (n) = 1 for each candidate image frame E (n). If the flag Flg (n) = 1, the count value Cnt is determined in step F109. (N) is incremented (+1 addition). When the flag Flg (n) = 0, the process proceeds to step F110 and the count value Cnt (n) is cleared.
Similar to the processing example 1, the count value Cnt (n) is a value of the stable presence level. That is, the count value is added in response to detection of a state in which the candidate image frame E (n) is “close” to the determination reference point SP. However, in this processing example 2, the count value Cnt (n) is cleared in step F110.

This means that once it is moved away from the determination reference point SP, the value of the stable presence level is returned to zero.
For example, even if a certain candidate image frame is close to the determination reference point SP, if it is far from a certain frame, the corresponding count value Cnt (n) is reset.

Therefore, each candidate image frame E (n) has a stable presence value (count value Cnt) as long as the subject is continuously taken close to the determination reference point SP (within the distance threshold Thr-diff). (N)) goes up. That is, in this processing example 2, the count value Cnt (n) indicates the “duration” in a state close to the determination reference point SP.
In the case where the count value Cnt (n) reaches the count threshold value CTthr in step F111, a certain candidate image frame continues to the determination reference point SP by the number of frames (time) corresponding to the count threshold value CTthr. This is the case when the condition of being close is continued.

In such a case, the control unit 30 advances the process from step F111 to F112, and the control unit 30 determines that the candidate image frame E (n) whose count value Cnt (n) has reached the count threshold value CTthr is the main subject, Set the main subject. The variable TSF = 1 is set.
The processing of other steps is the same as that in FIG.
In this processing example 2 as well, when the main subject cannot be determined even after a predetermined time limit has elapsed since the processing start time, the processing may be terminated with no main subject.

According to the above processing example 2, the subject that the user mainly wants to image is held in the state as close as possible to the determination reference point SP such as the center of the image (following the subject). It is automatically determined as a subject.
In particular, in Processing Example 2, it is evaluated whether or not the existence is stable according to the duration time. In this case, when a subject with little motion is targeted, the subject subject can be easily captured continuously in the center of the image or the like, so that there is a high possibility that the main subject desired by the user can be accurately set. Further, depending on the skill of the photographer, the subject that the user wants to take can be continuously maintained at the center of the screen or the like (position close to the determination reference point SP), so there is a high probability that the photographer's desired subject can be accurately determined as the main subject. . That is, it is possible to increase the possibility that the subject desired by the user can be determined as the main subject depending on the advanced imaging skill and the subject.
In addition, the main subject determination process is not necessarily performed for a certain period of time, and is terminated when the main subject is determined. Therefore, depending on the subject and the skill of the photographer, the main subject determination can be performed quickly. .

[4-3: Processing Example 3 (Type I; Judgment Reference Point; First Arrival Determination; Cumulative Existence; Extended Example)]

The processing example 3 is also a specific example of the processing type I described above. The processing example 3 is an example in which the processing example 1 is developed.
As the position state of the candidate image frame, a distance from the set determination reference point is obtained.
As the degree of stable presence for each candidate image frame, cumulative time information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is calculated.
A candidate image whose stable presence level reaches a predetermined value earliest from the start of main subject determination is determined as a main subject.
The above is the same as the processing example 1, but the processing example 3 can give a weight to the calculation of the stable presence level. In addition, a corresponding process is added when the degree of stable presence of a plurality of candidate image frames reaches a predetermined value at the same time, that is, when a plurality of candidate image frames are determined to be main subjects.

FIG. 11 shows a processing example 3. Steps F100 to F107 are the same as those in FIG.
In steps F108, F121, F122, and F123, the control unit 30 calculates the degree of stable presence of each candidate image frame E (n).
In this case, in step F108, it is confirmed whether or not the flag Flg (n) = 1 for each candidate image frame E (n).
If the flag Flg (n) = 1, the control unit 30 proceeds to step F121 and substitutes the variable set value A for the count variable OFST. The variable set value A is a value set as A> 0.
If the flag Flg (n) = 0, the control unit 30 proceeds to step F122 and substitutes the variable set value B for the count variable OFST. The variable set value B is a value set as A ≦ 0.
In any case, the control unit 30 determines in step F123 the count value Cnt (n) that is the value of the stable presence level.
Cnt (n) = Cnt (n) + OFST
Perform the operation.
Also, an average distance Ave-Diff (n) is calculated for each candidate image frame E (n).

Since the variable set value A> 0, if the flag Flg (n) = 1 in step F108, the value of the stable presence level (count value Cnt (n)) increases.
When the count value Cnt (n) reaches the count threshold value CTthr in step F111, a certain candidate image frame may be discontinuous, but is close to the determination reference point SP at a high frequency among a plurality of frames. This is when the condition is satisfied.

In such a case, the control unit 30 advances the process from step F111 to F112, and the control unit 30 determines that the candidate image frame E (n) whose count value Cnt (n) has reached the count threshold value CTthr is the main subject, Set the main subject. The variable TSF = 1 is set.
The processing in steps F113 and F114 is the same as that in FIG.
In this processing example 3 as well, if the main subject cannot be determined even after a predetermined time limit has elapsed since the processing start time, the processing may be terminated with no main subject.

The meanings of the processes in steps F108, F121, F122, and F123 are as follows.
The variable setting values A and B may be fixedly set by the control unit 30, or may be set based on a user instruction, an application program, or the like. Alternatively, it may be changed according to the progress of the number of frames or the elapsed time during the main subject determination. That is, the values of the variable setting values A and B can be set externally in view of the main subject determination process.
Hereinafter, setting examples of various variable setting values A and B will be described as (ex1) to (ex7).

(Ex1): When the variable setting values A = 1 and B = 0 are fixedly set In this way, the processing in FIG. 11 is the same as the processing in FIG. That is, if the flag Flg (n) = 1 in step F108, the count value Cnt (n) is incremented, and if the flag Flg (n) = 0, the count value Cnt (n) is kept without being incremented.

(Ex2): Variable setting value A = 2 (or 3 or more), B = 0 fixedly set For example, normally (ex1) above, but variable according to user instruction or application program instruction Increase the setting value A. In this case, the time for the count value Cnt (n) to reach the count threshold value CTthr can be shortened. Therefore, this setting is suitable when it is desired to finish main subject determination more quickly than usual, or when it is desired to increase the response speed of main subject determination because the subject is subject to much movement.

(Ex3): In the case where the variable setting values A = 1 and B = −1 are fixedly set. Increment is normally performed, but if the flag Flg (n) = 0, the count value Cnt (n) is decremented. It is.
That is, when a certain candidate image frame is separated from the determination reference point SP (when the flag Flg (n) = 0), the process of reducing the stable presence level is performed.
This is based on the concept that importance is placed on a subject that is maintained as close to the determination reference point SP as possible, and is similar to the above-described processing example 2, but is not limited to the duration time, and is only an accumulated time. It will be to judge. That is, the idea of the stable presence level of the processing example 1 is brought close to the idea of the processing example 2.

(Ex4): When the variable setting values A = 1 and B = −2 (or −3 or less) are fixedly set Similar to (ex3) above, but a candidate image frame is determined from the determination reference point SP. This is an example of increasing the degree of inhibition of the stable presence level when the separated flag Flg (n) = 0. In other words, the importance of determining that the subject is close to the determination reference point SP to the main subject determination is increased.

(Ex5): In the case where the variable setting value A = 2 (or 3 or more) and B = -2 (or -3 or less) are set in a fixed manner Processing in which the ideas of (ex2) and (ex4) above are applied in combination An example. The importance of determining that the subject is close to the determination reference point SP to the main subject determination is increased, and the determination responsiveness is improved.

(Ex6): The variable set value A is changed during the determination period.
For example, the determination period in which the process of FIG. 11 is executed is set to a first period, a second period,.
Then, the variable set value A is changed such that A = 1 in the first period, A = 2 in the second period, A = 3 in the third period, and the like.
This increases the value of being closer to the determination reference point SP as the second half is reached.
In general, when a photographer holds a camera aiming at a subject, the subject who wants to play a leading role is not initially captured at a desired position such as the center of the image, and the photographer gradually adjusts the direction of the camera. Considering this, at first, the subject that the photographer has in mind as the “main subject” is gradually captured at the center of the image as time advances.
Therefore, increasing the value of being closer to the determination reference point SP as time progresses during execution of the main subject determination process increases the possibility that the main subject can be determined in accordance with the photographer's idea. Become.

In (ex6), the setting of the divided period and the A value in each divided period can be considered variously. Conversely, depending on the imaging circumstances and environment, the variable set value A is changed so as to be gradually reduced, such as A = 3 in the first period, A = 2 in the second period, A = 1 in the third period, and the like. An example of increasing the value of being close to the determination reference point SP at the initial stage of determination is also conceivable.
For example, the main subject determination is initially performed at the timing of starting the motion of a stationary subject.
Further, the user may be able to select the setting of the division period and how the variable setting value A is changed.
As for the variable set value B in the case of (ex6), any of the above (ex1) to (ex5) or (ex7) described below can be combined.

(Ex7): The variable set value B is changed during the determination period.
For example, the determination period in which the process of FIG. 11 is executed is set to a first period, a second period,.
Then, the variable set value A is changed such that B = 1 in the first period, B = 2 in the second period, B = 3 in the third period, and the like.
This means that the importance (unstable presence) of being away from the determination reference point SP is heavier toward the second half, and the concept is the same as (ex6) above. That is, it is an idea that a subject that the photographer does not consider as a main subject is likely to move away from the determination reference point SP as the determination later. This also increases the possibility of determining the main subject that matches the photographer's idea.

Note that in this (ex7), the setting of the divided period and the B value in each divided period can be considered variously. Depending on the imaging circumstances and environment, the variable setting value B is gradually changed to be small, such as B = 3 in the first period, B = 2 in the second period, B = 1 in the third period, etc. An example of increasing the value of being close to the determination reference point SP is also conceivable.
Also, the user may be able to select the setting of the division period and how to change the variable setting value B.
For the variable setting value A in the case of (ex7), any of the above (ex1) to (ex6) can be combined.

Although the above is an example, the main subject determination according to the purpose and the situation can be performed by setting the variable setting values A and B as described above.
In the processing example 3, as in the processing example 1, the main subject determination is performed using the count value Cnt (n) as the accumulated time information close to the determination reference point SP. The main subject is added to or subtracted from the count value Cnt (n). The value is determined by the variable setting values A and B. For this reason, the count value Cnt (n) is not the cumulative “time” itself, but a time that does not meet the condition is subtracted, or the weight of the condition match changes depending on the divided period.
As described above, the count value Cnt (n) may not represent the accumulated time of “time (number of frames)” that meets the condition of being close to the determination reference point SP. In the above, “accumulated time” is used as a concept including the accumulated time (or the number of frames) after adding the subtraction and weighting in this way.

By the way, the idea of using the variable set value A can be applied to the above-described processing example 2.
That is, when the flag Flg (n) = 1 in FIG. 10 described above, the count value Cnt (n) is incremented in step F109.
Cnt (n) = Cnt (n) + variable setting value A
You may make it perform the operation of. Depending on the setting of the variable setting value A, it is possible to realize the processing described in the above (ex2) and (ex6).
In this case, the count value Cnt (n) may not directly represent the “duration” of the “time (number of frames)” that meets the condition of being close to the determination reference point SP. The “duration” is also used as a concept including the time (or the number of frames) after the subtraction or weighting is added.

  Next, in the processing example 3 of FIG. 11, the processing in step F112 when the count value Cnt (n) of the plurality of candidate image frames E (n) simultaneously reaches the count threshold value CTthr in step F111 will be described.

Of course, a case where the count value Cnt (n) of the plurality of candidate image frames E (n) reaches the count threshold value CTthr at the same time in step F111 at a certain time point is also conceivable.
In this case, it can be considered that a plurality of candidate image frames E (n) are both determined as main subjects, but here, an example in which only one candidate image frame is determined as a main subject will be described.

  When the stable presence degree (count value Cnt (n)) of the plurality of candidate image frames E (n) reaches the count threshold value CTthr at the same time, one candidate image frame is satisfied under the following (Condition 1) to (Condition 3). Can be considered as the main subject.

(Condition 1): A candidate image frame with the smallest average distance Ave-Diff (n) is selected.
The count value Cnt (n) corresponds to the determination results “flag Flg (n) = 1” and “flag Flg (n) = 0” in step F105 as to whether the distance Diff (n) is equal to or less than the distance threshold Thr-diff. Count up / down. Therefore, the count value Cnt (n) does not represent the absolute distance Diff (n).
Here, when the average distance Ave-Diff (n) is obtained in step F123, it can be determined which of the plurality of candidate image frames E (n) is “a state closer to the determination reference point SP”. .
In step F123, the average distance Ave-Diff (n) is
Ave-Diff (n) = (cumulative addition value of distance Diff (n)) / (number of additions of distance Diff (n))
Can be obtained.
When the stable presence degree (count value Cnt (n)) of the plurality of candidate image frames E (n) reaches the count threshold value CTthr at the same time, in step F112, the one having the smallest average distance Ave-Diff (n) is selected. What is necessary is just to determine with a main to-be-photographed object.

In addition, although it determined by average distance Ave-Diff (n), it may be determined by the cumulative addition value of distance Diff (n).
It is also conceivable to select one candidate image frame using an index such as whether the average distance was within a predetermined distance range or the average area range.
Also, the subject distance may be used. For example, the difference between the position (z value) as the subject distance of the candidate image frame E (n) in each frame and the position in the z-axis direction as the determination reference point SP is obtained. And the average value is calculated. This is an example of selecting a candidate image frame having the smallest subject distance difference average value.

(Condition 2): Select a candidate image frame whose size is within a predetermined range.
Each candidate image frame E (n) has a different size depending on the distance between the imaging device 10 and the subject and the size of the subject itself. Usually, the photographer adjusts the size of the subject on the image appropriately by adjusting the zoom to a certain extent or changing the standing position of the subject. Therefore, the subject that the photographer considers to be the main subject is often within a certain range on the image. In addition, a distant subject and a subject that the photographer has not zoomed / focused are often considerably small or too large.
Therefore, for example, a determination upper limit PXmax and a determination lower limit PXmim are set for the number of pixels PX of the candidate image frame,
PXmax ≧ PX (n) ≧ PXmim
Set the size condition.
When the degree of stable presence (count value Cnt (n)) of the plurality of candidate image frames E (n) reaches the count threshold value CTthr at the same time, it is determined in step F112 that a subject that satisfies this size size is determined as the main subject. Good.
The size condition may be PXmax ≧ PX (n) with only the upper limit, or PX (n) ≧ PXmim with only the lower limit.
The size may be determined using the width w and height h of the candidate image frame E (n) instead of the number of pixels PX.

(Condition 3): A candidate image frame within a predetermined distance range or a minimum distance at a certain time is selected.
A candidate image frame determined to be close to the determination reference point SP at a certain time point or a candidate image frame closest to the determination reference point SP is set as a main subject.
The certain time point is, for example, the last frame at the time of proceeding to Step F112, the first frame when the main subject determination process is started, or an intermediate frame.
For example, since it is aimed at a subject considered as primary photographer is, the more the last time, the subject considered photographer is the main is, it is believed that many close to the determination reference point SP. This is the same concept as the latter half weighting such as (ex6) described above.
Therefore, when the degree of stable presence (count value Cnt (n)) of the plurality of candidate image frames E (n) reaches the count threshold value CTthr at the same time, in step F112, the distance Diff (n) is minimum in the last frame. The candidate image frame E (n) may be determined as the main subject.

In the last frame, the candidate image frame E (n) in which the distance Diff (n) is within a predetermined range (that is, the distance threshold Thr-diff or less) may be determined as the main subject. However, in the case of the processing example 3, since the determination is made when the count value Cnt (n) reaches the count threshold value CTthr at the same time, the plurality of candidate image frames E (n) satisfy this condition. Although this condition cannot be used for selection, it is effective to select under this condition in the case of a process in which the main subject determination process is always executed for a certain period of time as in Process Examples 5 to 12 described later.
In addition, it may be appropriate to select a candidate image frame within the predetermined distance range or the minimum distance in the first frame, in the middle of the frame, or the like, instead of the last frame, depending on the imaging situation or the subject. is there.
Also, the subject distance may be used. For example, the difference between the position (z value) as the subject distance of the candidate image frame E (n) at a specific time, such as the last frame, from the position in the z-axis direction as the determination reference point SP is obtained, and the subject distance This is an example of selecting a candidate image frame having the smallest difference value.

  (Condition 1) to (Condition 3) have been exemplified above, but it is of course possible to select one candidate image frame as a main subject by combining (Condition 1) to (Condition 3) instead of one condition. It is also possible to select using conditions other than these.

As described above, when the control unit 30 determines in step F112 that the plurality of candidate image frames E (n) should be determined as the main subject from the degree of stable presence (count value Cnt (n)), the angle of view. One candidate image is selected using the position information or size information of the candidate image frame in the space, and the selected candidate image is set as the main subject.
This makes it possible to set one most suitable main subject.
Note that the method of selecting one of the plurality as the main subject can also be applied to the processing examples 1 and 2.

As described above, in the processing example 3 described with reference to FIG. 11, the same effects as those of the processing example 1 can be obtained, and appropriate main subject determination according to various situations can be performed by setting the variable setting values A and B.

[4-4: Process example 4 (Type I; determination reference point; first-come determination; cumulative existence; with condition determination)]

Subsequently, the processing example 4 is also a specific example of the processing type I described above. The processing example 4 is an example in which the processing example 3 is further developed.
As the position state of the candidate image frame, a distance from the set determination reference point is obtained.
As the degree of stable presence for each candidate image frame, cumulative time information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is calculated.
Then, from the start of the main subject determination, a candidate image whose stable presence level reaches the predetermined value earliest is determined as the main subject.
The above is the same as the processing example 1 and the processing example 3, but the processing example 4 is an example in which a condition other than the condition that the distance Diff (n) is equal to or less than the distance threshold Thr-diff is added to the calculation of the stable presence degree. It is.

  FIG. 12 shows a processing example 4. Steps F100 to F104, F105 to F108, F121 to F123, and F111 to F114 are the same as those in FIG. FIG. 12 is obtained by adding step F130 to FIG.

In steps F130, F105, F106, and F107, processing of the flag Flg (n) is performed.
Here, in step F130, the process branches depending on whether or not a certain condition is satisfied. If the condition is satisfied, the process proceeds to step F105, and it is determined whether or not the distance Diff (n) is equal to or smaller than the distance threshold Thr-diff. Based on the determination result, the flag Flg (n) is set in either step F106 or F107.
If it is determined in step F130 that the condition is not satisfied, the process proceeds to step F107 and the flag Flg (n) = 0 is set.
That is, the flag Flg (n) = 0 is set for the candidate image frame E (n) that does not satisfy the condition in step F130 regardless of the distance Diff (n).
The processing subsequent to the flag Flg (n) in steps F106 and F107 is the same as in FIG.

As conditions for step F130, the following examples (condition 11) to (condition 14) can be considered.
(Condition 11): The subject distance is within a predetermined range.
A difference between the position (z value) as the subject distance of the candidate image frame E (n) and the position in the z-axis direction as the determination reference point SP is obtained. It is a condition that the subject distance difference value is within a predetermined range.

(Condition 12): The size is within a predetermined range.
Each candidate image frame E (n) has a different size depending on the distance between the imaging device 10 and the subject and the size of the subject itself. Based on the same idea as (Condition 2) described in the above processing example 3, the photographer mainly adjusts the zoom to some extent on the subject to be photographed or changes his / her standing position to Since it is normal to adjust the upper size appropriately, it is estimated that an imager that is too small or too large is not considered as a main subject by the photographer.
Therefore, for example, a determination upper limit PXmax and a determination lower limit PXmim are set for the number of pixels PX of the candidate image frame,
PXmax ≧ PX (n) ≧ PXmim
Set the size condition. The size condition may be PXmax ≧ PX (n) with only the upper limit, or PX (n) ≧ PXmim with only the lower limit.
The size may be determined using the width w and height h of the candidate image frame E (n) instead of the number of pixels PX.

(Condition 13): A specific image type.
For example, it is possible to set conditions in the candidate image extraction process of the candidate detection unit 26. However, when the candidate detection unit 26 extracts candidate images in a particularly wide range of conditions, the condition as a candidate image may be narrowed down in step F130. Good.
For example, it is a condition that the image is a face, a human body, a dog, a cat, or the like as specified by the user. If it is a condition that the image is a dog image, if the type of the candidate image frame E (n) is a face image, a human body image, a cat image, or the like in step F130, it is determined that the condition is not satisfied, and the count value Cnt (n ) Is not incremented.

(Condition 14): A specific image state.
For example, the luminance information and color information of the extracted image itself are added in a predetermined range and a predetermined color. In the case of luminance information, an image that is too dark or too bright does not satisfy the condition. It is also possible to determine that the user designates a certain color and satisfies the condition if the subject is that color.
In order to determine the image state, the control unit 30 may receive candidate image data itself, or luminance information and color information from the candidate detection unit 26.

  (Condition 11) to (Condition 14) have been exemplified above. Of course, instead of one condition, (Condition 1) to (Condition 3) may be combined to select one candidate image frame as the main subject. Conditions other than these can also be set.

As described above, the control unit 30 uses the position information of the candidate image in the angle-of-view space, the size information of the candidate image, or the like for the calculation process of the stable presence level.
As a result, in the processing example 4 described with reference to FIG. 12, the same effects as those of the processing examples 1 and 3 are obtained, and the calculation of the stable presence is made more appropriate by the condition setting, and the photographer considers the main The subject can be automatically determined more accurately.

  Note that the above (condition 11) to (condition 14) may be determined in the final main subject determination in step F112. That is, in step F111, even when the count value Cnt (n) of a certain candidate image frame E (n) reaches the count threshold value CTthr, the condition is not set as a main subject. If it is not satisfied, it may be considered that the main subject is not determined. As described above, the control unit 30 may use the position information of the candidate image in the angle-of-view space, the size information of the candidate image, or the like for the main subject determination process in Step F112.

In addition, as described above, adding conditions to the calculation of the stable presence as in step F130 and adding conditions to the main subject determination in step F112 are examples of processing using the stable presence as the duration. 2 is also applicable.

[4-5: Process example 5 (Type I; determination reference point; determination for a certain period of time; cumulative existence)]

The processing example 5 is also a specific example of the processing type I, but this processing example 5 is an example in which main subject determination is performed in a certain fixed time unlike the processing examples 1 to 4 so far.
That is, the control unit 30 is a processing example in which a candidate image having the highest stable presence value in a certain main subject determination period is determined as a main subject.

As the position state of the candidate image frame, a distance from the set determination reference point is obtained.
As the degree of stable presence for each candidate image frame, cumulative time information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is calculated.
As the main subject determination, the candidate image having the highest stable presence value in the fixed main subject determination period as described above is selected.

FIG. 13 shows processing example 5.
When starting the main subject determination process, the control unit 30 first performs initial setting in step F100A. Similar to Step F100 of Processing Example 1 to Processing Example 4, the variable TSF = 0 and the count value Cnt (n) = 0. In addition, in this case, the timer TM is reset / started in order to count a fixed main subject determination period.
Steps F101 to F108 and F121 to F122 are the same as those in FIG.

In step F123A, addition / subtraction of the count value Cnt (n) is the same as step F123 in FIG. In addition, step F123A calculates an average value.
In FIG. 11, “average distance Ave-Diff (n)” is calculated, but “average value calculation” here is not limited to the average distance Ave-Diff (n), but an average within a predetermined distance range. This includes the meaning that it may be possible to calculate a value indicating whether or not, or a value indicating whether or not the average area is within a predetermined area range. As described above, when these average values finally result in the highest stable presence value (count value Cnt (n)) for a plurality of candidate image frames E (n), Can be used to select a candidate image frame E (n).

In Step F140, the control unit 30 confirms whether or not the count value of the timer TM has reached a predetermined time TMov. That is, it is a determination as to whether or not the elapsed time from the start of the main subject determination process has reached a predetermined time (a constant main subject determination period).
If the predetermined time has not been reached, the process returns to step F101 and continues.

When the count value of the timer TM reaches the predetermined time TMov, the control unit 30 proceeds to step F141 and performs main subject determination.
That is, the count value Cnt (n) for each candidate image frame E (n) at that time is confirmed, and the candidate image frame E (n) having the maximum value is set as the main subject. In response to this, the variable TSF = 1. Then, the process ends.

That is, in this processing example 5, the stable presence level calculation (processing of the count value Cnt (n)) is continued for a fixed time. The candidate image frame E (n) having the maximum count value Cnt (n) is set as the main subject after a predetermined time has elapsed.
At the time of step F141, the count values Cnt (n) of the plurality of candidate image frames E (n) may be the same number and the maximum value. In that case, one may be selected according to (Condition 1) (Condition 2) (Condition 3) and the like described in the processing example 3. When determining (Condition 1), the average value obtained in Step F123A can be used.

According to the above processing example 5, by holding the imaging device 10 (following the subject) so that the subject that the user mainly wants to image is as close to the determination reference point SP as possible in the center of the image, the subject is detected. It will be automatically determined as the main subject.
In particular, the processing example 5 determines the stable presence level based on the accumulated time of the state “close” to the determination reference point SP. In the case of a moving subject or a fast-moving subject such as an animal, it may be difficult for the photographer to continuously capture the main subject to be captured in the center of the image for a certain amount of time (for example, even for several seconds). Also, depending on the imaging skill of the photographer, it may be difficult to maintain the subject that is desired to be taken at the center of the image due to severe camera shake. Even in such a case, the main subject can be determined relatively quickly by using the accumulated time.
That is, it is suitable for a fast moving subject or a relatively unaccustomed user.

Further, the main subject determination process is a process of selecting the one having the highest degree of stable presence (count value Cnt (n)) in a fixed time. For this reason, even in a situation where the count value Cnt (n) does not progress much, there is an advantage that the main subject is determined in a certain time. For example, even when a moving subject is difficult to catch near the determination reference point SP, the main subject determination is completed in a certain time.
However, for any candidate image frame E (n), in a situation where the count-up does not progress too much, it may be better to determine that there is no corresponding than to select the main subject forcibly. Therefore, a lower limit condition may be set for the count value Cnt (n) in order to correspond to the main subject. That is, in step F141, the count value Cnt (n) selected as the maximum is compared with the lower limit value Lth, and if Cnt (n) <Lth, it may be considered that there is no main subject. Thereby, it can be avoided that the main subject is determined in a situation where the main subject does not originally exist.
Note that the lower limit value Lth may be the same value as the count threshold value CTthr in Processing Example 1 or the like, but may be a lower value. That is, by setting a value that is considered to be the minimum count value (stable presence) for setting as the main subject, for example, in the case of a subject that moves around, it is difficult to catch near the judgment reference point SP. However, the effect that the main subject determination can be completed in a certain time can be obtained.

[4-6: Processing Example 6 (Type I; Determination Reference Point; Determination for a Fixed Time; Continuing Existence)]

Process example 6 is also a specific example of process type I.
In this processing example 6, as the position state of the candidate image frame, the distance from the set determination reference point is obtained.
Further, as the degree of stable presence for each candidate image frame, duration information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is continuously calculated.
As for the main subject determination, similarly to the processing example 5, the candidate image having the highest stable presence value in the fixed main subject determination period is determined as the main subject.
The difference from the processing example 5 is that the stable presence level is set as the duration of the state close to the determination reference point SP. That is, the candidate image frame that has been in the state of being closest to the determination reference point SP for the longest time within a certain main subject determination period is determined as the main subject.

FIG. 14 shows Processing Example 6.
When starting the main subject determination process, the control unit 30 first performs initial setting in step F100B. As in Step F100A of Processing Example 5, the variable TSF = 0 and the count value Cnt (n) = 0. Further, the timer TM is reset / started in order to count a fixed main subject determination period. In addition, in this case, the duration holding variable CC (n) is initialized (= 0).

Steps F101 to F107 are the same as those in FIG.
In steps F108, F121, F123A, and F124 to F126, the count value Cnt (n) is processed as a process for calculating the degree of stable presence.
In step F108, the control unit 30 checks whether or not the flag Flg (n) = 1 for each candidate image frame E (n). A variable set value A (A> 0) is substituted into OFST. In step F123, the control unit 30 determines the count value Cnt (n) that is the value of the stable presence level.
Cnt (n) = Cnt (n) + OFST
Perform the operation. That is, the count value Cnt (n) is counted up by the A value.

On the other hand, if the flag Flg (n) = 0 in step F108, the control unit 30 proceeds to step F124 and compares the count value Cnt (n) at that time with the duration holding variable CC (n).
If CC (n)> Cnt (n), the count value Cnt (n) is cleared in step F126.
If CC (n)> Cnt (n) is not satisfied, the count value Cnt (n) at that time is substituted into the duration holding variable CC (n) in step F125, and then the count value Cnt (n) in step F126. To clear.

As in Process Examples 1 to 5, the count value Cnt (n) is a stable presence value. That is, the count value is added in response to detection of a state in which the candidate image frame E (n) is “close” to the determination reference point SP.
On the other hand, if it is determined that the candidate image frame E (n) is not “close” to the determination reference point SP (flag Flg (n) = 0), the count value Cnt (n) is cleared in step F126.
This means that the value of the stable presence level is returned to zero once it is moved away from the determination reference point SP as in the case described in the processing example 2.
Therefore, each candidate image frame E (n) has a stable presence value (count value Cnt (n)) that rises as long as the subject is continuously considered to be close to the determination reference point SP. If the continuation state disappears, the count value Cnt (n) = 0. Therefore, the count value Cnt (n) indicates the “duration” in a state close to the determination reference point SP.

In this case, the duration holding variable CC (n) holds the maximum “duration” within a certain main subject determination period.
For this reason, when the duration is interrupted, the duration holding variable CC is updated to the maximum value in steps F124 and F125. That is, if the count value Cnt (n), which is the “continuation time” at which the continuation is interrupted, is greater than the “duration” held in the duration holding variable CC, the count value Cnt (n) is held for the duration. It is assigned to variable CC (n) and updated to the maximum value.

In Step F140, the control unit 30 confirms whether or not the count value of the timer TM has reached a predetermined time TMov. If the predetermined time has not been reached, the process returns to step F101 and continues.
When the count value of the timer TM reaches the predetermined time TMov, the control unit 30 compares the count value Cnt (n) at that time with the duration holding variable CC (n) in step F127.
If CC (n)> Cnt (n) is not satisfied, the count value Cnt (n) at that time is substituted into the duration holding variable CC (n) in step F128. Steps F127 and F128 are the cases where the state close to the determination reference point SP is continued at the end of the fixed main subject determination period, and when it is the maximum value of the “duration”, This is a process of updating the duration holding variable CC (n).

Then, the control unit 30 proceeds to step F141 and performs main subject determination.
That is, the value of the duration holding variable CC (n) for each candidate image frame E (n) at that time is confirmed, and the candidate image frame E (n) having the maximum value is set as the main subject. To do. In response to this, the variable TSF = 1. Then, the process ends.

As described above, in the processing example 6, the stable presence calculation (processing of the count value Cnt (n)) is continued for a certain time. The candidate image frame E (n) having the maximum duration indicated by the duration holding variable CC (n) is set as the main subject after a predetermined time has elapsed.
Note that at the time of step F141, the duration holding variable CC (n) of the plurality of candidate image frames E (n) may have the same number and the maximum value. In that case, one may be selected according to (Condition 1) (Condition 2) (Condition 3) and the like described in the processing example 3.
In step F141, the maximum value of the duration holding variable CC (n) is compared with the lower limit Lth, and if CC (n) <Lth, it may be considered that there is no main subject. Thereby, it can be avoided that the main subject is determined in a situation where the main subject does not originally exist.

According to the above processing example 6, by holding the imaging apparatus 10 (following the subject) so that the subject that the user mainly wants to image is as close to the determination reference point SP as possible in the center of the image, the subject is detected. It will be automatically determined as the main subject.
Further, the main subject determination process is a process of selecting the one having the highest degree of stable presence (duration holding variable CC (n)) for a fixed time. For this reason, similarly to the processing example 5, even in a situation where the count value Cnt (n) does not progress so much, there is an advantage that the main subject is determined in a certain time.
Furthermore, the process example 6 evaluates a stable presence degree by duration. Similar to the processing example 2, it is possible to increase the possibility that the subject desired by the user can be determined as the main subject depending on the expert of the imaging skill and the subject.

[4-7: Processing Example 7 (Type I; Judgment Reference Area; First Arrival Determination; Cumulative / Continuous Existence)]

Processing example 7 is also a specific example of processing type I.
In this processing example 7, as the position state of the candidate image frame, the positional relationship with the set determination reference region is obtained.
In addition, as the degree of stable presence for each candidate image frame, cumulative time information (or duration information) that satisfies the condition that the position state (relationship with the determination reference region) satisfies a predetermined state is calculated.
As for the main subject determination, similarly to the first to fourth processing examples, the candidate image whose stable presence level reaches the predetermined value earliest from the start of the main subject determination is selected.

First, the positional relationship of the determination reference area will be described with reference to FIG.
FIG. 15 shows an example of the determination reference area SA.
FIG. 15A shows an example in which the square area at the center of the image is set as the determination reference area SA. Here, a situation is shown in which candidate image frame information of each of the candidate image frames E1, E2, and E3 is captured at a certain frame.
The positional relationship between each candidate image frame E (n) and the determination reference area SA is, for example,
Whether or not the center of gravity is in the judgment reference area SA, whether or not the whole is in the judgment reference area SA, whether or not at least a part is in the judgment reference area SA, and the distance from the outer edge of the judgment reference area Is within a predetermined range.

Note that the determination reference area SA is set at the center of the screen as an example.
For example, as shown in FIG. 15B, the determination reference area SA may be set at a position slightly upper left from the center. For example, when considering the composition of a still image, there are many cases where the main subject may be arranged at a position other than the center.
Alternatively, as shown in FIG. 15C, the entire image (entire frame) may be used as the determination reference area SA. In this case, the positional relationship between each candidate image frame E (n) and the determination reference area SA is “whether or not a candidate image exists in the image”.
The shape of the determination reference area SA is not limited to the square in FIG. 15A, but may be the rectangle in FIGS. 15B and 15C. Further, it may be a triangle, a pentagon or more polygon, a circle, an ellipse, an oval, an indefinite shape, or the like.
In addition, various sizes are possible.

Further, the determination reference area SA may be fixedly set in an area as shown in FIGS. 15A and 15B, for example, or may be arbitrarily designated by the user by a touch operation on the display screen.
In addition, some candidate areas for the determination reference area SA may be presented to the user on the screen of the display unit 34 so that the user can select.
Furthermore, it is also conceivable that the control unit 30 determines the optimum region in consideration of the composition and the like according to the image content, the image analysis result, etc., and automatically sets it.

That is, for the determination reference point SP,
-It is set to a predetermined fixed area centered on the center position of the image or a position shifted from the center.
The user arbitrarily designates all or part of the center position, shape, and size.
-Some candidate areas are presented to the user and set by the user selecting.
The control unit 30 determines the optimum area according to the image content and automatically variably sets it.
And so on.

The positional relationship between the candidate image frame E (n) and the determination reference area SA is obtained at the time of each frame and used for counting the stable presence level (count value Cnt (n)).
For example, as a positional relationship between each candidate image frame E (n) and the determination reference area SA, it is determined whether or not the center of gravity is in the determination reference area SA.
FIG. 16 shows the positional relationship in each frame with respect to the candidate image frames E1, E2, E3, that is, how the determination result fluctuates depending on whether the center of gravity is within or outside the determination reference area SA.
In the processing example 7, the stable presence level is an accumulated time in a state where the center of gravity is in the determination reference area SA. For this reason, the number of times determined to be within the area of the determination reference area SA is counted by the count value Cnt (n).

The main subject determination process of the control unit 30 as Process Example 7 will be described with reference to FIG. Note that FIG. 17 is an example in which the processing example 3 in FIG. 11 described above is changed to a process using the determination reference area SA.
Steps F100 to F102 and F104 are the same as those in FIG.
In the case of FIG. 17, the calculation process of the distance Diff (n) corresponding to step F103 in FIG. 11 is not necessary.

In step F150, the control unit 30 checks whether the center of gravity G of each candidate image frame E (n) is within the determination reference area SA.
Here, the origin O of the xy coordinates on the screen plane is the upper left vertex of the image as shown in FIG. 15A.
The xy coordinate values of the center of gravity G of each candidate image frame E (n) are set to (E (n) _cx, E (n) _cy).
The determination reference area SA is, for example, a rectangle, and the xy coordinate value of the upper left vertex of the determination reference area SA is (SAx, SAy), the width is SAw, and the height is SAh.
Then, in step F150, the control unit 30 determines the center of gravity G of each candidate image frame E (n).
SAx ≦ E (n) _cx ≦ (SAx + SAw)
And
SAy ≦ E (n) _cy ≦ (SAy + SAh)
It will be determined whether it is.
If this condition is satisfied, the center of gravity G of the candidate image frame E (n) is within the determination reference area SA.

  The control unit 30 performs the above determination for each candidate image frame E (n), and sets the flag Flg (n) = 1 (in the region) in step F106 if the center of gravity G is in the determination reference region SA. If the center of gravity G is not included in the determination reference area SA, the flag Flg (n) = 0 (outside the area) is set in step F107.

  Steps F108, F121, F122, F123B, F111, F112, F113, and F114 are basically the same as those in FIG. Note that step F123B differs from step F123 in FIG. 11 in that the average distance calculation is not performed. (Step F110 indicated by a broken line will be described later).

According to the seventh processing example, the stable presence degree of each candidate image frame E (n) is calculated based on the positional relationship with the determination reference area SA. The degree of stable presence is an accumulated time during which the position state (relationship with the determination reference region) satisfies the condition of a predetermined state.
Then, from the start of main subject determination, a candidate image whose stability presence (cumulative time) reaches the predetermined value earliest is determined as the main subject.
Therefore, the same effect as Processing Example 1 and Processing Example 3 can be obtained. In addition, by using the determination reference area SA instead of the determination reference point SP, the position for determining the main subject is widened, which is suitable for determining the main subject depending on the imaging environment, the subject, the imaging skill of the user, and the like. There is a case.

If the flag Flg (n) = 0 in step F108, the count value Cnt (n) may be cleared in step F110 as indicated by the broken line without executing steps F122 and F123B. .
When the process cleared in step F110 is employed, as described in process example 2, the accumulated time that satisfies the condition (in the case of FIG. 17, the condition that the count value Cnt (n) exists in the determination reference area SA). Instead, it shows the duration.
That is, when step F110 is employed, the process of FIG. 17 uses “duration” as an indicator of the stable presence level, and determines the main subject based on this.
In this case, the same effect as the processing example 2 can be obtained.

In FIG. 17, in step F150, the condition determination is performed as to whether or not the center of gravity of the candidate image frame E (n) is within the determination reference area SA. However, as described above, the entire candidate image frame E (n) In step F150, a condition whether (or a part) is in the determination reference area SA and a condition that the center of the candidate image frame E (n) is within a predetermined distance from the outer edge of the determination reference area are determined. You may make it determine.

[4-8: Process Example 8 (Type I; Determination Criteria Area; Determination for a Fixed Time; Cumulative / Continuous Existence)]

Process example 8 is also a specific example of process type I. This can be said to be a modification of Processing Example 5 to an example using the determination reference area SA.
In this processing example 8, as the position state of the candidate image frame, the positional relationship with the set determination reference area SA is obtained.
Further, as the degree of stable presence for each candidate image frame, cumulative time information (or duration information) in which the position state (relationship with the determination reference area SA) satisfies the condition of a predetermined state is calculated.
In addition, a candidate image having the highest stable presence value in a certain main subject determination period is determined as the main subject.

The main subject determination process of the control unit 30 as Process Example 8 will be described with reference to FIG. FIG. 18 is an example in which the above-described processing example 5 in FIG. 13 is changed to processing using the determination reference area SA.
Steps F100A, F101, F102, and F104 are the same as those in FIG.
In the case of FIG. 18, the process for calculating the distance Diff (n) corresponding to step F103 in FIG. 13 is not required.

In step F150, the control unit 30 checks whether the center of gravity G of each candidate image frame E (n) is within the determination reference area SA. This is the same as the processing described above with reference to FIG.
Then, for each candidate image frame E (n), the control unit 30 sets the flag Flg (n) = 1 (inside the area) in step F106 if the center of gravity G is within the determination reference area SA. If the center of gravity G is not included in the determination reference area SA, the flag Flg (n) = 0 (outside the area) is set in step F107.

  Steps F108, F121, F122, F123B, F140, and F141 are the same as those in FIG. In step F123B, the average value is not calculated, which is different from step F123A in FIG.

According to this processing example 8, the degree of stable presence of each candidate image frame E (n) is calculated based on the positional relationship with the determination reference area SA. The degree of stable presence is an accumulated time during which the position state (relationship with the determination reference region) satisfies the condition of a predetermined state.
Then, the candidate image having the maximum stable presence (cumulative time) is determined as the main subject for a fixed time from the start of the main subject determination.
Therefore, the same effect as the processing example 5 can be obtained. In addition, by using the determination reference area SA instead of the determination reference point SP, the position for determining the main subject is widened, which is suitable for determining the main subject depending on the imaging environment, the subject, the imaging skill of the user, and the like. There is a case.

Note that, as a modified example of the processing example 8, there may be an example in which the stable presence level is calculated not by the accumulated time but by the duration.
Although the flowchart in that case is omitted, step F103 in FIG. 14 may be deleted, and the process of step F150 described above may be executed instead of step F105.
Depending on the processing, the same effect as Processing Example 6 can be obtained.

In either case where the accumulated time is used or the duration is used, in step F141, the maximum value of the count value Cnt (n) is compared with the lower limit value Lth, and Cnt (n) <Lth. Then, it may be considered that there is no main subject. Thereby, it can be avoided that the main subject is determined in a situation where the main subject does not originally exist.
In FIG. 18, in step F150, a condition determination is made as to whether or not the center of gravity of the candidate image frame E (n) is within the determination reference area SA, but the entire (or part) of the candidate image frame E (n). In step F150, the determination is made in step F150 as to whether the distance between the center of the candidate image frame E (n) and the outer edge of the determination reference area is within a predetermined range. Also good.

[4-9: Treatment Example 9 (Type II; Judgment Reference Point; Average Presence)]

Processing example 9 will be described. Process example 9 is a specific example of process type II.
In this processing example 9, the distance from the set determination reference point SP is obtained as the position state of the candidate image frame.
In addition, as the degree of stable presence for each candidate image frame, an average value (average distance) of the position state (distance to the determination reference point) is calculated. The average distance is an index indicating that the position is close to the determination reference point with a high frequency in time. For example, “the value of the average distance is small” has the same meaning as “the accumulated time is long” in the above-described processing examples.
Then, the candidate image frame that satisfies the condition that the average distance is the shortest and is within a predetermined threshold is determined as the main subject.
Since the processing type is II, the main subject determination process is executed for a predetermined time or more when the buffering of the candidate image frame information is completed unless interrupted by a user operation or the like.

The main subject determination process of the control unit 30 as Process Example 9 will be described with reference to FIG.
When the main subject determination process is started, the control unit 30 first sets a variable TSF = 0 as a flag indicating whether or not a main subject has been set in step F200, and sets a count value BfC = 0.
The count value BfC is obtained by the control unit 30 acquiring candidate image frame information from the candidate detection unit 26, and the buffering times (number of frames) value of the distance Diff (n) for each candidate image frame E (n) calculated. Is a value to count.

  In step F <b> 201, the control unit 30 takes in candidate image frame information for a certain frame from the candidate detection unit 26. For example, for each candidate image frame E (n), as position information, the x value and y value as the two-dimensional (xy) coordinate values of the image data, and the width w and height h of the candidate image frame as size information. Get information. The candidate image frame information may include subject distance (z value), the number of pixels, and the like.

In step F202, the control unit 30 calculates the coordinates of the center of gravity G for each candidate image frame E (n). The process is the same as that described in step F102 of FIG.
In step F203, the control unit 30 calculates a distance Diff (n) from the centroid G of each candidate image frame E (n) to the determination reference point SP. The process is the same as that described in step F103 of FIG.

In step F204, the control unit 30 stores (buffers) the distance Diff (n) calculated in step F203 in the area Diff (n) [BfC], for example, in the internal memory area.
In step F205, the control unit 30 increments the count value BfC.
In step F206, the control unit 30 determines whether or not the count value BfC has reached the count threshold value THrTM corresponding to a certain time, and if not, returns to step F201.

The area Diff (n) [BfC] in step F204 described above means an area for storing information on the candidate image frame E (n) in the frame FR (BfC) captured the (BfC) th from the start of the main subject determination process. It is said.
For example, if candidate image frames E1, E2, and E3 exist in the frame FR (0) when the candidate image frame information of the frame FR (0) is first fetched in step F201, each candidate image frame is displayed. The distances Diff1, Diff2, and Diff3 calculated in Step F203 are stored in the areas Diff (1) [0], Diff (2) [0], and Diff (3) [0].
If there are candidate image frames E1, E2, and E3 in the frame FR (1) captured in the next step F201, the distances Diff1, Diff2, and Diff3 calculated in step F203 for each candidate image frame are the regions Diff. (1) Stored in [1], Diff (2) [1], and Diff (3) [1].

  That is, the processing of steps F201 to F205 is repeated for a certain time (THrTM), so that the distance Diff (n) of each candidate image frame E (n) for each of the frames FR (0) to (m). Will be buffered.

When the predetermined time has elapsed, the buffering is terminated, and the control unit 30 advances the process from step F206 to F207.
In step F207, the control unit 30 calculates an average distance Ave-Diff (n) for each candidate image frame E (n).
Ave-Diff (n) = (cumulative addition value of buffered distance Diff (n)) / (number of additions of distance Diff (n))

In step F208, the control unit 30 determines the minimum distance MIN [Ave-Diff (n)] among the average distances Ave-Diff (n) of the candidate image frames E (n). A candidate image frame E (n) having the smallest average distance Ave-Diff (n) is obtained.
If the minimum distance MIN [Ave-Diff (n)] is determined, the control unit 30 determines whether or not the minimum distance MIN [Ave-Diff (n)] is equal to or smaller than the distance threshold Thr-diff in step F209. .

If the minimum distance MIN [Ave-Diff (n)] is less than or equal to the distance threshold Thr-diff, the control unit 30 advances the process to Step F210 and selects the candidate image having the minimum distance MIN [Ave-Diff (n)]. The frame E (n) is determined as the main subject, and the main subject is set. The variable TSF = 1 is set.
If the minimum distance MIN [Ave-Diff (n)] is not less than or equal to the distance threshold Thr-diff, the control unit 30 sets the variable TSF = 0 in step F211. In this case, the result is that the main subject is not applicable.
Note that if the process of determining the main subject without fail at least when the candidate image frame E (n) exists, the candidate image frame E () having the minimum distance MIN [Ave-Diff (n)] is eliminated by eliminating step F209. n) may be determined as a main subject as it is.

According to the above processing example 9, by holding the imaging apparatus 10 (following the subject) such that the subject that the user mainly wants to capture is as close to the determination reference point SP as the center of the image as much as possible, It will be automatically determined as the main subject.
Since the candidate image frame information is acquired for a certain period of time, the same effects as those of the processing example 5 can be obtained.

  Of course, also in this processing example 9, a plurality of candidate image frames may correspond to the minimum distance MIN [Ave-Diff (n)]. When one candidate image frame E (n) is set as the main subject, the candidate image frame E as the main subject is obtained by the method using (condition 1) to (condition 3) described in the explanation of the processing example 3. (N) may be selected.

Further, in the description of FIG. 5 regarding the processing type II, it has been described that the step F21 may be performed after it is determined that the accumulation is completed in the step F22.
When processing corresponding to this is performed, the processing in steps F202 and F203 in FIG. 19 may be executed when the processing proceeds from step F206 to F207.
That is, in step F204, the candidate image frame information itself is buffered, and the distance Diff (n) of each candidate image frame E (n) in each frame is set to the buffered candidate image frame information after completion of buffering. It is an example to calculate using.

[4-10: Processing Example 10 (Type II; Judgment Reference Point; Cumulative Existence)]

Processing example 10 is also an example of processing type II, and uses a determination reference point. As the degree of stable presence for each candidate image frame, cumulative time information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is calculated.
Then, the candidate image having the highest stable presence value during the buffering period is determined as the main subject.

FIG. 20 shows main subject determination processing of the control unit 30 as Processing Example 10.
When the main subject determination process is started, the control unit 30 first sets the variable TSF = 0 and the count value BfC = 0 in step F200A. In addition, a variable L = 0 for loop control is initialized, and a count value Cnt (n) = 0 corresponding to the degree of stable presence is initialized.

Steps F201 to F206 are the same as those in FIG. 19, and capture of candidate image frame information and buffering of the distance Diff (n) of each candidate image frame E (n) are performed for a certain period.
When the count value BfC as a timer reaches the count threshold value THrTM in step F206, the buffering is terminated, and the control unit 30 advances the process from step F206 to F210.

In step F210, the control unit 30 reads the distance Diff (n) buffered in the region Diff (n) [L].
Since the variable L = 0 at the beginning, each candidate image frame E (buffered in the area Diff (n) [0] at the time when the candidate image frame information of the frame FR (0) is first captured in step F201. The distance Diff (n) of n) is read out.

In Steps F211, F212, and F213, the control unit 30 sets a distance threshold Thr-diff to determine whether or not the distance Diff (n) of each candidate image frame E (n) is close to or close to a predetermined condition, that is, the determination reference point SP. Use to determine.
If Diff (n) <Thr-diff, the flag Flg (n) = 1 (close) is set in step F212. If Diff (n) <Thr-diff is not satisfied, the flag Flg (n) = 0 (not close) is set in step F213. This is the same processing as Steps F105, F106, and F107 in Processing Example 1.

In steps F214, F215, F216, and F217, the count value Cnt (n) is processed as the calculation of the stable presence level.
In step F214, the control unit 30 checks whether or not the flag Flg (n) = 1 for each candidate image frame E (n). If the flag Flg (n) = 1, the control unit 30 counts the variable in step F215. A variable set value A (A> 0) is substituted into OFST. In step F217, the control unit 30 determines the count value Cnt (n) that is the value of the stable presence level.
Cnt (n) = Cnt (n) + OFST
Perform the operation. That is, the count value Cnt (n) is counted up by the A value.
If the flag Flg (n) = 0, the control unit 30 substitutes the variable setting value B (B ≦ 0) into the count variable OFST in step F216. In step F217, the control unit 30 determines the count value Cnt (n) that is the value of the stable presence level.
Cnt (n) = Cnt (n) + OFST
Perform the operation. That is, the count value Cnt (n) is counted down (or kept) by the B value.
In step F217, the average value is also calculated.
The above is the same as steps F108, F121, and F123 in FIG. 11 (or F123A in FIG. 13).

In step F218, the control unit 30 checks whether or not the variable L has reached the buffering count value BfC. If not, the variable L is incremented in step F220, and the process returns to step F210.
Therefore, next, in step F210, the distance of each candidate image frame E (n) buffered in the area Diff (n) [1] at the time when the candidate image frame information of the frame FR (1) was fetched in step F201. Diff (n) is read out.
Then, the processes of steps F211 to F217 are performed.

The above steps F210 to F217 are repeated until the variable L = BfC in step F218. As a result, processing for the buffered distance Diff (n) is executed.
When the processing is completed, the control unit 30 proceeds to step F219 and performs main subject determination.
That is, the count value Cnt (n) for each candidate image frame E (n) at that time is confirmed, and the candidate image frame E (n) having the maximum value is set as the main subject. In response to this, the variable TSF = 1. Then, the process ends.
When the count value Cnt (n) of the plurality of candidate image frames E (n) reaches the same number and the maximum value at the time of step F219, the above-described processing example 3 (condition 1) (condition 2) (condition 3) One may be selected according to the above. When performing the determination of (Condition 1), the average value obtained in Step F127 can be used.
In step F219, the maximum value of the count value Cnt (n) is compared with the lower limit value Lth, and if Cnt (n) <Lth, the main subject may not be applicable. Thereby, it can be avoided that the main subject is determined in a situation where the main subject does not originally exist.
Also by the above processing example 10, the same effect as the processing example 9 can be obtained.

[4-11: Processing Example 11 (Type II; Judgment Reference Point; Continuing Existence)]

Processing example 11 is also an example of processing type II, and uses a determination reference point. As the degree of stable presence for each candidate image frame, duration information that satisfies the condition that the position state (distance to the determination reference point) is within a predetermined threshold is calculated.
Then, the candidate image having the highest stable presence value during the buffering period is determined as the main subject.

FIG. 21 shows main subject determination processing of the control unit 30 as Processing Example 11.
When the main subject determination process is started, the control unit 30 first sets the variable TSF = 0 and the count value BfC = 0 in step F200B. In addition, a variable L = 0 for loop control is initialized, and a count value Cnt (n) = 0 corresponding to the degree of stable presence is initialized. Further, the duration holding variable CC (n) is initialized (= 0).

Steps F201 to F206 are the same as those in FIG. 19, and capture of candidate image frame information and buffering of the distance Diff (n) of each candidate image frame E (n) are performed for a certain period.
When the count value BfC as a timer reaches the count threshold value THrTM in step F206, the buffering is terminated, and the control unit 30 advances the process from step F206 to F210.

Steps F210 to F215, F217, F218, and F220 are the same as those in FIG.
On the other hand, if the flag Flg (n) = 0 in step F214, the control unit 30 proceeds to step F224, and compares the count value Cnt (n) at that time with the duration holding variable CC (n).
If CC (n)> Cnt (n), the count value Cnt (n) is cleared in step F226.
If CC (n)> Cnt (n) is not satisfied, the count value Cnt (n) at that time is substituted into the duration holding variable CC (n) in step F225, and then the count value Cnt (n) in step F226. To clear.
Steps F224, F225, and F226 are processes having the same meaning as steps F124, F125, and F126 of FIG. 14, and are for setting the count value Cnt (n) to a value that means the duration.

In step F218, the control unit 30 checks whether or not the variable L has reached the buffering count value BfC, and if it has reached, proceeds to step F227. Steps F227 and F228 are the same processes as steps F127 and F128 of FIG.
That is, the control unit 30 compares the count value Cnt (n) at that time with the duration holding variable CC (n) in step F227. If CC (n)> Cnt (n) is not satisfied, the count value Cnt (n) at that time is substituted into the duration holding variable CC (n) in step F228.
This process is performed when a certain candidate image frame E (n) continues to be close to the determination reference point SP at the end of the buffering period, which is the maximum “duration”. If the value is a value, the duration holding variable CC (n) is updated.

Then, the control unit 30 proceeds to step F219 and performs main subject determination.
That is, the count value Cnt (n) for each candidate image frame E (n) at that time is confirmed, and the candidate image frame E (n) having the maximum value is set as the main subject. In response to this, the variable TSF = 1. Then, the process ends.
When the count value Cnt (n) of the plurality of candidate image frames E (n) reaches the same number and the maximum value at the time of step F219, the above-described processing example 3 (condition 1) (condition 2) (condition 3) One may be selected according to the above. When performing the determination of (Condition 1), the average value obtained in Step F127 can be used.
In step F219, the maximum value of the count value Cnt (n) is compared with the lower limit value Lth, and if Cnt (n) <Lth, the main subject may not be applicable. Thereby, it can be avoided that the main subject is determined in a situation where the main subject does not originally exist.
According to the above processing example 10, the same effect as the processing example 6 can be obtained.

[4-12: Treatment Example 12 (Type II; Judgment Reference Area; Average Presence)]

Process example 12 is an example of process type II, and is an example in which process example 9 is changed to a process using determination criterion area SA.
In this processing example 12, as the position state of the candidate image frame, the positional relationship with the set determination reference area SA is obtained.
Further, as the degree of stable presence for each candidate image frame, an average time (cumulative time) in which the position state (relationship with the determination reference area SA) satisfies the condition of a predetermined state is calculated.
Then, the candidate image having the highest stable presence value during the buffering period is determined as the main subject.

FIG. 22 shows Processing Example 12.
Steps F200 to F202 are the same as those in FIG.
In the buffering period, in step F230, it is confirmed whether or not the centroid G of each candidate image frame E (n) is within the determination reference area SA. This is the same processing as step F150 in FIG.

For each candidate image frame E (n), if the center of gravity G is within the determination reference area SA, the control unit 30 sets the condition satisfaction flag AR (n) = 1 in step F231.
For each candidate image frame E (n), if the center of gravity G is not within the determination reference area SA, the control unit 30 sets the condition satisfaction flag AR (n) = 0 in step F232.

In step F233, the control unit 30 stores (buffers) the condition satisfaction flag AR (n) in the area AR (n) [BfC], for example, in the internal memory area.
In step F205, the control unit 30 increments the count value BfC.
In step F206, the control unit 30 determines whether or not the count value BfC has reached the count threshold value THrTM corresponding to a certain time, and if not, returns to step F201.

The area AR (n) [BfC] in the above step F233 means an area for storing information on the candidate image frame E (n) in the frame FR (BfC) captured in the (BfC) th from the start of the main subject determination process. It is said.
For example, if candidate image frames E1, E2, and E3 exist in the frame FR (0) when the candidate image frame information of the frame FR (0) is first fetched in step F201, each candidate image frame is displayed. Are stored in the areas AR (1) [0], AR (2) [0], and AR (3) [0].
If there are candidate image frames E1, E2, and E3 in the frame FR (1) captured in step F201, the condition satisfaction flags AR1, AR2, and AR3 set in steps F230 to F232 for each candidate image frame are displayed. Are stored in the areas AR (1) [1], AR (2) [1], and AR (3) [1].

  That is, by repeating the processing of steps F201 to F206 for a certain time (THrTM), the condition satisfaction flag AR () for each candidate image frame E (n) for each of the frames FR (0) to (m). n) (information on whether or not it is in the determination reference area SA) is accumulated.

When the predetermined time has elapsed, the buffering is terminated, and the control unit 30 advances the process from step F206 to F240.
In step F240, the control unit 30 calculates the existence accuracy in the determination reference area SA for each candidate image frame E (n).
This existence accuracy is the average value Ave [AR (n)] of the condition satisfaction flag AR (n).
Average value (existence accuracy) Ave [AR (n)] is
Ave [AR (n)] = (cumulative addition value of condition satisfaction flag AR (n) buffered for candidate image frame E (n)) / (number of additions of AR (n))
And it is sufficient.

  In step F241, the control unit 30 sets the maximum value in the existence accuracy Ave [AR (n)] of each candidate image frame E (n) as the maximum accuracy MAX [Ave [AR (n)]]. That is, the existence accuracy of the candidate image frame E (n) that has been in the determination reference area SA for the longest time is selected.

  When the maximum accuracy MAX [Ave [AR (n)]] is determined, the control unit 30 determines whether or not the maximum accuracy MAX [Ave [AR (n)]] is greater than or equal to the accuracy threshold Thr-AR in step F242. judge.

If the maximum accuracy MAX [Ave [AR (n)]] is equal to or greater than the accuracy threshold Thr-AR, the control unit 30 advances the process to Step F243, and reaches the maximum accuracy MAX [Ave [AR (n)]]. Candidate image frame E (n) is determined as the main subject, and main subject setting is performed. The variable TSF = 1 is set.
If the maximum accuracy MAX [Ave [AR (n)]] is not equal to or greater than the accuracy threshold Thr-AR, the control unit 30 sets the process to variable TSF = 0 in step F244. In this case, the result is that the main subject is not applicable.
Note that if the process of determining the main subject without fail at least when the candidate image frame E (n) exists, the candidate image frame E having the maximum accuracy MAX [Ave [AR (n)]] without the step F242 is realized. (N) may be determined as a main subject as it is.

According to the above processing example 12, by holding the imaging device 10 (following the subject) so that the subject that the user mainly wants to image is as close to the determination reference point SP as possible in the center of the image, the subject is It will be automatically determined as the main subject.
Since the candidate image frame information is acquired for a certain period of time, the same effect as in Processing Example 8 can be obtained.

The existence accuracy indicating the degree of stable presence is the average value of the accumulated time existing in the determination reference area SA, but the accumulated time itself may be used.
In addition, a candidate image frame that has a maximum duration in the determination reference area SA instead of the accumulated time may be used as the main subject.
In that case, the number of consecutive frames in which the buffered condition satisfaction flag AR (n) is “1” may be counted and used as the existence accuracy.

  Of course, also in this processing example 12, a plurality of candidate image frames may correspond to the maximum accuracy MAX [Ave [AR (n)]]. When one candidate image frame E (n) is set as the main subject, the candidate image frame E as the main subject is obtained by the method using (condition 1) to (condition 3) described in the explanation of the processing example 3. (N) may be selected.

Further, in the description of FIG. 5 regarding the processing type II, it has been described that the step F21 may be performed after it is determined that the accumulation is completed in the step F22.
When processing corresponding to this is performed, the processing in steps F202 to F232 in FIG. 22 may be executed when the processing proceeds from step F206 to F240.
That is, in step F204, the candidate image frame information itself is buffered, and the condition satisfaction flag AR (n) of each candidate image frame E (n) in each frame is set after buffering is completed. It is an example calculated using image frame information.

<5. Application to Program and Computer Device>

Although the embodiments of the image processing apparatus 1 and the imaging apparatus 10 have been described above, the main subject determination process described above can be executed by hardware or can be executed by software.

The program according to the embodiment is a program that causes an arithmetic processing device such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) to execute the above-described main subject determination processing.
That is, this program acquires candidate image information indicating candidate images detected as main subject candidates for a plurality of frames of image data, and positions the candidate images indicated by the candidate image information in the angle-of-view space. The arithmetic processing unit is caused to execute a position state determination process for determining a state.
In addition, a stable presence degree calculation process for obtaining a stable presence degree in the image data over a plurality of frames of the candidate image from the position state of the candidate image in each frame determined in the position state determination process is performed on the arithmetic processing device. Let it run.
Further, the arithmetic processing device is caused to execute main subject determination processing for determining a main subject among candidate images using the stable presence level obtained in the stable presence level calculation processing.

Specifically, the program according to the embodiment may be a program that causes the arithmetic processing device to execute the processing described in FIGS. 2, 4, and 5, and further, processing examples 1 to 12.
With such a program, an apparatus for executing the main subject determination process described above can be realized using an arithmetic processing apparatus.

Such a program can be recorded in advance in an HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
Alternatively, it may be temporarily stored on a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magnet optical) disk, DVD (Digital Versatile Disc), Blu-ray disk, magnetic disk, semiconductor memory, memory card, etc. Can be stored (recorded) or permanently. Such a removable recording medium can be provided as so-called package software.
Further, such a program can be downloaded from a removable recording medium to a personal computer or the like, or downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Also, such a program is suitable for providing a wide range of image processing apparatuses according to the embodiments. For example, by downloading a program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant) or the like, the portable information processing device or the like performs main subject determination processing. The disclosed image processing apparatus can be obtained.
For example, in the computer device as shown in FIG. 23, the same processing as the main subject determination processing in the image processing device 1 or the imaging device 10 in FIG. 1 may be executed.

In FIG. 23, the CPU 71 of the computer device 70 executes various processes according to a program stored in the ROM 72 or a program loaded from the storage unit 78 to the RAM 73. The RAM 73 also appropriately stores data necessary for the CPU 71 to execute various processes.
The CPU 71, ROM 72, and RAM 73 are connected to each other via a bus 74. An input / output interface 75 is also connected to the bus 74.

  The input / output interface 75 includes an input unit 76 such as a keyboard and a mouse, a display such as a CRT (Cathode Ray Tube) or LCD, or an organic EL panel, an output unit 77 such as a speaker, and a hard disk. A communication unit 79 including a storage unit 78 and a modem is connected. The communication unit 79 performs communication processing via a network including the Internet.

  A drive 80 is connected to the input / output interface 75 as necessary, and a removable medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is Installed in the storage unit 78 as necessary.

  When the main subject determination process described above is executed by software, a program constituting the software is installed from a network or a recording medium.

  For example, as shown in FIG. 23, the recording medium is distributed to distribute a program to a user separately from the apparatus main body, and includes a magnetic disk (including a flexible disk) on which a program is recorded, an optical disk ( Blu-ray disc (including Blu-ray Disc (registered trademark), CD-ROM (compact disc-read only memory), DVD (digital versatile disc)), magneto-optical disc (including MD (mini disc)), or semiconductor memory Or a hard disk included in the storage unit 78, or a ROM 72 that stores a program and is distributed to the user in a state of being incorporated in the apparatus main body in advance.

In such a computer device 70, when the moving image data is input by the reception operation by the communication unit 79 or the reproduction operation by the drive 80 (removable medium 81) or the recording unit 78, the CPU 71 is based on the program. The functions of the candidate image detection unit (3, 26) and the main subject determination unit (2, 31) described above are executed. That is, the main subject determination can be performed in the input image data by executing the processing as the processing examples 1 to 12 in FIGS.

<6. Modification>

Various modifications can be considered for the above-described embodiment.
More specific examples of the main subject determination processing are assumed.
For example, the centroid G is used for the distance Diff (n) between the candidate image frame E (n) and the determination reference point SP, and the positional relationship with the determination reference area SA. For example, the vertex position of the candidate image frame E (n) may be used.
The shape of the candidate image frame is not limited to a square or a rectangle, but may be a circle, an ellipse, a polygon, an indefinite shape, or the like. Depending on the shape of the candidate image frame, the center position or the end point position may be used for calculating the distance Diff.

Further, in the embodiment, the position state of the candidate image in the angle of view space has been described mainly as a positional relationship with the determination reference point SP and the determination reference area SA in the angle of view space as a two-dimensional space. The positional relationship with the determination reference point SP and the determination reference area SA in a three-dimensional space coordinate space (xyz coordinate space) in which the subject distance is added to the two-dimensional.
Further, the angle of view space is considered as a one-dimensional space, and the positional relationship (for example, distance) of the candidate image with the determination reference point SP or the determination reference region SA is determined as the position state of the candidate image in the angle of view space. It is good. Of course, similarly, only the y-coordinate value or only the z-coordinate value may be set as a positional relationship with the determination reference point SP or the determination reference area SA.

In addition to the accumulated time and duration of condition matching, the average presence value, and the average presence accuracy, there are various indexes that indicate the stable presence level. For example, it may be an average value of the duration of matching conditions.
Furthermore, the degree of variation in the size (number of pixels) of the candidate image frame in each frame may be used as an indicator of the stable presence level.

  Further, the result of the main subject determination process may be added as metadata to still image data or moving image data that has been captured and recorded thereafter. That is, information indicating the main subject is added to a still image file or the like.

Further, the main subject designation operation by the operation of the photographer may be enabled while the through image is displayed and the main subject determination process is simultaneously performed.
When an operation for specifying the main subject is performed, it is preferable to set the variable TSF = 1 to thereby interrupt the main subject determination process by the control unit 30.

  In the embodiment, the process of determining the main subject mainly assuming still image capturing has been described. However, the image is captured even during standby for moving image capturing or during execution of moving image capturing and recording. The processing of the above-described embodiment can be applied as processing for performing main subject determination from a plurality of frames.

In addition, this technique can also take the following structures.
(1) a candidate detection unit that detects candidate images that are candidates for main subjects for a plurality of frames of image data;
A main subject determination unit that determines a stable presence in image data over a plurality of frames for the candidate image detected by the candidate detection unit, and uses the stable presence to determine a main subject among the candidate images;
An image processing apparatus.
(2) The main subject determination unit
A position state determination process for determining a position state in the angle-of-view space for the candidate image;
From the position state of the candidate image in each frame determined in the position state determination process, a stable presence degree calculation process for obtaining a stable presence degree in the image data over a plurality of frames for the candidate image;
Main subject determination processing for determining a main subject among candidate images using the stable presence level obtained in the stable presence level calculation processing;
The image processing apparatus according to (1), wherein:
(3) The image processing device according to (2), wherein the main subject determination unit determines the distance of the candidate image with respect to the determination reference point set in the angle of view space as the position state.
(4) The image processing apparatus according to (3), wherein the main subject determination unit sets the determination reference point according to an operation input.
(5) The image processing device according to (2), wherein the main subject determination unit determines the positional relationship of the candidate image with respect to the determination reference region set in the angle of view space as the position state.
(6) The image processing device according to (5), wherein the main subject determination unit sets the determination reference region according to an operation input.
(7) The image processing device according to any one of (2) to (6), wherein the main subject determination unit calculates cumulative time information in which the position state satisfies a predetermined condition as the degree of stable presence.
(8) The image according to any one of (2) to (6), wherein the main subject determination unit calculates duration information in which the position state continues and satisfies a predetermined condition as the degree of stable presence. Processing equipment.
(9) The main subject determination unit according to any one of (1) to (8), wherein the main subject determination unit determines that a candidate image whose stability presence has reached a predetermined value earliest from the start of main subject determination is the main subject. Image processing apparatus.
(10) The image processing according to any one of (1) to (8), wherein the main subject determination unit determines, as a main subject, a candidate image having the highest stable presence value during the main subject determination period. apparatus.
(11) The main subject determination unit uses the position information of the candidate image in the angle-of-view space or the size information of the candidate image in the stable degree calculation process or the main subject determination process (1). The image processing device according to any one of (10) to (10).
(12) When the plurality of candidate images are determined as the main subject based on the degree of stable presence, the main subject determination unit uses the position information or size information of the candidate images in the angle of view space as one candidate. The image processing apparatus according to any one of (1) to (11), wherein an image is selected and the selected candidate image is a main subject.
(13) The image processing device according to any one of (1) to (12), wherein the candidate detection unit detects a human or animal face or body image as a candidate image that is a candidate for a main subject.

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 2 Main subject determination part, 3 Candidate detection part, 10 Imaging device, 11 Optical system, 12 Imager, 13 Optical system drive part, 14 Sensor part, 15 Recording part, 16 Communication part, 20 Digital signal processing part , 21 Pre-processing unit, 22 Synchronization unit, 23 YC generation unit, 24 Resolution conversion unit, 25 Codec unit, 26 Candidate detection unit, 30 Control unit, 31 Main subject determination unit, 32 UI controller, 33 User interface, 34 Display Part, 35 operation part, 70 computer device, 71 CPU

Claims (29)

A candidate detection unit that detects candidate images that are candidates for main subjects for a plurality of frames of image data;
A position state determination process for determining a positional relationship of the candidate image with respect to a determination reference region set in the angle of view space according to an operation input as a position state in the angle of view space of the candidate image detected by the candidate detection unit; ,
From the positional relationship of the candidate images in each frame determined in the position state determination process, a stable presence calculation process for obtaining a stable presence in the image data over a plurality of frames for the candidate image,
Main subject determination processing for determining a candidate image having the highest value of the stable presence level as a main subject in the main subject determination period among the candidate images using the stable presence level obtained in the stable presence level calculation processing. A main subject determination unit to perform;
The image processing apparatus Ru comprising a. A candidate detection unit that detects candidate images that are candidates for main subjects for a plurality of frames of image data;
A position state determination process for determining a position state in the angle-of-view space for the candidate image detected by the candidate detection unit;
From the position state of the candidate image in each frame determined in the position state determination process, a stable presence degree calculation process for obtaining a stable presence degree in the image data over a plurality of frames for the candidate image;
Main subject determination processing for determining a candidate image having the highest value of the stable presence level as a main subject in the main subject determination period among the candidate images using the stable presence level obtained in the stable presence level calculation processing. A main subject determination unit to perform,
With
In the stable presence level calculation process, the stable existence reflecting the determination result of the position state so that the value of the determination result of the position state is greater in the final period than in the initial period in the main subject determination period. An image processing apparatus that calculates a degree value . A candidate detection unit that detects candidate images that are candidates for main subjects for a plurality of frames of image data;
A position state determination process for determining a position state in the angle-of-view space for the candidate image detected by the candidate detection unit;
From the position state of the candidate image in each frame determined in the position state determination process, a stable presence degree calculation process for obtaining a stable presence degree in the image data over a plurality of frames for the candidate image;
Main subject determination processing for determining a candidate image having the highest value of the stable presence level as a main subject in the main subject determination period among the candidate images using the stable presence level obtained in the stable presence level calculation processing. A main subject determination unit to perform,
With
In the position state determination processing, as the position state, an image processing device that determines the positional relationship of candidate images according to whether or not a predetermined portion of the main subject is in a determination reference region set in an angle of view space . For each of a plurality of frames of image data, detect candidate images that are candidates for the main subject,
For the detected candidate image, as a position state in the angle of view space, determine the positional relationship of the candidate image with respect to the determination reference region set according to the operation input in the angle of view space,
From the determined positional relationship of the candidate images in each frame, the degree of stable presence in the image data over a plurality of frames for the candidate images is obtained,
Using the obtained stable presence level, the candidate image having the highest value of the stable presence level in the main subject determination period among the candidate images is determined as the main subject.
Image processing method. For each of a plurality of frames of image data, detect candidate images that are candidates for the main subject,
For the detected candidate image, determine the position state in the angle of view space,
From the position state of the candidate image in each frame determined as described above, the stable presence in the image data over a plurality of frames for the candidate image is obtained,
Using the obtained stable presence level, a main subject determination process for determining a candidate image having the highest value of the stable presence level as a main subject in a main subject determination period among candidate images,
The value of the stable presence level that reflects the determination result of the position state is calculated so that the value of the determination result of the position state is greater in the final period than in the initial period in the main subject determination period.
Image processing method. For each of a plurality of frames of image data, detect candidate images that are candidates for the main subject,
For the detected candidate images, the positional relationship of the candidate images is determined according to whether or not the predetermined part of the main subject is in the determination reference area set in the angle of view space as the position state in the angle of view space. And
From the determined positional relationship of the candidate images in each frame, the degree of stable presence in the image data over a plurality of frames for the candidate images is obtained,
Using the obtained stable presence level, a main subject determination process for determining a candidate image having the highest value of the stable presence level as a main subject in the main subject determination period among the candidate images is performed.
Image processing method. The position of the candidate image with respect to the determination reference region set in the angle of view space according to the operation input as the position state in the angle of view space of the candidate image detected as the candidate for the main subject for each of the plurality of frames of the image data A position state determination process for determining a relationship;
From the positional relationship of the candidate images in each frame determined in the position state determination process, a stable presence calculation process for obtaining a stable presence in the image data over a plurality of frames for the candidate image,
A main subject determination process for determining a candidate image having the highest value of the stable presence level as a main subject in a main subject determination period among the candidate images using the stable presence level obtained in the stable presence level calculation process;
A program that causes an arithmetic processing unit to execute. A position state determination process for determining a position state in the angle-of-view space for each candidate image detected as a candidate for a main subject for a plurality of frames of image data;
From the position state of the candidate image in each frame determined in the position state determination process, a stable presence degree calculation process for obtaining a stable presence degree in the image data over a plurality of frames for the candidate image;
A main subject determination process for determining a candidate image having the highest value of the stable presence level as a main subject in a main subject determination period among the candidate images using the stable presence level obtained in the stable presence level calculation process ;
Processing for calculating the value of the stable presence that reflects the determination result of the position state so that the value of the determination result of the position state is greater in the final period than in the initial period in the main subject determination period. When,
A program that causes an arithmetic processing unit to execute. For a plurality of frames of image data, the candidate image detected as a candidate for the main subject respectively, the position state in the angle of view space as the position state in the angle of view space includes a predetermined location of the main subject in the determination reference area set in the angle of view space. Position state determination processing for determining the positional relationship of candidate images according to whether or not
From the positional relationship of the candidate images in each frame determined in the position state determination process, a stable presence calculation process for obtaining a stable presence in the image data over a plurality of frames for the candidate image,
A main subject determination process for determining a candidate image having the highest value of the stable presence level as a main subject in a main subject determination period among the candidate images using the stable presence level obtained in the stable presence level calculation process;
A program that causes an arithmetic processing unit to execute. A candidate detection unit that detects candidate images that are candidates for main subjects from each frame of image data including a plurality of frames;
A position state determination process for determining a positional relationship of the candidate image with respect to a determination reference region set in the angle of view space according to an operation input as a position state in the angle of view space of the candidate image detected by the candidate detection unit ; ,
From the positional relationship of the candidate images in each frame determined in the position state determination process, a stable presence calculation process for obtaining a stable presence on the time axis in the image data over a plurality of frames for the candidate image,
When a plurality of candidate images are present by determining a main subject among the candidate images using the stable presence obtained in the stable presence degree calculation process, one main subject is selected from the plurality of candidate images. A main subject determination unit for determining
An image processing apparatus. A candidate detection unit that detects candidate images that are candidates for main subjects from each frame of image data including a plurality of frames;
  A position state determination process for determining a position state in the angle-of-view space for the candidate image detected by the candidate detection unit;
From the position state of the candidate image in each frame determined in the position state determination process, a stable presence degree calculation process for obtaining a stable presence degree on the time axis in the image data over a plurality of frames for the candidate image,
When a plurality of candidate images are present by determining a main subject among the candidate images using the stable presence obtained in the stable presence degree calculation process, one main subject is selected from the plurality of candidate images. A main subject determination unit for determining
  With
In the stable presence level calculation process, the stable presence level reflecting the determination result of the position state so that the value of the determination result of the position state is greater in the final period than in the initial period in the main subject determination period. Calculate the value of
Image processing device. A candidate detection unit that detects candidate images that are candidates for main subjects from each frame of image data including a plurality of frames;
  A position state determination process for determining a position state in the angle-of-view space for the candidate image detected by the candidate detection unit;
From the position state of the candidate image in each frame determined in the position state determination process, a stable presence degree calculation process for obtaining a stable presence degree on the time axis in the image data over a plurality of frames for the candidate image,
When a plurality of candidate images are present by determining a main subject among the candidate images using the stable presence obtained in the stable presence degree calculation process, one main subject is selected from the plurality of candidate images. A main subject determination unit for performing a main subject determination process for determining
With
In the position state determination process, as the position state, the positional relationship of the candidate images is determined depending on whether or not a predetermined portion of the main subject is in the determination reference region set in the angle of view space.
Image processing device. Detect candidate images that are candidates for the main subject from each frame of the image data consisting of a plurality of frames,
For the detected candidate image, as a position state in the angle of view space, determine the positional relationship of the candidate image with respect to the determination reference region set according to the operation input in the angle of view space,
From the above-determined positional relationship of the candidate images in each frame, obtain the stable presence on the time axis in the image data over a plurality of frames for the candidate images,
By determining the main subject in the candidate images using the obtained stable presence degree, when there are a plurality of candidate images, one main subject is determined from the plurality of candidate images.
Image processing method. Detect candidate images that are candidates for the main subject from each frame of the image data consisting of a plurality of frames,
  For the detected candidate image, determine the position state in the angle of view space,
Based on the position state of the candidate image in each frame determined as described above, the stable presence on the time axis in the image data over a plurality of frames is obtained for the candidate image, and the final period is longer than the initial period in the main subject determination period. The value of the stable presence level reflecting the determination result of the position state is calculated so that the value of the determination result of the position state is greater,
By determining the main subject in the candidate images using the obtained stable presence degree, when there are a plurality of candidate images, one main subject is determined from the plurality of candidate images.
Image processing method. Detect candidate images that are candidates for the main subject from each frame of the image data consisting of a plurality of frames,
For the detected candidate images, the positional relationship of the candidate images is determined according to whether or not the predetermined part of the main subject is in the determination reference area set in the angle of view space as the position state in the angle of view space. And
From the above-determined positional relationship of the candidate images in each frame, obtain the stable presence on the time axis in the image data over a plurality of frames for the candidate images,
By determining the main subject in the candidate images using the obtained stable presence degree, when there are a plurality of candidate images, one main subject is determined from the plurality of candidate images.
Image processing method.   A candidate image detected as a candidate for a main subject from each frame of image data consisting of a plurality of frames, as a position state in the angle of view space, with respect to a determination reference region set in accordance with an operation input in the angle of view space A position state determination process for determining a positional relationship between candidate images;
From the positional relationship of the candidate images in each frame determined in the position state determination process, a stable presence calculation process for obtaining a stable presence on the time axis in the image data over a plurality of frames for the candidate image,
When a plurality of candidate images are present by determining a main subject among the candidate images using the stable presence obtained in the stable presence degree calculation process, one main subject is selected from the plurality of candidate images. The process of determining
A program that causes an arithmetic processing unit to execute. A position state determination process for determining a position state in an angle-of-view space for each candidate image detected as a candidate for a main subject from each frame of image data including a plurality of frames;
From the position state of the candidate image in each frame determined in the position state determination process, the degree of stable existence on the time axis in the image data over a plurality of frames is obtained for the candidate image, and the initial in the main subject determination period A stable presence degree calculation process for calculating a value of a stable presence level that reflects the determination result of the position state, so that the value of the determination result of the position state is greater in the final period than in the period;
When a plurality of candidate images are present by determining a main subject among the candidate images using the stable presence obtained in the stable presence degree calculation process, one main subject is selected from the plurality of candidate images. The process of determining
A program that causes an arithmetic processing unit to execute. For each candidate image detected as a candidate for the main subject from each frame of the image data consisting of a plurality of frames, the predetermined state of the main subject is set in the determination reference area set in the angle of view space as the position state in the angle of view space. Position state determination processing for determining the positional relationship of candidate images according to whether or not
From the positional relationship of the candidate images in each frame determined in the position state determination process, a stable presence calculation process for obtaining a stable presence on the time axis in the image data over a plurality of frames for the candidate image,
When a plurality of candidate images are present by determining a main subject among the candidate images using the stable presence obtained in the stable presence degree calculation process, one main subject is selected from the plurality of candidate images. Main subject determination processing for determining
A program that causes an arithmetic processing unit to execute. The image processing apparatus according to claim 2 , wherein the main subject determination unit determines a distance of a candidate image with respect to a determination reference point set in an angle of view space as the position state. The image processing apparatus according to claim 19 , wherein the main subject determination unit sets the determination reference point according to an operation input. The image processing apparatus according to claim 2 , wherein the main subject determination unit determines a positional relationship of candidate images with respect to a determination reference region set in an angle of view space as the position state. The image processing apparatus according to claim 3, wherein the main subject determination unit sets the determination reference region according to an operation input. 13. The main subject determination unit according to claim 1 , wherein the main subject determination unit calculates cumulative time information in which the position state satisfies a predetermined condition as the degree of stable presence. Image processing device. 13. The main subject determination unit according to claim 1 , wherein the main subject determination unit calculates duration information in which the position state continues and satisfies a predetermined condition as the degree of stable presence . An image processing apparatus according to 1. The main subject determining section, positional information of the candidate images in angle space, or the size information of the candidate image, the calculation processing of the degree of stable presence, or claims 1 to 3 used in the determination processing of the main subject An image processing apparatus according to any one of claims 10 to 12 . The main subject determination unit selects one candidate image using position information or size information of the candidate image in the angle-of-view space when a plurality of candidate images are determined as the main subject from the degree of stable presence. The image processing apparatus according to any one of claims 1 to 3, and 10 to 12, wherein the selected candidate image is a main subject. The image according to any one of claims 1 to 3, and 10 to 12, wherein the candidate detection unit detects a human or animal face or body image as a candidate image that is a candidate for a main subject. Processing equipment. The image processing apparatus according to claim 1 , wherein the main subject determination unit uses information on a subject distance in processing for determining a main subject. The main subject determination unit reflects the determination result of the position state so that the value of the determination result of the position state is greater in the final period than in the initial period in the main subject determination period waiting for release . The image processing apparatus according to any one of claims 1 to 3, and 10 to 12, which calculates a value of a stable presence level.

Images