JP2016048863A - Subject detection device, control method of the same, imaging device, display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a subject detection device that can select a main subject on the basis of the intension of a person to be photographed.SOLUTION: There is provided a subject detection device that detects predetermined subjects from images supplied at predetermined time intervals and selects a main subject from the detected predetermined subjects, the subject detection device including: detection means for detecting the predetermined subjects from the images; operation recognition means for recognizing specific operations of the subjects detected by the detection means; and main subject selection means for selecting the main subject from the subjects detected by the detection means, where the main subject selection means has a main subject selection method associated with the specific operations recognized by the operation recognition means, and when the operation recognition means recognizes the specific operations, the main subject selection means selects the main subject on the basis of the main subject selection method associated with the specific operations.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a subject detection device, a control method thereof, an imaging device, a display device, and a program.

  An image processing method for automatically detecting a specific subject pattern from an image is very useful, and can be used, for example, for specifying a human face area. Such methods can be used in many areas such as teleconferencing, man-machine interfaces, security, monitor systems for tracking human faces, image compression, and the like. In digital cameras and digital video cameras, a specific subject is detected from a captured image, and the focus and exposure are optimized by using the detection result as a control target. When a plurality of subjects are detected from the captured image, a main subject is selected from the detected plurality of subjects, and optimal AF, AE, and the like are performed on the subject.

As described above, when a plurality of subjects are detected from the captured image, accurate AF and AE are performed.
For example, it is necessary to select a main subject. Patent Document 1 describes a selection method that suppresses the change of the main face extremely frequently while respecting changes in the state of the subject existing in the imaging range. Here, the state of the subject means various parameters representing the distance from the imaging device to the subject, the area of the subject in the imaging range, the position of the subject in the imaging range, the distance between subjects, and the like. Patent Document 2 describes a method of selecting one of a plurality of detected subject areas as a main subject according to an instruction from a user.

JP 2008-005438 A JP 2009-038749 A

  However, in the method described in Patent Document 1, since the main subject is selected depending on the composition and state of shooting, the optimal subject for the photographer and the subject may not be selected as the main subject. In Patent Document 2, since the photographer selects the main subject, the optimal subject can be selected as the main subject, but the main subject cannot be selected based on the intention of the subject.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide a subject detection apparatus capable of selecting a main subject based on an intention of a subject.

The above purpose is
A subject detection device that detects a predetermined subject from images supplied at predetermined time intervals and selects a main subject from the detected predetermined subject,
Detecting means for detecting a predetermined subject from the image;
Action recognition means for recognizing a specific action of the subject detected by the detection means;
A main subject selecting means for selecting a main subject from the subjects detected by the detecting means;
The main subject selection means has a main subject selection method associated with a specific action recognized by the action recognition means,
Achieved by a subject detection device characterized in that when a specific motion is recognized by the motion recognition means, the main subject selection means selects a main subject based on a main subject selection method associated with the specific motion. Is done.

The above purpose is
A method for controlling a subject detection apparatus that detects a predetermined subject from images supplied at predetermined time intervals and selects a main subject from the detected predetermined subject,
A detection step of detecting a predetermined subject from the image;
An action recognition step for recognizing a specific action of the subject detected in the detection step;
A main subject selection step of selecting a main subject from the subjects detected by the detection step;
The main subject selection step includes a main subject selection method associated with a specific motion recognized in the motion recognition step,
Control of a subject detection apparatus, wherein when a specific motion is recognized by the motion recognition step, the main subject selection step selects a main subject based on a main subject selection method associated with the specific motion. It is also achieved by the method.

  According to the subject detection device of the present invention, the main subject can be selected based on the intention of the subject.

It is a block diagram which shows the structure of the imaging device which concerns on 1st Embodiment. It is a figure which shows the flow of the process of main subject selection which concerns on 1st Embodiment. It is a figure which shows the operation example recognized by the operation | movement recognition part which concerns on 1st Embodiment. It is a figure which shows the example of the selection result in the main subject selection part which concerns on 1st Embodiment. It is a block diagram which shows the structure of the imaging device which concerns on 2nd Embodiment. It is a figure which shows the flow of a process of the main subject selection state notification which concerns on 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. Further, the embodiment of the present invention shows the most preferable mode of the invention, and does not limit the scope of the invention.

[Example 1]
The configuration and operation of the imaging apparatus according to the first embodiment will be described below with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 100 according to the first embodiment. In the imaging apparatus 100 of the present embodiment, a human face is detected as a subject, and a main subject is selected from the detected subjects.

  As shown in FIG. 1, in the imaging apparatus 100, light rays representing a subject image are collected by an imaging lens 101 and are incident on an imaging element 102 such as a CCD image sensor or a CMOS image sensor. The image sensor 102 outputs an electrical signal corresponding to the intensity of the incident light beam in units of pixels. This electrical signal is an analog video signal indicating a subject image captured by the image sensor 102.

  The video signal output from the image sensor 102 is subjected to analog signal processing such as correlated double sampling (CDS) in the analog signal processing unit 103. The video signal output from the analog signal processing unit 103 is converted into a digital data format by the A / D conversion unit 104 and input to the imaging control unit 105 and the image processing unit 106.

  The imaging control unit 105 is a CPU, a microcontroller, or the like, and centrally controls the operation of the imaging apparatus 100. Specifically, the imaging control unit 105 controls each unit of the imaging apparatus 100 by developing the program code stored in the ROM in the work area of the RAM and sequentially executing the program code.

  The image processing unit 106 performs image processing such as gamma correction and white balance processing on the input digital video signal. In addition to the normal image processing, the image processing unit 106 also has a function of performing image processing using information regarding a specific subject area in an image supplied from a main subject selection unit 111 described later.

  The video signal output from the image processing unit 106 is sent to the display unit 107. The display unit 107 is, for example, an LCD or an organic EL display, and displays a video signal. The imaging apparatus 100 can cause the display unit 107 to function as an electronic viewfinder (EVF) by sequentially displaying images sequentially captured in time series by the imaging element 102 on the display unit 107. In addition, it is possible to display the position of a subject area by a subject detection unit 109 and a main subject selection unit 111 described later.

  The video signal output from the image processing unit 106 is recorded on a recording medium 108, for example, a removable memory card. Note that the recording destination of the video signal may be the built-in memory of the imaging apparatus 100 or an external apparatus (both not shown) that are communicably connected via a communication interface.

  The video signal output from the image processing unit 106 is also supplied to the subject detection unit 109. The subject detection unit 109 detects a predetermined target subject in the image and specifies a subject region, and specifies a human face region as the subject region. When there are a plurality of human faces in the image, an area corresponding to the number of persons is detected. As a detection method in the subject detection unit 109, a known face detection method is used. Known techniques for face detection include a method of using knowledge about the face (skin color information, parts such as eyes, nose, mouth, etc.) and a method of configuring a classifier for face detection by a learning algorithm represented by a neural network. is there. Further, in face detection, face recognition is generally performed by combining them in order to improve the recognition rate. Specifically, there is a method for detecting a face using wavelet transform and an image feature amount described in JP 2002-251380 A.

  In addition, the detection target of the subject detection unit 109 in the present embodiment is not limited to a human face, and other subject areas can be applied as detection targets. For example, a person's upper body etc. are mentioned as a detection target. As a detection method, a method described in Japanese Patent Application Laid-Open No. 2009-211311 is given as an example. In this example, the edge strength of the local upper body contour is detected as a local feature amount. As a method for extracting a feature amount from an image, there are various methods such as a Sobel filter, a Prewitt filter, and a Haar filter. The extracted local feature is discriminated between the upper body and the non-upper body using a person classifier.

  The discrimination in the person discriminator is performed based on machine learning such as AdaBoost learning. The result of the subject detection unit 109 is supplied to an operation recognition unit 110 and a main subject selection unit 111 described later.

  The motion recognition unit 110 recognizes the motion of the subject detected by the subject detection unit 109. The recognition result by the motion recognition unit 110 is supplied to the main subject selection unit 111. The movement in the present embodiment refers to a change with physical movement such as “waving the hand left and right” for a person. Assume that the subject detection unit 109 detects a human face, and an example of a processing method performed by the motion recognition unit 110 will be described. A shape model is applied to the image of the subject detected by the subject detection unit 109. A shape model is a shape model of each part of the human body that is approximated by a simplified three-dimensional part model (cylinder, ellipsoid, super-quadrics, etc.), based on the position and orientation of the face (head). Estimate the existence range of arms, hands, feet, moving objects, etc.

  Feature values (local and geometric features, motion vectors, color component features, etc.) for each part constituting the target person are stored in a predetermined storage means as an average feature value near the representative point of the part. , Extract time-series image data of representative point positions in parts, compare the extracted time-series image data and behavior model data, and check whether or not behavior model data corresponding to the time-series image data exists Determine. Here, the motion model data refers to a feature amount of the motion to be recognized, and is a time-series feature amount by one or more parts. For details of human motion extraction technology, the technology described in the literature (Hoshino, “Extraction and reconstruction of personality in human motion”, Image Electronics Society of Japan, Vol. 30, pp. 631-640, 2001) is used. That's fine.

  As a basic method of matching processing with motion model data, a method called Dynamic time warping (TJDarell and APPentland, 1993 “Space-Time Gestures,”) ), A method using a hidden Markov model that performs state transition of motion with a stochastic model (J. Yamato, J. Ohya, and K. Ishii, 1992 “Recognizing Human Action in Time-Sequential Images Using Hidden Markov Model,” Proc. CVPR, pp. 379-385), Continuous Dynamic Programming Method of Space-Time Edge Image Vector Sequence (Nishimura, Oka “Non for Spotting Recognizing Reverse Time Motion and Still Motion of Gesture Video” -monotonic continuous DP ", IEICE Technical Report, PRMU 96-32, pp. 49-56, 1996).

  The main subject selection unit 111 selects a main subject from the subjects detected by the subject detection unit 109. The main subject selection method in the main subject selection unit 111 differs depending on whether or not a specific motion is recognized in the motion recognition unit 110. When the motion recognition unit 110 does not recognize a specific motion, the main subject is selected according to the position and size of the subject output by the subject detection unit 109. A subject whose size is large and whose position is close to the center of the image is preferentially selected as the main subject. Further, in order to suppress the time-series switching of the main subject, the subject selected as the main subject in the previous frame may be preferentially selected as the main subject.

  On the other hand, when the motion recognition unit 110 recognizes a specific motion, the main subject is selected by a selection method associated in advance with the specific motion. For example, the main subject selection method defined as motion model data in the motion recognition unit 110 “was left hand to the left and right” and “uses the subject whose motion has been recognized as the main subject” is associated. Based on this association, when the motion detection unit 110 recognizes the motion model data “shake the left hand to the left and right”, the subject detected by the subject detection unit 109 is selected as the main subject.

  The imaging control unit 105 controls imaging conditions such as a focus situation and an exposure situation when imaging with the imaging device 102. Specifically, the imaging control unit 105 controls a focus control mechanism and an exposure control mechanism (neither shown) of the imaging lens 101 based on the video signal output from the A / D conversion unit 104. For example, the focus control mechanism is an actuator that drives the imaging lens 101 in the optical axis direction, and the exposure control mechanism is an actuator that drives an aperture or a shutter.

  The imaging control unit 105 can use information on the main subject region selection result supplied from the main subject selection unit 111 for the control of the focus control mechanism and the exposure control mechanism. Specifically, focus control using the contrast value of the main subject area and exposure control using the brightness value of the subject area are performed. Therefore, the imaging apparatus 100 can perform imaging processing in consideration of a specific subject area in the captured image. The imaging control unit 105 also performs readout control of the image sensor 102 such as output timing and output pixels of the image sensor 102.

  A flow of main subject selection processing according to the first embodiment will be described with reference to FIG. The image processing unit 106 reads, as an input image, an image obtained by digitizing the captured image obtained via the image sensor 102 by the A / D conversion unit 104 (S201). Next, the subject detection unit 109 detects a subject region in the input image and outputs detection data (S202).

  Next, the motion recognition unit 110 recognizes a specific motion with respect to the subject detected by the subject detection unit 109 (S203). In this embodiment, the operation shown in FIG. 3 is recognized as the specific operation. In (A) of FIG. 3, “the left hand of the subject is swung left and right with the head”, in (B) of FIG. 3, “the hands of the subject are swung left and right in front of the moving object”, and in (C) of FIG. , “Operation to point both hands of the subject in one direction on the head” is shown. In the motion recognition unit 110, a time-series positional relationship between human body parts as shown in FIG. 3 is stored in advance as motion model data. A three-dimensional shape model composed of a plurality of human body parts is fitted to the input image, and the trajectory of each part is matched with the motion model data to determine whether or not they match.

  Next, it is determined whether or not a specific action is recognized by the action recognition unit 110 (S204). Depending on the recognition result of the motion recognition unit 110, the selection method in the main subject selection unit 111 differs. FIG. 4 shows an example of the main subject selection result. A subject surrounded by a solid rectangular frame in FIG. 4 indicates a main subject, and a subject surrounded by a dotted rectangular frame indicates a non-main subject.

  When the motion recognition unit 110 does not recognize a specific motion (determined as NO in S204), the main subject is selected based on the position and size of the subject obtained from the subject detection unit 109 (S205). In order to preferentially select a subject whose position is close to the center of the screen and is large in size as a main subject, a main subject selection result as shown in FIG. 4A is obtained.

  On the other hand, when a specific motion is recognized by the motion recognition unit 110 (YES in S204), the main subject is selected by a selection method associated with the recognized motion (S206). As a main subject selection method related to a specific motion, a method that preferentially selects a subject that recognizes a specific motion as a main subject, and a subject that recognizes a specific motion preferentially as a non-main subject. And a method for preferentially selecting the subject indicated by the subject that has recognized the motion as the main subject. In this embodiment, it is assumed that the operation shown in FIG. 3A is associated with a method for preferentially selecting a subject whose motion has been recognized as a main subject.

  Therefore, as shown in FIG. 4B, the result is that the subject whose motion has been recognized is selected as the main subject. Further, it is assumed that the operation shown in FIG. 3B is associated with a method for preferentially selecting a motion-recognized subject as a non-main subject. Therefore, as shown in FIG. 4C, the subject whose motion has been recognized is selected as a non-main subject, and then the subject whose position is close to the center of the image and whose size is large is selected as the main subject. It becomes. Further, it is assumed that the operation indicating the direction shown in FIG. 3C is associated with a method of preferentially selecting the subject indicated by the subject whose motion has been recognized as the main subject. Therefore, as shown in FIG. 4D, the result is that the subject existing at the position indicated by the motion-recognized subject is selected as the main subject.

  Here, the process of the motion recognition unit 110 is necessary for each subject detected by the subject detection unit 109, and the processing load increases when the number of subjects detected by the subject detection unit 109 is large. Therefore, it is necessary to take measures to reduce the processing load. Therefore, processing efficiency is improved by narrowing down the types of motion models recognized by each subject according to the selection result of the main subject selection unit 110. In the selection result of the main subject selection unit 110, regarding the subject currently selected as the main subject, there is no need to recognize the operation associated with the method of selecting the subject as the main subject. For a subject that is currently selected as a non-main subject, it is not necessary to recognize an operation associated with a method of selecting the subject as a non-main subject.

  In other words, it is not necessary to recognize an action associated with a selection method that does not change the current main subject selection result, and it is possible to improve the efficiency of the process by excluding the unnecessary action from the recognition target. In this embodiment, with respect to a subject that is currently selected as a main subject, a method of preferentially selecting a subject that has recognized a specific motion as a non-main subject, and a specific motion is a motion indicating a direction. Related to the method of preferentially selecting the subject that recognized a specific motion as the non-main subject, with only the motion associated with the method of preferentially selecting the subject indicated by the subject recognized as the main subject Excluded actions are excluded from recognition.

  On the other hand, for a subject that is currently selected as a non-subject subject, a method of preferentially selecting a subject that has recognized a specific motion as a main subject, a specific motion is a motion indicating direction, and the motion has been recognized. An operation associated with a method that preferentially selects a subject that recognizes a specific motion as a non-main subject, with only the motion associated with the method that preferentially selects the subject indicated by the subject as the main subject Are excluded from recognition.

  As described above, according to the first embodiment, the subject can intentionally select the main subject based on the intention of the subject by performing an operation associated with the main subject selection method. it can.

[Second Embodiment]
The configuration and operation of the imaging apparatus according to the second embodiment will be described focusing on differences from the first embodiment. Note that the description of the same configuration and operation as in the first embodiment is omitted.
FIG. 5 is a block diagram illustrating a configuration of an imaging apparatus 500 according to the second embodiment. The configuration and operation of the imaging apparatus 500 according to the second embodiment are different from those of the first embodiment in that a notification unit 501 is included.

  With the imaging apparatus 100 according to the first embodiment, the subject can consciously perform an operation associated with the main subject selection method, so that the main subject can be selected based on the intention of the subject. Thus, it is possible to perform imaging processing in consideration of the main subject area. However, the subject cannot recognize which subject is selected as the main subject in the imaging apparatus 100. Therefore, the imaging apparatus 500 according to the second embodiment includes a notification unit 501 that notifies the subject of the current selection state of the main subject selection unit 110.

  Note that in the motion recognition unit 110 in the imaging apparatus 500, in addition to the motion model to be recognized described in the first embodiment, a specific motion is associated with a motion that prompts notification of the main subject selection state.

  A flow of processing related to the notification of the main subject selection state according to the second embodiment will be described with reference to FIG. The image processing unit 106 reads, as an input image, an image obtained by digitizing the captured image obtained via the image sensor 102 by the A / D conversion unit 104 (S601). Next, the subject detection unit 109 detects a subject region in the input image and outputs detection data (S602). Next, the motion recognition unit 110 recognizes a specific motion with respect to the subject detected by the subject detection unit 109 (S603).

  Here, it is determined whether or not the motion recognition unit 110 has recognized a motion that prompts notification of the main subject selection state (S604). When the operation for prompting the notification of the main subject selection state is not recognized (determined as NO in S604), the processing related to the notification of the main subject selection state for the input image ends. If an operation that prompts notification of the main subject selection state is recognized (YES in S604), it is determined whether or not the subject that recognized the operation is the main subject (S605). When the subject that has recognized the specific motion is the main subject (YES in S605), the notification unit 501 performs the notification method 1 (S606). When the subject that has recognized the specific motion is a non-main subject (determined as NO in S605), the notification unit 501 performs the notification method 2 (S607).

  The notification unit 501 is an LED, for example, and changes the LED emission method according to the main subject selection state of the subject whose operation has been recognized. For example, the notification method 1 emits red light, and the notification method 2 emits blue light.

  As described above, according to the second embodiment, the subject can recognize which subject is selected as the main subject in the imaging apparatus.

[Other Embodiments]
In the above-described embodiment, the case where the subject detection device is applied to the imaging device is illustrated, but the device to which the subject detection device is applied is not limited to the imaging device. For example, a subject detection device that performs subject detection may be applied to a display device that displays an image (reproduction data) supplied from an external device or a recording medium. In this display device, the reproduction data is used as subject region detection processing data, and subject region detection processing is performed. A control unit such as a microcontroller in the display device controls display conditions for displaying an image based on subject information (position, size, etc. of the subject in the image) extracted by subject detection processing.

  Specifically, information indicating a subject such as a frame is superimposed on the position of the subject in the image, and the brightness and color of the display image are controlled according to the brightness and color information of the subject portion. Further, the display method of the frame to be superimposed may be changed depending on whether or not the detected subject is the main subject. Moreover, the description in embodiment mentioned above shows an example, and is not limited to this. The configuration and operation in the embodiment described above can be changed as appropriate.

  The above-described embodiment can also be realized in software by a computer of a system or apparatus (or CPU, MPU, etc.). Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto. A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code. Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers. That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which the computer program for realizing the above-described embodiment is encrypted and distributed is distributed, and key information for decrypting is supplied to a user who satisfies a predetermined condition, and the user's computer Installation may be allowed. The key information can be supplied by being downloaded from a homepage via the Internet, for example. Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer. Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

100 imaging device, 101 imaging lens, 102 imaging device,
103 analog signal processing unit, 104 A / D conversion unit, 105 photographing control unit,
106 image processing unit, 107 display unit, 108 recording medium, 109 subject detection unit,
110 motion recognition unit, 111 main subject selection unit, 501 notification unit

Claims (10)

A subject detection device that detects a predetermined subject from images supplied at predetermined time intervals and selects a main subject from the detected predetermined subject,
Detecting means for detecting a predetermined subject from the image;
Action recognition means for recognizing a specific action of the subject detected by the detection means;
A main subject selecting means for selecting a main subject from the subjects detected by the detecting means;
The main subject selection means has a main subject selection method associated with a specific action recognized by the action recognition means,
An object detection apparatus characterized in that, when a specific action is recognized by the action recognition means, the main subject selection means selects a main subject based on a main subject selection method associated with the specific action. The main subject selection method associated with a specific operation in the main subject selection means is:
A method of preferentially selecting a subject that recognizes the specific movement as a main subject;
A method of preferentially selecting a subject that recognizes the specific movement as a non-main subject;
2. The method according to claim 1, wherein the specific motion is a motion indicating a direction and is at least one of methods for preferentially selecting a subject indicated by a subject that has recognized the motion as a main subject. Subject detection device. The subject detection apparatus according to claim 1, wherein the motion recognition unit changes a type of a specific motion to be recognized according to a main subject selection state. The motion recognition means includes
For the subject selected as the main subject, the operation related to the method of preferentially selecting as the main subject in the main subject selecting means is excluded from the recognition operation,
For a subject selected as a non-main subject, an operation associated with a method of preferentially selecting the subject as a non-main subject in the main subject selection means is excluded from the recognition operation. The subject detection apparatus according to claim 3. A notification means for notifying the main subject selection state;
The main object selection state is notified by the notification means when the specific action is recognized as an action that prompts notification of a main subject selection state, and the specific action is recognized by the action recognition means. The subject detection apparatus according to any one of claims 1 to 4. The notification means is means for notifying a main subject selection state by changing a light emitting method,
The light emitting method of the notifying unit is changed depending on whether the subject that has recognized the action of prompting notification of the main subject selection state by the motion recognizing unit is a main subject or a non-main subject. The subject detection apparatus according to claim 5. An imaging means for supplying a captured image;
A subject detection device according to any one of claims 1 to 6,
Control means for controlling imaging conditions when the imaging means captures images according to information on a main subject selected by the subject detection device;
An imaging apparatus comprising: Display means for displaying the supplied image;
A subject detection device according to any one of claims 1 to 6,
Control means for controlling display conditions displayed by the display means in accordance with information on a main subject selected by the subject detection device;
A display device comprising: A method for controlling a subject detection apparatus that detects a predetermined subject from images supplied at predetermined time intervals and selects a main subject from the detected predetermined subject,
A detection step of detecting a predetermined subject from the image;
An action recognition step for recognizing a specific action of the subject detected in the detection step;
A main subject selection step of selecting a main subject from the subjects detected by the detection step;
The main subject selection step includes a main subject selection method associated with a specific motion recognized in the motion recognition step,
Control of a subject detection apparatus, wherein when a specific motion is recognized by the motion recognition step, the main subject selection step selects a main subject based on a main subject selection method associated with the specific motion. Method. The program for making a computer perform each process of the control method of the to-be-photographed object detection apparatus of Claim 9.