JP7483352B2 - Photographing device, its control method, and program - Google Patents

Description

The present invention relates to an imaging device and a control method and program thereof, and in particular to an imaging device that determines a main subject area and a control method and program thereof.

Conventionally, imaging devices have existed that detect a subject from an image and perform imaging control such as AUTO FOCUS (hereafter referred to as AF). In addition, technology is also known that automates the determination of the main subject area when performing imaging control such as AF, thereby supporting users in taking photographs.

For example, there is an imaging device that is provided with a first means for selecting one of multiple main subject candidates as the main subject, which selects the subject closest to the center of the image as the main subject, and a second means for selecting a subject having a specified color as the main subject (see, for example, Patent Document 1).

JP 2002-77711 A

However, with the first method of Patent Document 1, if the subject that the user wants to photograph is not near the center of the image, a subject contrary to the user's intention will be selected as the main subject. On the other hand, with the second method of Patent Document 1, it is necessary to set information (subject type or color information) that identifies the color of the subject to be photographed in advance. Therefore, when there are multiple subjects within the angle of view, the user needs to adjust the orientation of the imaging device so that the subject to be photographed is near the center of the image, or to add information that identifies the color of the subject before shooting. Such adjustments to the orientation of the imaging device and additional settings of information can be an obstacle when the user wants to immediately photograph a subject that is within the angle of view.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide an imaging device, a control method thereof, and a program that can select a main subject from among multiple subjects present within the angle of view in accordance with the user's intentions.

An imaging device according to a first aspect of the present invention is an imaging device comprising: object detection means for detecting an object included in an image captured by the imaging device and detecting a type of the detected object; motion type estimation means for estimating a motion type of the imaging device; priority assignment means for, when a plurality of objects are detected by the object detection means, assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the motion type of the imaging device estimated by the motion type estimation means and the respective types of the plurality of objects detected by the object detection means ; and measurement means for measuring a position and orientation change of a main body of the imaging device, wherein the motion type estimation means: The present invention is characterized in that a movement type of the imaging device is estimated based on an output of the measurement means during a predetermined period, the types of objects detectable by the object detection means are types corresponding to the expected movement of each object detectable by the object detection means, the movement type estimation means calculates a moving speed of the imaging device during the predetermined period based on the output of the measurement means during the predetermined period, and the priority assignment means assigns a priority to each of the plurality of objects detected by the object detection means as a target for imaging control, based on the calculated moving speed of the imaging device during the predetermined period and the expected movement of each of the plurality of objects .
An imaging device according to a second aspect of the present invention is an imaging device comprising: object detection means for detecting an object included in an image captured by the imaging device and detecting a type of the detected object; motion type estimation means for estimating a motion type of the imaging device; priority assignment means for, when a plurality of objects are detected by the object detection means, assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the motion type of the imaging device estimated by the motion type estimation means and the respective types of the plurality of objects detected by the object detection means; and measurement means for measuring a position and orientation change of a main body of the imaging device, wherein the motion type estimation means: The present invention is characterized in that a motion type of the imaging device is estimated based on an output of the measuring means during a predetermined period, the type of object that can be detected by the object detection means is a type corresponding to a degree of acceleration/deceleration expected for the object, the motion type estimation means calculates a degree of acceleration/deceleration of the imaging device during the predetermined period based on the output of the measuring means during the predetermined period, and the priority assignment means assigns a priority to each of the plurality of objects detected by the object detection means as a subject of imaging control based on the calculated degree of acceleration/deceleration of the imaging device during the predetermined period and the degree of acceleration/deceleration expected for each type of the plurality of objects.
An imaging device according to claim 3 of the present invention is an imaging device comprising: object detection means for detecting an object included in an image captured by the imaging device and detecting a type of the detected object; motion type estimation means for estimating a motion type of the imaging device; priority assignment means for, when a plurality of objects are detected by the object detection means, assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the motion type of the imaging device estimated by the motion type estimation means and the types of each of the plurality of objects detected by the object detection means; measurement means for measuring a change in position and orientation of a main body of the imaging device; and analysis means for analyzing frequency components of an output of the measurement means during the specified period, wherein the motion type estimation means estimates the motion type of the imaging device based on the output of the measurement means during the specified period and the frequency components analyzed by the analysis means.
An imaging device according to claim 6 of the present invention is an imaging device comprising: object detection means for detecting an object included in an image captured by the imaging device; type detection means for detecting the types of the plurality of objects; acquisition means for acquiring shooting settings specified by a user for the imaging device; and priority assignment means for, when a plurality of objects are detected by the object detection means, classifying each of the plurality of objects into one of a plurality of predefined classes, and assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the shooting settings acquired by the acquisition means and the class into which each of the plurality of objects is classified, wherein the class is a moving speed on the captured image calculated based on a speed defined for each type of object detectable by the object detection means, and the priority assignment means assigns a priority to each of the plurality of objects detected by the object detection means as a subject of imaging control in accordance with the moving speed on the captured image calculated for each of the plurality of objects.
An imaging device according to claim 7 of the present invention is an imaging device comprising: an object detection means for detecting an object included in an image captured by the imaging device; an acquisition means for acquiring shooting settings specified by a user for the imaging device; a priority assignment means for classifying each of the plurality of objects into one of a plurality of predefined classes when a plurality of objects are detected by the object detection means, and assigning a priority for making each of the plurality of objects a subject of imaging control to each of the plurality of objects according to the shooting settings acquired by the acquisition means and the class into which each of the plurality of objects is classified; and a type detection means for detecting a type of the plurality of objects, wherein the class is a moving speed on the captured image calculated based on a speed defined for each type of object detectable by the object detection means, and the priority assignment means, when one of the shooting settings is a shutter speed priority mode by the acquisition means, assigns a priority for making each of the plurality of objects a subject of imaging control to each of the plurality of objects detected by the object detection means according to the shutter speed included in the shooting settings acquired by the acquisition means and the class into which each of the plurality of objects is classified.

An imaging device according to claim 11 of the present invention is an imaging device comprising: an object detection means for detecting an object included in an image captured by the imaging device; an acquisition means for acquiring shooting settings specified by a user for the imaging device; a priority assignment means for, when a plurality of objects are detected by the object detection means, classifying each of the plurality of objects into one of a plurality of predefined classes and assigning a priority to each of the plurality of objects as a subject of imaging control according to the shooting settings acquired by the acquisition means and the class into which each of the plurality of objects is classified; and a type area detection means for detecting the types of the plurality of objects and the area in which the object is located in the captured image, wherein the speed defined for each type of object detectable by the object detection means is a relative speed with respect to a subject size defined for each type of object detectable by the object detection means, and for each of the plurality of objects, a moving speed on the captured image is calculated based on a speed defined as the relative speed of the type detected by the type area detection means and the size of the area detected by the type area detection means.

According to the present invention, a main subject can be selected from among multiple subjects present within the field of view in accordance with the user's intentions.

1 is a block diagram showing a configuration of an imaging device according to a first embodiment. 5 is a flowchart showing a procedure of an imaging control process according to the first embodiment. 10 illustrates an example of classification of motion types of the imaging device in the first embodiment, detectable objects, and examples of priorities given to each detectable object defined for each motion type of the imaging device as an imaging control target. 11A to 11C are diagrams illustrating an application example of an imaging control process according to the first embodiment. FIG. 1 is a diagram illustrating the basic configuration of a CNN. 11A to 11C are diagrams showing details of feature detection processing and feature integration processing. 10 is a flowchart showing a procedure of an imaging control process according to a second embodiment. 13 is a schematic diagram of an example of calculating the velocity of a detected object in a coordinate system on an imaging element in the second embodiment. FIG. 13 is a schematic diagram of an example of calculating the velocity of a detected object on a captured image in the second embodiment. FIG. 13 is an example showing a priority as an object to be subjected to imaging control, which is determined according to the degree of narrowing and the type of each detectable object in the second embodiment.

The following describes in detail the embodiments of the present invention with reference to the attached drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention.

(Configuration of the imaging device)
The configuration of an image pickup apparatus according to this embodiment will be described with reference to FIG.

Figure 1 is a block diagram showing the configuration of an imaging device 100 according to the first embodiment.

The imaging device 100 may be a digital still camera or video camera capable of recording video and still image data obtained by photographing a subject on various media such as tape, solid-state memory, optical disk, and magnetic disk, but is not limited to these. For example, the imaging device 100 may be a mobile information terminal with an imaging function such as a smartphone or tablet capable of transferring video and still image data obtained by photographing a subject to cloud storage.

The various components within the imaging device 100 are connected to each other via a bus 160 so that they can communicate with each other. In addition, the various components within the imaging device 100 are controlled by a CPU 151 (central processing unit).

The lens unit 101 has a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed third group lens 121, a focus lens 131, a zoom motor 112 (ZM), an aperture motor 104 (AM), and a focus motor 132 (FM). The fixed first group lens 102, the zoom lens 111, the aperture 103, the fixed third group lens 121, and the focus lens 131 constitute a photographing optical system. Each of these lenses may be composed of multiple lenses. The lens unit 101 may also be configured as a detachable interchangeable lens.

The aperture control unit 105 drives the aperture 103 via the aperture motor 104 in accordance with instructions from the CPU 151, thereby adjusting the aperture diameter of the aperture 103 and adjusting the amount of light during shooting.

The zoom control unit 113 changes the focal length (angle of view) by driving the zoom lens 111 via the zoom motor 112.

The focus control unit 133 determines the drive amount for driving the focus motor 132 based on the amount of deviation in the focus direction of the lens unit 101. In addition, the focus control unit 133 controls the focus state by driving the focus lens 131 via the focus motor 132. AF control is achieved by controlling the movement of the focus lens 131 by the focus control unit 133 and the focus motor 132. Note that the method for calculating the amount of deviation in the focus direction of the lens unit 101 in this embodiment is not particularly limited, but is calculated, for example, by a phase difference AF method or a contrast AF method.

The subject image (captured image) formed on the image sensor 141 via the lens unit 101 is converted into an electrical signal (image signal) by the image sensor 141 and output from the image sensor 141. The image sensor 141 has photoelectric conversion elements that perform photoelectric conversion arranged in a matrix with M elements in the horizontal direction and N elements in the vertical direction. The image signal output from the image sensor 141 is prepared as captured image data by the image signal processing unit 142 and then output to the image capture control unit 143.

The imaging control unit 143 controls the RAM 154 (random access memory) to temporarily store the captured image data output by the imaging signal processing unit 142.

When the RAM 154 temporarily stores the captured image data under the control of the imaging control unit 143, it transmits the captured image data in parallel to both the image compression/decompression unit 153 and the image processing unit 152. In addition, in the imaging control process (FIG. 2) described later, it also transmits the captured image data to the object detection unit 162.

The RAM 154 can also be used as a ring buffer. In this case, the RAM 154 can buffer a plurality of captured image data captured within a predetermined period, the detection results of the object detection unit 162 corresponding to each captured image data, and the change in position and orientation (angular velocity) of the imaging device 100 output from the position and orientation change acquisition unit 161.

The image compression/decompression unit 153 compresses the captured image data sent from the RAM 154, and then outputs it to the image recording medium 157 for recording.

The image processing unit 152 processes the captured image data sent from the RAM 154 and changes the size of the captured image data to an optimal size for display on the monitor display 150. The image processing unit 152 then sends the resized captured image data to the monitor display 150 as appropriate. The image processing unit 152 also processes the captured image data sent from the RAM 154 and performs operations such as calculating the similarity between the captured image data.

The monitor display 150 displays a preview image or a through image based on the captured image data sent from the image processing unit 152. When performing such display, the monitor display 150 can also superimpose the object detection results of the object detection unit 162 as a rectangle or the like.

The operation switch 156 is an input interface that includes a touch panel integrated with the monitor display 150 and physical buttons such as a release button. The user can issue various instructions to the imaging device 100 by using the touch panel to select various function icons displayed on the monitor display 150.

The CPU 151 determines the accumulation time of the image sensor 141 and the gain setting value when outputting from the image sensor 141 to the image signal processing unit 142 based on a user instruction via the operation switch 156 and the size of the captured image data temporarily stored in the RAM 154. The CPU 151 then transmits the determined accumulation time and gain setting value to the image capture control unit 143.

The imaging control unit 143 controls the imaging element 141 based on the accumulation time and gain setting values transmitted from the CPU 151.

The object detection unit 162 performs object detection processing on the captured image data sent from the RAM 154 to detect objects contained in the captured image, as well as to detect the type of object and the area in the captured image in which the object is located (type detection means/type area detection means). The object detection processing in the object detection unit 162 is realized by feature extraction processing using CNN (CONVOLUTINAL NEURAL NETWORKS). The learning processing of the object detection unit 162 will be described in detail later. The CPU 151 assigns a priority to each of the objects detected by the object detection unit 162 in the imaging control processing of FIG. 2 described later.

The focus control unit 133 realizes AF control for the area within the angle of view where the object to be the main subject is located (hereinafter referred to as the main subject area).

The aperture control unit 105 performs exposure control using the brightness value of the main subject area.

The image processing unit 152 performs gamma correction, white balance processing, etc. to optimize the image of the main subject area.

The battery 159 is appropriately managed by the power management unit 158 and provides a stable power supply to the entire imaging device 100.

Flash memory 155 records the control programs necessary for the operation of imaging device 100 and parameters used for the operation of each part. When imaging device 100 is started by a user's operation (when the power is switched from OFF to ON), the control programs and parameters stored in flash memory 155 are loaded into a part of RAM 154. CPU 151 controls the operation of imaging device 100 according to the control programs and constants loaded into RAM 154.

The position and orientation change acquisition unit 161 is composed of a position and orientation sensor such as a gyro, acceleration sensor, or electronic compass, and measures the change in position and orientation of the imaging device 100 relative to the shooting scene. In this embodiment, the position and orientation change (angular velocity) caused by rotation around each axis in the yaw and pitch directions of the imaging device 100 is measured and buffered in the RAM 154 for use, but the present invention is not intended to be limited to this.

(Overall processing flow)
Next, the imaging control process according to this embodiment will be described with reference to the flowchart in Fig. 2. This process is started when the user half-presses the release button, which is the operation switch 156, and is executed by the CPU 151 based on a control program loaded into the RAM 154.

First, in step S201, the imaging control unit 143 supplies captured image data to each unit including the image processing unit 152 and the object detection unit 162 via the RAM 154. The captured image data supplied to the image processing unit 152 in step S201 is displayed as a through image on the monitor display 150 after size adjustment and the like in the image processing unit 152.

Next, in step S202, the object detection unit 162 performs object detection processing on the captured image data supplied in step S201. Details of the object detection processing will be described later, but the type of object previously learned by the object detection unit 162 and the area in which the detected object is located are obtained as detection results for each object included in the captured image. At this time, each detected object is displayed as a main subject candidate on the through image displayed on the monitor display 150. For example, a dotted frame is displayed around each detected object on the through image.

Next, in step S203, it is determined whether or not two or more objects were detected by the object detection unit 162 in step S202. If two or more objects were detected, the process proceeds to step S204; if not, the process proceeds to step S206.

Next, in step S204, the motion type of the imaging device 100 is estimated (motion type estimation means). Here, the motion type of the imaging device 100 is a type for classifying how the user moved the imaging device 100 (camera work) with the intention of tracking a subject within a predetermined period up to the current time. Specific examples of the motion types of the imaging device 100 and the estimation method thereof will be described later.

Next, in step S205, a priority is assigned to each object detected in step S202 as a subject of imaging control according to the type of motion of the imaging device 100 estimated in step S204 (priority assignment means). The imaging control in this embodiment includes exposure compensation, AF, and vibration reduction. At this time, the object with the highest assigned priority may be highlighted on the through image displayed on the monitor display 150. This allows the user to determine whether the highest priority has been assigned to the object that the user wants to use as the main subject among the objects present within the angle of view. Note that the highlighting is not particularly limited as long as it is a display that allows the user to visually recognize which object has the highest assigned priority among the objects detected by the object detection unit 162. For example, examples of highlighting include a configuration in which a black frame is displayed around the object with the highest assigned priority on the through image, or an arrow pointing to the object with the highest assigned priority is displayed.

Next, in step S206, it is determined whether or not an end command has been received from the user. If an end command has been received, this process ends, and if an end command has not been received, the series of processes from step S201 are repeated.

For example, if the user fully presses the release button, which is the operation switch 156, it is determined that an end command has been issued from the user. In this case, the object that was assigned the highest priority in step S205 is determined to be the main subject, and after AF control by the focus control unit 133 and exposure control by the aperture control unit 105 are performed on the main subject area, the CPU 151 starts shooting.

In addition, if the user touches one of the main subject candidates on the through image using the touch panel of the operation switch 156, it is also determined that an end instruction has been given from the user. In this case, the main subject candidate touched by the user is determined to be the main subject, regardless of the priority level assigned in step S205.

Next, various application examples of the imaging control process according to this embodiment will be described with reference to FIG.

If it is determined in step S203 of FIG. 2 that multiple objects have been detected by the object detection unit 162, it is necessary to narrow down which objects are to be subject to imaging control. Therefore, in this embodiment, various types of motion of the imaging device 100 are defined in advance, and in step S204 of FIG. 2, it is estimated which type of motion the camera work of the imaging device 100 corresponds to. Furthermore, in step S205 of FIG. 2, a priority is assigned to each of the multiple detected objects based on the estimated type of motion of the imaging device 100.

(Application Example 1 of Example 1: Smoothly Moving Subject)
Hereinafter, application example 1 of this embodiment will be described with reference to FIG.

In this application example, the CPU 151 determines through the imaging control process of FIG. 2 that the user is attempting to photograph a moving car among the three main subject candidates detected by the object detection unit 162: a building, a dog, and a car.

The curve in the graph shown in FIG. 4A is the change in position and orientation in the yaw direction of the image capture device 100 during a predetermined period up to the current time, obtained from the position and orientation change acquisition unit 161 in this application example. From the curve in this graph, it can be inferred that the user is tracking an object moving in the horizontal direction, and therefore a stationary object is not an object that the user wants to use as the main subject. Therefore, the CPU 151 assigns the lowest priority to a stationary object, a building, among the main subject candidates. Also, as shown in the graph in FIG. 4A, when the user is panning the image capture device 100 with a small speed change, a higher priority is assigned to a car, among the remaining main subject candidates, than to a dog, which is likely to move randomly (has a large degree of acceleration and deceleration). Note that a specific determination of whether the speed change of the panning of the image capture device 100 is small will be described later.

There is a known method for selecting the subject whose position changes the least within the angle of view while the camera is panning as the main subject. This method is effective when the user wants to capture the main subject in the center of the angle of view.

However, this is not effective when the camera is panned slightly slower than the speed of the subject's movement to emphasize the fast movement of the subject or the appearance of the subject, and the subject is photographed crossing the screen. Also, in such shooting, the user usually expects the subject to move at a stable speed and reduces the change in the panning speed of the camera. Therefore, as in this application example, when the user pans the imaging device 100 with small speed changes, if the user selects a dog that is likely to move randomly in the field of view as the main subject, the user's shooting intention will not be reflected.

Therefore, in this application example, the objects that can be detected by the object detection unit 162 and information indicating the characteristics of the movement of the objects are defined in advance, and the camera determines which subject the user is intending to make the main subject based on camera work such as panning.

(Application Example 2 of Embodiment 1: Object undergoing rapid acceleration and deceleration)
Hereinafter, application example 2 of this embodiment will be described with reference to FIG.

In this application example, the CPU 151 determines through the imaging control process of FIG. 2 that the user is attempting to photograph a running dog among the three main subject candidates detected by the object detection unit 162: a building, a dog, and a car.

The curve of the graph shown in FIG. 4B is the change in position and orientation in the yaw direction of the image capture device 100 during a predetermined period up to the current time, obtained from the position and orientation change acquisition unit 161 in this application example. From the curve of this graph, it can be inferred that the user is tracking an object moving in the horizontal direction, and therefore the stationary object is not the object that the user wants to use as the main subject, as in the application example 1 shown in FIG. 4A. Therefore, the CPU 151 assigns the lowest priority to a stationary object, which is a building, among the main subject candidates. Also, as shown in the graph of FIG. 4B, when the user is panning the image capture device 100, which repeatedly accelerates and decelerates sharply, a higher priority is assigned to the dog, among the remaining main subject candidates, than to the car. This is because the possibility that the car will suddenly change direction in the opposite direction is low (the degree of acceleration and deceleration is small).

(Example of classification and priority of types of motion of imaging device)
3 shows an example of a table illustrating the types of motion of the image capture device 100 in this embodiment, the types of objects detectable by the object detection unit 162 (hereinafter referred to as detectable objects), and the priority of each detectable object defined for each estimable type of motion. A specific example of estimating the types of motion of the image capture device 100 will be described later.

In this embodiment, as shown in FIG. 3, the camerawork of the imaging device 100 is classified into five types of movement. The object detection unit 162 detects the six types of object types shown in each column of FIG. 3. However, there are no limitations on the types of movement of the imaging device 100 to be classified and the types of objects to be detected. For example, the object detection unit 162 may detect objects that are likely to move randomly on the ground, such as dogs, cats, insects, and children, as one type of object. In other words, the type of object to be detected may be a type that corresponds to the expected movement of the object.

The priority assigned to each object corresponding to the estimated type of motion of the imaging device 100 is shown in each cell of the corresponding row in FIG. 3. In this embodiment, five levels of priority are assigned, from 1 to 5, and the higher the value, the higher the priority for imaging control.

The priority level assigned by this embodiment can also be used in combination with priority levels assigned by other criteria. For example, if there are two main subject candidates to which the same priority level is assigned by this embodiment, a higher priority level can be assigned to the main subject candidate closer to the center of the angle of view.

(Explanation of Means for Estimating the Type of Motion of the Imaging Device)
Next, a method for estimating the type of motion of the image capture device 100 in this embodiment will be described. In this embodiment, the angular velocity (two-dimensional in the yaw and pitch directions) output from the position and orientation change acquisition unit 161 is sampled at regular intervals and buffered in the RAM 154. A Fourier transform is performed on each of the sampled angular velocities during a predetermined period up to the current time to obtain frequency components, and the type of motion of the image capture device 100 is estimated according to the magnitude of each frequency component.

For example, as shown by the curve in the graph in FIG. 4(A), when the user pans the image capture device 100 at a slow speed with a small change in speed, the motion type of the image capture device 100 is determined to be slow panning (motion type 1 in FIG. 3).

Specifically, in this embodiment, first, the frequency components of all sampled angular velocities in the yaw direction are frequency analyzed, and as a result, it is determined whether low-frequency components below 3 Hz are predominant (whether they account for a certain percentage a or more of all frequency components).

If this frequency analysis determines that low frequency components are predominant, it is determined that the change in panning speed is small. Next, it is determined whether the panning speed is slow or fast. The panning speed is calculated, for example, by averaging the angular velocity in the yaw direction sampled within the above-mentioned specified period. If the calculated panning speed is equal to or greater than a preset threshold value α, the panning speed is determined to be fast, and the motion type of the imaging device 100 is determined to be motion type 2 in Figure 3. On the other hand, if it is less than the threshold value α, the panning speed is determined to be slow, and the motion type of the imaging device 100 is determined to be motion type 1 in Figure 3.

Also, as shown by the curve in the graph in FIG. 4(B), when the user is panning the image capture device 100 with repeated rapid acceleration and deceleration, the motion type of the image capture device 100 is determined to be rapid acceleration and deceleration in the Yaw direction (motion type 3 in FIG. 3).

Specifically, in this embodiment, if the result of the frequency analysis indicates that low frequency components below 3 Hz are not predominant, a determination is made as to whether the proportion of frequency components in a predetermined band, specifically between 3 Hz and 10 Hz, is equal to or greater than a predetermined proportion b. If the result of this determination is that the proportion is equal to or greater than the predetermined proportion b, it is determined that the change in panning speed is large. The reason why 10 Hz is set as the threshold here is to distinguish between camera work by the user and camera shake.

If it is determined that the change in the panning speed is large, it is then determined whether the panning in the yaw direction is accelerated or decelerated sharply or smoothly. Specifically, if the rate of change in the frequency component of the angular velocity in the yaw direction between samples is equal to or greater than a preset threshold value β, it is determined that the panning in the yaw direction is accelerated or decelerated sharply, and the motion type of the imaging device 100 is determined to be motion type 3 in FIG. 3. On the other hand, if it is less than the threshold value β, it is determined that the panning in the yaw direction is accelerated or decelerated smoothly, and the motion type of the imaging device 100 is determined to be motion type 4 in FIG. 3. The values of the predetermined ratio a, the predetermined ratio b, the threshold value α, and the threshold value β can be set appropriately depending on the manner of movement and speed of the object to be detected.

(Description of Object Detection Unit)
In this embodiment, the object detection unit 162 is configured using a CNN.

Figure 5 shows the basic structure of a CNN that detects a subject from an input two-dimensional captured image (hereinafter referred to as an input image) and a position map. The process flow proceeds from the input image in the direction of the arrow. The CNN is structured hierarchically with two layers, a feature detection layer (S layer) and a feature integration layer (C layer), as one set. In the CNN, the S layer detects the next feature based on the features detected in the previous layer. In addition, the features detected in the S layer are integrated in the C layer and output to the next layer as the detection result in that layer. The S layer is composed of feature detection cell planes, and detects different features for each feature detection cell plane. In addition, the C layer is composed of feature integration cell planes, and pools the detection results of the previous feature detection cell planes. In the following, unless there is a need to distinguish them, the feature detection cell planes and the feature integration cell planes are collectively referred to as feature planes. In this embodiment, the output layer, which is the final layer, does not use the C layer and is composed only of the S layer.

Figure 6 shows the details of the feature detection process and feature integration process. The feature detection process is performed on the feature detection cell plane. The feature integration process is performed on the feature integration cell plane. The feature detection cell plane is composed of multiple feature detection neurons. The feature detection neurons are connected to the C layer of the previous layer in a specified structure. The feature integration cell plane is also composed of multiple feature integration neurons, which are connected to the S layer of the same layer in a specified structure. In the Mth cell plane of the S layer of the Lth layer shown in Figure 6, the output value of the feature detection neuron at position (1) is denoted as (2). Each variable is expressed as follows:

In addition, in the Mth cell plane of layer C of the Lth hierarchy, the output value of the feature integration neuron at position (1) is denoted as (3). In this case, if the coupling coefficients of the respective neurons are (4) and (5), then each output value can be expressed as the following "Equation 2" and "Equation 3".

In equation 2, f is an activation function, which is a sigmoid function such as a logistic function or a hyperbolic tangent function, and can be realized, for example, by a tanh function. The internal state of the feature detection neuron at position (1) on the Mth cell surface of layer S of the Lth hierarchy is expressed by the above equation (6). Equation 3 is a simple linear sum equation without using an activation function. When no activation function is used as in equation 3, the internal state (7) of the neuron is equal to the output value (3). Furthermore, (8) in equation 2 and (9) in equation 3 are referred to as the connection destination output value of the feature detection neuron and the feature integration neuron, respectively.

The "ξ, ζ, u, v, n" in Equation 2 and Equation 3 will be explained. Position (1) corresponds to the position coordinate in the input image. For example, if the output value (2) is a high output value, it means that there is a high possibility that the feature to be detected in the Mth cell surface of the S layer of the Lth hierarchy exists at pixel position (1) of the input image. In addition, n means the nth cell surface of the C layer of the L-1th hierarchy in Equation 2, and is called the integration target feature number. Basically, a product-sum operation is performed for all cell surfaces that exist in the C layer of the L-1th hierarchy. "(u, v)" is the relative position coordinate of the coupling coefficient, and the product-sum operation is performed in a finite range "(u, v)" according to the size of the feature to be detected. Such a finite range of "(u, v)" is called the receptive field. Hereinafter, the size of the receptive field is called the receptive field size, and the receptive field size is expressed as the number of horizontal pixels x the number of vertical pixels in the combined range.

In addition, in equation 2, when L=1, that is, in the first S layer, (8) becomes the input image (10) or the input position map (11). Since the distribution of neurons and pixels is discrete, and the connection destination feature numbers are also discrete, "ξ, ζ, u, v, n" are not continuous variables, but take discrete values. Here, "ξ, ζ" are non-negative integers, "n" is a natural number, and u and v are integers, all of which are in a finite range.

(4) in Equation 2 is the coupling coefficient for detecting a specific feature, and by adjusting this coupling coefficient to an appropriate value, it becomes possible to detect the specific feature. This adjustment of the coupling coefficient is learning (machine learning), and in building a CNN, the coupling coefficient (2) is repeatedly modified using various test patterns so that an appropriate output value is obtained. This is how the coupling coefficient is adjusted.

Equation 3 (5) uses a two-dimensional Gaussian function and can be expressed as Equation 4 below.

Since "(u, v)" is a finite range, as in the explanation of feature detection neurons, this finite range is referred to as the receptive field, and the size of the receptive field range is referred to as the receptive field size. The receptive field size may be set to a value according to the size of the Mth feature in the Sth layer of the Lth hierarchy. "σ" in Equation 3 is a feature size factor, and may be set to a constant according to the receptive field size. For example, it is preferable to set it so that the outermost value of the receptive field is a value that can be considered to be almost 0.

By performing the above-mentioned calculations at each layer, subject detection is performed at the final layer, the S layer. This allows subject detection using CNN in this embodiment to be performed. In the above example, an example of subject detection using CNN with a captured image as an input image has been described. For subject detection using CNN, information such as acceleration sensor information and depth information may be added to the input image.

(How to learn)
A specific learning method will be described. In this embodiment, each coupling coefficient is adjusted by supervised learning. In supervised learning, a test pattern for learning is given to actually obtain the output value of a neuron, and the coupling coefficient (4) is corrected based on the relationship between the output value and a teacher signal (the desired output value that the neuron should output). In the learning of this embodiment, the feature detection layer of the final stage layer uses the least squares method, and the feature detection layer of the intermediate stage layer uses the backpropagation method to correct the coupling coefficient. Details of the coupling coefficient correction method such as the least squares method and the backpropagation method are omitted because they are conventional techniques (see S. Haykin, "Neural Networks A Comprehensive Foundation 2nd Edition", Prentice Hall, pp. 156-255, July 1998).

In this embodiment, many specific patterns that should be detected and those that should not be detected are prepared as test patterns for learning. Each test pattern is a set of an input image and a teacher signal.

When using a tanh function as the activation function, when a specific pattern to be detected is presented as a test pattern, a teacher signal is given to the neurons in the area where the specific pattern exists on the feature detection cell surface of the final stage layer so that their output becomes 1. Conversely, when a specific pattern that should not be detected is presented as a test pattern, a teacher signal is given to the neurons in the area where the specific pattern exists so that their output becomes -1.

The above steps construct a CNN as a trained model for detecting objects from two-dimensional captured images. In actual object detection, calculations are performed using the combination coefficients (4) constructed by training, and if the neuron output on the feature detection cell surface of the final layer is equal to or greater than a predetermined value, it is determined that an object is present there.

As described above, according to this embodiment, when an object is detected from a captured image, it is possible to provide an imaging device 100 that can infer the object that the user wants to be the main subject based on the type of movement of the imaging device 100 and automatically assign a priority to the object as a target for imaging control.

The following describes Example 2. Note that the configuration of the imaging device 100 is the same as that of Example 1, except that it further includes a distance calculation unit, so the same components are given the same reference numerals and duplicated descriptions are omitted.

The distance calculation units are connected to each other via bus 160 so that they can communicate with each other, and are units controlled by CPU 151, and calculate the distance from imaging device 100 to any subject in the captured image. An example of the distance calculation process will be described later. The generated distance information is stored in RAM 154 and is referenced by image processing unit 152.

(Overall processing flow)
The imaging control process in this embodiment will be described with reference to the flowchart in Fig. 7. Like the imaging control process in Fig. 2, this process is also started when the user half-presses the release button, which is the operation switch 156, and is executed by the CPU 151 based on a control program loaded into the RAM 154.

Note that steps S701 to S703 and S706 in FIG. 7 are the same as steps S201 to S203 and S206 in FIG. 2, respectively, so duplicate explanations will be omitted.

First, the processes of steps S701 to S703 are executed, and then in step S704, the shooting settings previously specified by the user for the imaging device 100 are acquired. The contents of the shooting settings in this embodiment will be described later.

Next, in step S705, a priority is assigned to each detected object as a subject of imaging control according to the shooting settings acquired in step S704. As in the first embodiment, imaging control in this embodiment includes frame display, exposure compensation, AF, and vibration reduction. At this time, the object with the highest assigned priority may be highlighted on the through image displayed on the monitor display in step S701, as in the first embodiment. This allows the user to determine whether the highest priority has been assigned to the object that the user wishes to use as the main subject among the objects present within the angle of view. Note that the highlighting in this embodiment is the same as in the first embodiment, so a duplicated explanation will be omitted.

Next, in step S706, similar to step S206 in FIG. 2, the process returns to step S701 and repeats the series of processes from step S701 until an end command is received.

(About shooting settings)
The shooting settings in this embodiment refer to at least one of the settings designated by the user to control the behavior of the imaging device 100. For example, the shooting settings include shooting modes designated by the user, such as shutter speed priority mode, aperture value priority mode, and strobe mode. If the user designates the shutter speed priority mode, the shooting settings also include the shutter speed designated at that time. If the user designates the aperture value priority mode, the shooting settings also include the aperture value designated at that time.

Shutter speed priority mode refers to a mode in which the user specifies the exposure time (shutter speed), and aperture priority mode refers to a mode in which the user specifies the aperture value. Additionally, strobe mode refers to a mode in which the user specifies that shooting will be performed using a strobe, which is an external device. Note that the shooting settings also include a mode in which the user specifies both the shutter speed and the aperture value. Additionally, when using aperture value priority mode, the user may also specify the ISO sensitivity, in which case the ISO sensitivity specified by the user is also included in the shooting settings.

Furthermore, the shooting settings in this embodiment also include the contents that the user intentionally determines for shooting with the imaging device 100, such as the focal length value and whether or not external devices such as a strobe or ND filter are used.

Next, various application examples of the imaging control process according to this embodiment will be described.

When it is determined in step S703 of FIG. 7 that multiple objects have been detected by the object detection unit 162, it is necessary to narrow down which objects are to be subject to imaging control. Therefore, in this embodiment, priorities are defined in advance for which objects detected by the object detection unit 162 are to be subject to imaging control in accordance with various shooting settings. Furthermore, in step S705 of FIG. 7, priorities are assigned to each of the multiple detected objects based on the defined priorities in accordance with the shooting settings acquired in step S704.

For example, in this embodiment, when the user specifies the shutter speed priority mode, if the object detection unit 162 detects multiple objects, the object that is assigned the highest priority in step S705 of FIG. 7 is determined to be the main subject. There are two possible use cases for the user to use the shutter speed priority mode. One is to suppress blurring of the subject when photographing a moving object, and the other is to prevent camera shake, which occurs when the shutter speed is automatically determined under conditions of insufficient light. Therefore, if the shooting setting is the shutter speed priority mode and the conditions are not insufficient, it is determined that the user intends to photograph a moving object, and still objects are excluded from the options for imaging control targets. In this embodiment, if the photometric value obtained by the photometric sensor (not shown) is equal to or greater than a predetermined threshold value, or if the shutter speed set by the user is equal to or greater than threshold value 1 (if the shutter speed is the same speed as threshold value 1 or faster), it is determined that the conditions are not insufficient.

(Application Example 1 of Example 2)
Hereinafter, a first application example of the second embodiment will be described, which is executed when the user specifies the shutter speed priority mode and the condition is not one of insufficient light.

In this application example, three classes are predefined as classes for classifying each object detectable by the object detection unit 162 according to the expected movement of each object: "stationary object", "slowly moving object", and "fastly moving object". When multiple objects are detected by the object detection unit 162, the lowest priority is assigned to the detected objects classified as "stationary object". Furthermore, if the shutter speed is equal to or greater than threshold 2 (> threshold 1), the highest priority is assigned to the object classified as "fastly moving object". Furthermore, if the shutter speed is less than threshold 2, the highest priority is assigned to the object classified as "slowly moving object".

Conventionally, there is known a technique for determining whether a subject is moving by continuously detecting the subject from a live view image. Therefore, when the user specifies the shutter speed priority mode, it is conceivable to use this technique to determine which subjects are moving quickly and select them as the main subject.

However, subjects that can move quickly do not always move quickly. For example, subjects such as cats, birds, and insects may stay still before you take a picture, but may quickly run away the moment you try to take a picture.

In such a case, the user predicts the speed at which the subject will move, and then selects the shutter priority mode and sets the shutter speed before executing the imaging control process of FIG. 7. Therefore, in this application example, when the user intentionally sets the shutter speed, the main subject is not selected based on the live view image, but is selected based on the set shutter speed and the speed at which the subject is likely to move.

The priority level assigned in this application example can also be used in combination with another priority level assigned by other means. For example, if there are two main subject candidates assigned the same priority level in this application example, the main subject candidate closer to the center of the angle of view can be given a higher priority.

(Application Example 2 of Example 2)
Hereinafter, a second application example of this embodiment will be described, which is executed when the user has designated the shutter speed priority mode and the condition is not one of insufficient light.

In the above-described application example 1 of this embodiment, three classes, namely, stationary objects, objects moving at a low speed, and objects moving at a high speed, were defined in advance to classify each object that can be detected by the object detection unit 162. In contrast, this application example uses the moving speed v' on the image sensor 141 (captured image) as a more detailed class.

Specifically, in this application example, a unique speed vi (i = 1, 2, ..., C) is defined for each type of object that can be detected by the object detection unit 162. Here, C is the number of types of detectable objects. In this application example, a representative speed value in real space in a scene where each object that can be detected by the object detection unit 162 is often photographed is defined as the speed _vi , but the maximum speed value at which each object can move may also be defined as the speed vi.

For each object detected by the object detection unit 162, the CPU 151 acquires the subject distance L from the imaging device 100 from the distance calculation unit, and also acquires the focal length f from the zoom control unit 113. Using these values, the CPU 151 scales the speed v _i of each object detected by the object detection unit 162 to a moving speed v' on the imaging element 141 as shown in the following "Equation 5" (see FIG. 8).

In this application example, it is assumed that the optimal shutter speed for photographing a subject is roughly proportional to the moving speed v' on the image sensor 141, and the proportionality coefficient is set to A. A is determined by the pixel pitch of the image sensor 141 and the allowable degree of blur, and may be adjusted independently by the user depending on the latter. If the shutter speed included in the shooting settings acquired in step S704 is S, a higher priority is assigned to detected objects whose S and Av' are close.

(Application Example 3 of Example 2)
Hereinafter, a third application example of this embodiment will be described, which is executed when the user has designated the shutter speed priority mode and the condition is not one of insufficient light.

In this application example, the moving speed v" on the captured image is used as a more detailed class than in application example 1 of this embodiment described above.

Specifically, in the above-mentioned application example 2 of this embodiment, a specific speed in the real world is defined for each type of object that can be detected by the object detection unit 162. In contrast, in this application example, a relative speed with respect to the subject size is defined for each type of object that can be detected by the object detection unit 162.

In addition, in this application example, the object detection unit 162 outputs, as the detection result, information on the type of object and the area in the captured image in which the object is located. More specifically, this area information is output as a bounding box (center coordinates, width, and height in the captured image). In addition, in this application example, the relative speed of each object type that can be detected by the object detection unit 162 with respect to the size of the area in which the object is located is defined in advance. In this application example, the width included in the bounding box is used as this size, but the height may also be used.

For each object detected by the object detection unit 162, the CPU 151 multiplies the width included in the bounding box output from the object detection unit 162 by the relative speed, and calculates a moving speed v" that indicates how many pixels per second the object can move on the captured image.

For example, if the width of an object output by the object detection unit 162 is W and the predefined relative speed of the object type is u _i , then the moving speed v" on the captured image is Wu _i [pixels/second] (see FIG. 9 ). In this embodiment, it is assumed that the optimal shutter speed for photographing a subject is roughly proportional to the moving speed v" on the captured image, and its proportionality coefficient is B. B may be determined based on the allowable degree of blur, or may be adjusted by the user. Alternatively, B may be calculated based on the angular velocity obtained from the position and orientation change acquisition unit 161 and may be varied accordingly. If the shutter speed included in the shooting settings acquired in step S704 is S, a higher priority is assigned to detected objects whose S and Bv" are close to each other.

(Example of aperture value specified by user)
Hereinafter, a fourth application example of this embodiment will be described.

Application examples 1 to 3 of this embodiment are application examples in which the user specifies the shutter speed priority mode. In contrast, this application example is an application example in which the user specifies the aperture value in the aperture value priority mode.

Below, we will use Figure 10 to explain this application example, in which when a user specifies an aperture value in aperture priority mode, the main subject is determined according to that value.

In this application example, the aperture value (F-number) of the aperture 103 is classified into three predefined aperture levels: small, medium, and large.

The CPU 151 determines which aperture level the user-specified aperture value included in the shooting settings acquired in step S704 belongs to, and assigns a priority to each object type detected by the object detection unit 162 based on the table shown in FIG. 10.

(Example of a strobe)
Hereinafter, application example 5 of this embodiment will be described.

In application examples 1 to 4 of this embodiment, the priority of which object is to be the subject of imaging control is determined according to the imaging settings specified by the user for the main body of the imaging device 100. In contrast, this application example is an application example in which the user specifies a strobe mode, which is a mode in which a strobe, which is an external device of the imaging device 100, is used during imaging.

The following describes an example of this application, in which, when the user has specified the strobe mode, the priority is determined as to which object detected by the object detection unit 162 is to be subject to imaging control.

One use case in which a user uses a strobe when shooting with the imaging device 100 is when the user intends to shoot a fast-moving subject without blur. Therefore, in this application example, two classes are predefined for classifying objects detectable by the object detection unit 162 in strobe mode: "fast-moving objects" within an angle of view that is highly suitable for strobe shooting, and "other objects."

If the shooting setting acquired in step S704 is strobe mode, and both an object classified as a "fast-moving object" and an object classified as an "other object" are detected, the former is given a higher priority than the latter.

If the shooting setting acquired in step S704 is not the strobe mode, the same priority is assigned to both. In this case, as in another application example of this embodiment, for example, application example 1 of this embodiment, a further priority is assigned to each object detected by the object detection unit 162 with reference to the shutter speed.

(Distance Generation)
Next, a method for the distance calculation section to calculate the distance (subject distance) from each object detected by the object detection section 162 to the imaging device 100 will be described.

In this embodiment, a distance map is first generated based on the parallax image, and the value of the distance map corresponding to the coordinates of each detected object is set as the subject distance of that object.

Specifically, the distance map value is calculated by detecting the amount of image shift by performing correlation calculation processing on a pair of parallax images with horizontal parallax, and multiplying the amount of image shift by a predetermined conversion coefficient. In addition, the amount of image shift is detected by performing correlation calculation for each small block obtained by dividing a pair of parallax images into small regions, as disclosed in, for example, JP 2008-15754 A.

The distance map may be generated without using a parallax image. For example, the position of the focus lens 131 where the contrast evaluation value is maximized may be calculated as the distance map value for each pixel of the image sensor 141. The distance map value may be calculated for all pixels of the image sensor 141, or may be calculated only for pixels in a partial area where an object detection result exists.

There are no limitations on the method of acquiring the parallax image, but in this embodiment, the image sensor 141 has multiple pixels made up of multiple photoelectric conversion elements that share one microlens, and the parallax image is acquired using the image sensor 141. The configuration and optical principle of the image sensor 141 in this case can be based on known technology such as that disclosed in JP 2008-15754 A. Note that the image sensor 100 may be a multi-lens camera such as a stereo camera to acquire the parallax image, or data of the parallax image captured by any method may be acquired from a storage medium or an external device.

The above-mentioned embodiments are merely specific examples intended to aid in understanding the present invention, and are not intended to limit the present invention in any way to the above-mentioned embodiments. All embodiments falling within the scope of the claims are encompassed by the present invention. In other words, the present invention can be modified and changed in various ways within the scope of its gist. The present invention can also be realized by supplying a program that realizes one or more functions of each of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

REFERENCE SIGNS LIST 100 Imaging device 103 Aperture 104 Aperture motor 105 Aperture control unit 111 Zoom lens 112 Zoom motor 113 Zoom control unit 141 Imaging element 151 CPU
161 Position and orientation change acquisition unit 162 Object detection unit

Claims (18)

1. An imaging device, comprising:
an object detection means for detecting an object included in an image captured by the imaging device and detecting a type of the detected object;
a motion type estimation means for estimating a motion type of the imaging device;
a priority assignment means for assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with a motion type of the imaging device estimated by the motion type estimation means and the types of each of the plurality of objects detected by the object detection means when the plurality of objects are detected by the object detection means ;
a measuring means for measuring a change in position and orientation of a main body of the imaging device,
the motion type estimation means estimates a motion type of the imaging device based on an output of the measurement means for a predetermined period of time;
the type of object detectable by the object detection means is a type corresponding to an expected manner of movement of each object detectable by the object detection means,
the motion type estimation means calculates a moving speed of the imaging device within the predetermined period based on an output of the measurement means during the predetermined period;
the priority assignment means assigns a priority to each of the plurality of objects detected by the object detection means as a subject of the imaging control in accordance with the calculated moving speed of the imaging device during the specified period and an expected manner of movement of each of the plurality of objects for each type . 1. An imaging device, comprising:
an object detection means for detecting an object included in an image captured by the imaging device and detecting a type of the detected object;
a motion type estimation means for estimating a motion type of the imaging device;
a priority assignment means for assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with a motion type of the imaging device estimated by the motion type estimation means and the types of each of the plurality of objects detected by the object detection means when the plurality of objects are detected by the object detection means ;
a measuring means for measuring a change in position and orientation of a main body of the imaging device,
the motion type estimation means estimates a motion type of the imaging device based on an output of the measurement means for a predetermined period of time;
the type of object that can be detected by the object detection means is a type corresponding to a degree of acceleration/deceleration expected for the object,
the motion type estimation means calculates a degree of acceleration/deceleration of the imaging device within the predetermined period based on an output of the measurement means during the predetermined period;
the priority assignment means assigns a priority to each of the plurality of objects detected by the object detection means as a subject of the imaging control in accordance with the calculated degree of acceleration/deceleration of the imaging device during the specified period and the degree of acceleration/deceleration expected for each type of the plurality of objects . 1. An imaging device, comprising:
an object detection means for detecting an object included in an image captured by the imaging device and detecting a type of the detected object;
a motion type estimation means for estimating a motion type of the imaging device;
a priority assignment means for assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with a motion type of the imaging device estimated by the motion type estimation means and the types of each of the plurality of objects detected by the object detection means when the plurality of objects are detected by the object detection means ;
a measuring means for measuring a change in position and orientation of a main body of the imaging device;
an analysis means for analyzing a frequency component of the output of the measurement means for the predetermined period of time,
The motion type estimating means estimates a motion type of the imaging device based on an output of the measuring means for a predetermined period of time and frequency components analyzed by the analyzing means . 4. The image pickup apparatus according to claim 3 , wherein the motion type estimating means calculates a degree of acceleration or deceleration of the image pickup apparatus in accordance with a ratio of frequency components in a predetermined band obtained by the analyzing means. 5. The imaging device according to claim 4, wherein the predetermined frequency band is equal to or greater than 3 Hz and less than 10 Hz. 1. An imaging device, comprising:
an object detection means for detecting an object included in an image captured by the imaging device;
A type detection means for detecting the types of the plurality of objects;
an acquisition means for acquiring a shooting setting designated by a user for the imaging device;
a priority assigning means for, when a plurality of objects are detected by the object detection means, classifying each of the plurality of objects into one of a plurality of predefined classes, and assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the photography setting acquired by the acquisition means and the class into which each of the plurality of objects is classified ,
the class is a moving speed on the captured image calculated based on a speed defined for each type of object that can be detected by the object detection means,
The priority assignment means assigns a priority to each of the plurality of objects detected by the object detection means for use in the imaging control in accordance with the calculated moving speed of each of the plurality of objects on the captured image . 1. An imaging device, comprising:
an object detection means for detecting an object included in an image captured by the imaging device;
an acquisition means for acquiring a shooting setting designated by a user for the imaging device;
a priority assigning means for classifying each of the plurality of objects into one of a plurality of predefined classes when a plurality of objects are detected by the object detection means, and assigning a priority for making each of the plurality of objects a subject of imaging control to each of the plurality of objects according to the photographing settings acquired by the acquisition means and the class into which each of the plurality of objects is classified;
a type detection means for detecting a type of the plurality of objects,
the class is a moving speed on the captured image calculated based on a speed defined for each type of object that can be detected by the object detection means,
The imaging device is characterized in that, when one of the shooting settings acquired by the acquisition means is a shutter speed priority mode, the priority assignment means assigns a priority to each of the multiple objects detected by the object detection means as a subject of the imaging control, depending on the shutter speed included in the shooting setting acquired by the acquisition means and the class into which each of the multiple objects is classified . 8. The imaging device according to claim 7, wherein the speed defined for each type of object detectable by the object detection means is a moving speed in real space defined for each type of object detectable by the object detection means. 9. The imaging device according to claim 8, further comprising a first acquisition means for acquiring a focal length at the time of shooting of each of the plurality of objects, wherein for each of the plurality of objects, a moving speed in the captured image is calculated based on a speed defined as a moving speed in real space of the type of object and the acquired focal length. a second acquisition unit that acquires a subject distance from the image capture device at the time of photographing each of the plurality of objects;
9. The imaging device according to claim 7 or 8, wherein for each of the plurality of objects, a moving speed in the captured image is calculated based on a speed defined as a moving speed in real space of the type of object and the acquired subject distance. 1. An imaging device, comprising:
an object detection means for detecting an object included in an image captured by the imaging device;
an acquisition means for acquiring a shooting setting designated by a user for the imaging device;
a priority assigning means for classifying each of the plurality of objects into one of a plurality of predefined classes when a plurality of objects are detected by the object detection means, and assigning a priority for making each of the plurality of objects a subject of imaging control to each of the plurality of objects according to the photographing settings acquired by the acquisition means and the class into which each of the plurality of objects is classified;
a type area detection means for detecting the types of the plurality of objects and the areas in which the objects are located in the captured image,
The speed defined for each type of object detectable by the object detection means is a relative speed with respect to a subject size defined for each type of object detectable by the object detection means,
An imaging device characterized in that, for each of the plurality of objects, a moving speed on the captured image is calculated based on a speed defined as a relative speed of the type detected by the type area detection means and a size of the area detected by the type area detection means . 12. The imaging device according to claim 1, further comprising a display unit that displays the captured image in a highlighted manner an object that has been assigned the highest priority by the priority assignment unit among the objects detected by the object detection unit. The imaging device according to claim 1 , wherein the object detection means is configured using a trained model. A control method for an imaging device, comprising:
an object detection step of detecting an object included in an image captured by the imaging device and detecting a type of the detected object;
a motion type estimating step of estimating a motion type of the imaging device;
a priority assignment step of assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the type of motion of the imaging device estimated in the motion type estimation step and the types of each of the plurality of objects detected in the object detection step, when a plurality of objects are detected in the object detection step ;
a measuring step of measuring a change in position and orientation of a main body of the imaging device,
the motion type estimating step estimates a motion type of the imaging device based on an output of the measuring step during a predetermined period of time;
a type of object detectable in the object detection step according to an expected manner of movement of each object detectable in the object detection step;
the motion type estimating step calculates a moving speed of the imaging device within the predetermined period based on an output of the measuring step during the predetermined period;
a control method characterized in that in the priority assignment step, a priority for making each of the plurality of objects detected in the object detection step a subject of the imaging control is assigned to each of the plurality of objects detected in the object detection step in accordance with the calculated moving speed of the imaging device during the specified period and an expected manner of movement of each of the plurality of objects for each type . A control method for an imaging device, comprising:
an object detection step of detecting an object included in an image captured by the imaging device and detecting a type of the detected object;
a motion type estimating step of estimating a motion type of the imaging device;
a priority assignment step of assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the type of motion of the imaging device estimated in the motion type estimation step and the types of each of the plurality of objects detected in the object detection step, when a plurality of objects are detected in the object detection step ;
a measuring step of measuring a change in position and orientation of a main body of the imaging device,
the motion type estimation step estimates a motion type of the imaging device based on an output of the measurement step during a predetermined period of time;
a type of the object that can be detected in the object detection step is a type corresponding to a degree of acceleration/deceleration expected for the object,
the motion type estimating step calculates a degree of acceleration/deceleration of the imaging device within the predetermined period based on an output of the measuring step during the predetermined period;
a control method for an imaging device, characterized in that in the priority assignment step, a priority for making the object subject to imaging control is assigned to each of the plurality of objects detected in the object detection step, in accordance with the calculated degree of acceleration/deceleration of the imaging device during the specified period and the degree of acceleration/deceleration expected for each type of the plurality of objects . A control method for an imaging device, comprising:
an object detection step of detecting an object included in an image captured by the imaging device and detecting a type of the detected object;
a motion type estimating step of estimating a motion type of the imaging device;
a priority assignment step of assigning a priority to each of the plurality of objects as a subject of imaging control in accordance with the type of motion of the imaging device estimated in the motion type estimation step and the types of each of the plurality of objects detected in the object detection step, when a plurality of objects are detected in the object detection step ;
a measuring step of measuring a change in position and orientation of a main body of the imaging device;
and an analysis step of analyzing a frequency component of the output of the measurement step for the predetermined period,
A method for controlling an imaging device , wherein the motion type estimating step estimates a motion type of the imaging device based on an output of the measuring step for a predetermined period and frequency components analyzed in the analyzing step . A control method for an imaging device, comprising:
an object detection step of detecting an object included in an image captured by the imaging device;
a type detection step of detecting types of the plurality of objects;
an acquisition step of acquiring a shooting setting designated by a user for the imaging device;
a priority assignment step of, when a plurality of objects are detected by the object detection step, classifying each of the plurality of objects into one of a plurality of predefined classes, and assigning a priority to each of the plurality of objects as a subject of imaging control according to the photography setting acquired by the acquisition step and the class into which each of the plurality of objects is classified ,
the class is a moving speed on the captured image calculated based on a speed defined for each type of object detectable in the object detection step,
a control method for an imaging device, characterized in that in the priority assignment step, a priority for making the object subject to imaging control is assigned to each of the plurality of objects detected in the object detection step in accordance with the calculated moving speed on the captured image . A program for executing the control method according to any one of claims 14 to 17 .

Images