JP2019186911A - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Abstract

To provide an image processing apparatus capable of detecting a subject suitable for an amount of blur and clarity of a main subject in subject detection using machine learning, an image processing method, and an imaging apparatus.SOLUTION: Using parameters generated on the basis of machine learning, subject detection processing is applied to image data. In the subject detection processing, a parameter selected from a plurality of parameters stored in advance is used. According to an amount of blur and clarity of a subject, the subject detection processing is applied using different parameters.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus, and more particularly to a subject detection technique.

  A subject detection technique for automatically detecting a specific subject pattern from an image is very useful. Patent Document 1 discloses an imaging apparatus that detects a region corresponding to a specific subject pattern such as a human face from a photographed image and optimizes the focus and exposure of the detected region.

  It is also known to learn and recognize a subject in an image using a technique called deep learning (Non-Patent Document 1). The convolutional neural network (CNN) is a typical technique for deep learning. In general, CNN is a multi-layer structure in which a convolution layer that spatially integrates local features of an image, a pooling layer or sub-sampling layer that compresses features in the spatial direction, and a fully connected layer, an output layer, etc. Have Since the CNN can acquire a complicated feature expression through stepwise feature conversion with a multi-layer structure, the category recognition of the subject in the image and the subject detection can be performed with high accuracy based on the feature representation.

JP 2005-318554 A

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, “ImageNet classification with deep convolutional neural networks”, NIPS'12 Proceedings of the 25th International Conference on Neural Information Processing Systems-Volume 1, PP.1097-1105

  In an imaging apparatus that focuses on a subject area detected from a captured image as in Patent Document 1, the request for subject detection differs before and after determination of a main subject (a subject to be focused). Before determining the main subject, it is better to be able to detect the subject to be detected without omission regardless of the size of the blur. On the other hand, after determining the main subject, it is not necessary to detect a subject with large blur.

  However, conventionally, in subject detection using machine learning, a method for realizing subject detection suitable before determination of the main subject and subject detection suitable after determination of the main subject has not been proposed. In other words, a method for subject detection suitable for the amount of blur (or definition) of the main subject has not been proposed. The blur amount and sharpness of the main subject are not limited to the degree of focus, but also vary depending on the degree of subject blurring and camera shake, the magnitude of image noise, and the like.

  The present invention has been made in view of the problems of the prior art. An object of the present invention is to provide an image processing device, an image processing method, and an imaging device capable of realizing subject detection suitable for a blur amount and a sharpness of a main subject in subject detection using machine learning.

  The above-described object is provided by subject detection means for applying subject detection processing to an image using parameters generated based on machine learning, storage means for storing a plurality of parameters used for subject detection processing, and storage means. Selection means for selecting a parameter to be used by the subject detection means from the stored parameters, and the selection means selects different parameters depending on whether the main subject is determined or not. This is achieved by an image processing apparatus characterized by the following. Further, in the same configuration, the selecting means selects different parameters depending on whether or not it is estimated that the shake is large, whether or not the position and orientation change is large, and whether or not the gain applied to the image is large. This is achieved by the featured image processing apparatus.

  According to the present invention, in subject detection using machine learning, an image processing apparatus and an image processing method capable of realizing subject detection suitable before the main subject is determined and subject detection suitable after the main subject is determined, and An imaging device can be provided. In addition, it is possible to provide an image processing device, an image processing method, and an imaging device capable of realizing subject detection suitable for the movement status of the main subject, the movement status of the imaging device, and the imaging conditions.

1 is a schematic vertical sectional view of a digital single-lens reflex camera as an example of an image processing apparatus according to a first embodiment. 1 is a block diagram showing an example of a functional configuration of a digital single lens reflex camera according to a first embodiment. The flowchart regarding the outline | summary of imaging | photography operation | movement which concerns on 1st Embodiment. 5 is a flowchart regarding subject detection processing according to the first embodiment. FIG. 3 is a schematic diagram illustrating a configuration example of a CNN used by a subject detection unit according to the first embodiment. FIG. 6 is a schematic diagram illustrating a configuration of part of the CNN in FIG. 5. The block diagram which shows the function structural example of the digital single-lens reflex camera which concerns on 2nd-4th embodiment. The schematic diagram which shows an example of the motion vector calculation process which concerns on 2nd Embodiment. 10 is a flowchart regarding subject detection processing according to the second embodiment. 10 is a flowchart regarding subject detection processing according to the third embodiment. 10 is a flowchart regarding subject detection processing according to the fourth embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Further, although a plurality of features are described in the embodiment, all of them are not necessarily essential to the invention, and a plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same reference numerals are assigned to the same or similar components, and duplicate descriptions are omitted.

  In the following embodiments, a case where the present invention is implemented by a digital single lens reflex camera (DSLR) will be described. However, the present invention can be implemented with any electronic device capable of handling image data, and a digital single-lens reflex camera is only an example of an image processing apparatus according to the present invention. Electronic devices that can implement the present invention include, but are not limited to, personal computers, smartphones, tablet terminals, game machines, robots, and the like.

<< First Embodiment >>
● (Configuration of imaging device)
FIG. 1 is a vertical sectional view of a digital single-lens reflex camera (DSLR) 100 according to this embodiment. FIG. 2 is a block diagram showing a functional configuration example of the DSLR 100. Like reference numerals refer to like elements throughout the drawings.

  The DSLR 100 includes a main body 101 and a photographing lens 102 (interchangeable lens) that can be attached to and detached from the main body 101. Mount contact groups 115 are provided on the attachment and detachment portions (mounts) of the main body 101 and the photographing lens 102, respectively. When the photographic lens 102 is attached to the main body 101, the mount contact group 115 comes into contact, and electrical connection between the photographic lens 102 and the main body 101 is established.

  The system control unit 201 includes one or more programmable processors, a ROM 2011, and a RAM 2012, and controls operations of the main body 101 and the photographing lens 102 by reading a program stored in the ROM 2011 into the RAM 2012 and executing it. The ROM 2011 stores various setting values, GUI data, and the like in addition to programs executed by the system control unit 201.

  The photographing lens 102 is provided with a focus lens 113 that adjusts the focusing distance, and a diaphragm 114 (and a motor or actuator that drives them) that adjusts the amount of light incident on the main body 101. The driving of the focus lens 113 and the diaphragm 114 is controlled by the camera body 101 through the mount contact group 115.

  The main mirror 103 and the sub mirror 104 constitute a quick return mirror. A part of the main mirror 103 has a reflectance (transmittance) controlled to separate a light beam incident from the photographing lens 102 into a light beam traveling toward the finder optical system (upper side in the drawing) and a light beam traveling toward the sub mirror 104. .

  FIG. 1 shows a state when the optical viewfinder is used (when not photographing), and the main mirror 103 is positioned in the optical path of the light beam incident from the photographing lens 102. In this state, the reflected light of the main mirror 103 enters the finder optical system, and the light beam bent by the pentaprism 107 is emitted from the eyepiece 109. Therefore, the user can see the optical subject image by looking into the eyepiece 109.

  Further, the transmitted light of the main mirror 103 is reflected by the sub mirror 104 and enters the AF sensor 105. The AF sensor 105 forms a secondary imaging surface of the photographing lens 102 on the line sensor, and generates a pair of image signals (focus detection signals) that can be used for focus detection by the phase difference detection method. The generated focus detection signal is transmitted to the system control unit 201. The system control unit 201 obtains the defocus amount of the focus lens 113 using the focus detection signal, and controls the drive direction and drive amount of the focus lens 113 based on the defocus amount.

  The focus plate 106 is disposed on the planned imaging plane of the photographing lens 102 in the finder optical system. A user looking into the eyepiece 109 observes the optical image formed on the focusing plate 106. In addition to the optical image, shooting information such as shutter speed and aperture value can also be provided.

  The photometric sensor 108 generates an image signal (exposure control signal) from the incident light beam and transmits it to the system control unit 201. The system control unit 201 performs automatic exposure control using the received exposure control signal, and controls subject detection by a subject detection unit 204 described later. The photometric sensor 108 is an image sensor in which pixels including a photoelectric conversion unit are two-dimensionally arranged. The subject detection unit 204 applies subject detection processing to the image signal output from the photometric sensor 108, but may apply subject detection processing to the image signal output from the image sensor 111.

  When the image sensor 111 is exposed, the main mirror 103 and the sub mirror 104 move out of the optical path of the light beam incident from the photographing lens 102. Further, a focal plane shutter 110 (hereinafter simply referred to as a shutter) is opened.

  Pixels including a photoelectric conversion unit are two-dimensionally arranged in the image sensor 111, subject optical images formed by the photographing lens 102 are photoelectrically converted by each pixel, and an image signal is transmitted to the system control unit 201. The system control unit 201 generates image data from the received image signal, saves it in the image storage unit 202, and displays it on the display unit 112 such as an LCD. The image data generated by the image sensor 111 may also be supplied to the subject detection unit 204 for subject detection. Note that the system control unit 201 may perform focus detection by a contrast method using image data. Note that the system control unit 201 may perform focus detection by a contrast method using image data.

  The operation unit 203 is a generic name for a group of input devices that are provided in the main body 101 and the photographing lens 102 and can be operated by the user. A release button, a power switch, a direction key, an enter button, a menu button, an operation mode selection dial, and the like are specific examples of the input device included in the operation unit 203, but are not limited thereto. The operation of the operation unit 203 is detected by the system control unit 201. The release button includes a switch SW1 that is turned on by a half-press operation and a switch SW2 that is turned on by a full-press operation.

  For example, when a half-press operation of the release button (SW1 is turned on) is detected, the system control unit 201 starts a still image shooting preparation operation. The shooting preparation operation is, for example, an operation related to automatic focus detection (AF) and automatic exposure control (AE). In addition, when a full pressing operation of the release button (ON of SW2) is detected, the system control unit 201 performs still image shooting and recording operations. The system control unit 201 displays an image obtained by shooting on the display unit 112 for a certain period of time.

  In addition, during moving image shooting (in shooting standby state or during moving image recording), the system control unit 201 displays the moving image obtained by shooting on the display unit 112 in real time, thereby displaying the display unit 112 in an electronic viewfinder (EVF). To function as. A moving image and its frame image that are displayed when the display unit 112 functions as an EVF are referred to as a live view image or a through image. Whether to shoot still images or moving images can be selected through the operation unit 203, and the system control unit 201 switches the control method of the camera body 101 and the photographing lens 102 between still image shooting and moving image shooting.

  The subject detection unit 204 is configured by a GPU (Graphic Processing Unit). The GPU is originally a processor for image processing, but has a plurality of product-sum operation units and is good at matrix calculation, so it is often used as a processor that performs processing for learning. In general, GPU is also used in the process of performing deep learning. For example, as the subject detection unit 204, a Jetson TX2 module manufactured by NVIDIA can be used. As the subject detection unit 204, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like may be used.

  The subject detection unit 204 applies subject detection processing to the supplied image data using one learning model selected by the system control unit 201 among a plurality of learning models stored in the learning model storage unit 205. Details of the subject detection process will be described later. The learning model storage unit 205 may be a rewritable nonvolatile memory, for example, or may be a part of the ROM 2011. In the present embodiment, the learning model storage unit stores detection learning models 206 and 207 corresponding to characteristics required for subject detection. The system control unit 201 determines the main subject region from the subject region detected by the subject detection unit 204, and optimizes the focus and exposure for the main subject region. Specifically, the exposure conditions can be determined so that the main subject area is properly exposed, or the focus can be adjusted so that the main subject area is in focus, but the present invention is not limited thereto.

(Dictionary switching for subject detection)
The DSLR 100 (system control unit 201) of the present embodiment uses the subject area detected by the subject detection unit 204 for determining exposure conditions and focus detection. More specifically, the system control unit 201 determines a main subject from the subjects detected by the subject detection unit 204, and uses the main subject region for determination of exposure conditions and focus detection. Therefore, the characteristics required for subject detection differ before and after the main subject is determined. Note that the main subject may be determined by the system control unit 201 according to predetermined conditions (for example, the position and size of the subject region), or may be determined according to the user's selection.

  The image shot before the main subject is determined is an image that is focused so that the center of the screen is in focus, for example, and the user may not be focused on the subject that the user wants to be the main subject. Therefore, the subject detection performed before the main subject is determined requires that the subject to be detected can be detected even if the amount of blur is large. In other words, subject detection that gives priority to suppression of detection omission over suppression of erroneous detection is required.

  On the other hand, when the main subject is determined, focus detection is performed so that the area of the main subject is in focus thereafter, so that the amount of blur of the main subject in the image obtained thereafter becomes small. For this reason, after the main subject is determined, even if the subject is a detection target, the value of detecting a subject with a large blur amount decreases. In other words, subject detection that gives priority to suppression of false detection over suppression of detection omission is required.

  There is a tendency that the probability of erroneous detection of an area in which the subject subject is not captured increases as the subject with a larger amount of blur is detected. This is because, as the blur amount of the subject increases, image variations similar to the subject increase and the characteristics of the subject are lost. As described above, it is important to detect from a subject with a small amount of blur to a subject with a large amount of blur before the main subject is determined.After the main subject is determined, erroneous detection is suppressed and a subject with a small amount of blur is accurately detected. It is important to detect.

  The subject detection unit 204 of the present embodiment applies subject detection using processing parameters based on a learning model generated in advance through machine learning to an image. In this embodiment, the learning model storage unit 205 stores a detection learning model 206 for subject detection before main subject determination and a detection learning model 207 for subject detection after main subject determination. A detection learning model 206 for detecting a subject before determining a main subject is generated by machine learning using an image of a large subject from a subject with a small amount of blur. In addition, the detection learning model 207 for subject detection after the main subject is determined is generated by machine learning that places importance on subject images with a small amount of blur.

  The system control unit 201 switches the subject detection characteristics of the subject detection unit 204 by switching the learning model used by the subject detection unit 204 before and after the main subject is determined. The system control unit 201 uses the detection learning model 206 for detecting a subject before determining the main subject before the determination of the main subject, and uses the detection learning model 207 for detecting the subject after determining the main subject after determining the main subject. Switch to.

(Imaging operation)
Next, with reference to FIG. 3 and FIG. 4, the photographing operation of the DSLR 100 of the present embodiment will be described.
FIG. 3 is a flowchart relating to the outline of the photographing operation, and the processing of each step is realized by the execution of a program read from the ROM 2011 into the RAM 2012 by the programmable processor of the system control unit 201. Here, it is assumed that the power source of the main body 101 is ON and is in a photographing standby state. In the shooting standby state, the system control unit 201 performs a predetermined operation such as, for example, shooting a moving image and displaying the obtained moving image on the display unit 112 to function as an EVF.

  In step S301, the system control unit 201 detects the state of the switches SW1 and SW2 included in the release button. If any of the switches SW1 and SW2 is on, the process proceeds to step S302. On the other hand, if both the switches SW1 and SW2 are off, the system control unit 201 repeatedly executes S301.

  In step S <b> 302, the system control unit 201 performs exposure processing (charge accumulation) of the photometric sensor 108. The exposure process of the photometric sensor 108 is realized by performing charge accumulation for a predetermined time using a so-called electronic shutter. The system control unit 201 controls the operation of the photometric sensor 108, performs charge accumulation for a predetermined time, and reads an image signal (exposure control signal) from the photometric sensor 108. The system control unit 201 also performs exposure processing (charge accumulation) for the AF sensor 105 and reads an image signal (focus detection signal).

  In step S <b> 303, the system control unit 201 supplies an exposure control signal to the subject detection unit 204 and acquires a subject detection result from the subject detection unit 204. Further, the system control unit 201 determines the main subject based on the subject detection result. The main subject can be determined by an arbitrary method. The system control unit 201 may automatically determine the main subject. The detected subject area is shown to the user, and the subject in the user-selected area is set as the main subject. You may decide. When the system control unit 201 automatically determines the main subject, for example, it can be determined based on the position and size of the region of the subject. For example, the subject closest to the center of the image or the subject with the largest area can be determined as the main subject. It should be noted that the main subject can be determined by considering other conditions such as the reliability of subject detection and past detection results, or by combining a plurality of conditions. Details of the subject detection process will be described later with reference to FIG.

  In step S304, the system control unit 201 selects a focus detection region that is closest to the position of the main subject determined in step S303 from the selectable focus detection regions. Then, the focus state (defocus amount and direction) of the selected focus detection region is detected from the focus detection signal related to the selected focus detection region among the focus detection signals read out in S302.

  If no subject is detected in S303, the system control unit 201 obtains the focus state (defocus amount and direction) for all selectable focus detection areas based on the focus detection signal. Then, the focus detection area where the subject is present at the closest distance is selected.

  In step S305, the system control unit 201 adjusts the focus distance of the photographing lens 102 by controlling the position of the focus lens 113 based on the focus state of the focus detection area selected in step S304.

  In step S306, the system control unit 201 determines shooting conditions (aperture value (AV value), shutter speed (TV value), ISO sensitivity (ISO value)) using the exposure control signal read in step S302. Although there is no particular limitation on the method for determining the shooting condition, here, the shooting condition corresponding to the luminance (Bv value) obtained based on the exposure control signal is determined with reference to a pre-stored program diagram. And Note that the shooting condition may be determined using the luminance of the subject area detected by the subject detection process.

  In step S307, the system control unit 201 detects the state of the switch SW2. If the switch SW2 is on, the system control unit 201 advances the process to step S308. If the switch SW2 is off, the process returns to step S301.

  In step S308, the system control unit 201 performs still image shooting processing. The system control unit 201 moves the main mirror 103 and the sub mirror 104 to positions that do not intersect with the light flux from the photographic lens 102, and drives the shutter 110 according to the shutter speed determined in S306. Thereby, the image sensor 111 is exposed by the optical image formed by the photographing lens 102. The image sensor 111 generates an image signal obtained by converting the charge accumulated in each pixel during the exposure period into a voltage. The system control unit 201 reads an image signal from the image sensor and generates image data by applying predetermined image processing such as A / D conversion, noise reduction, white balance adjustment, and color interpolation. The system control unit 201 saves the generated image data as an image data file in the image storage unit 202, or generates a display image signal based on the image data and displays it on the display unit 112. Thereafter, the system control unit 201 returns the process to S301.

(Subject detection process flow)
Next, details of the subject detection processing in S303 of FIG. 3 will be described using the flowchart shown in FIG.
In step S401, the system control unit 201 determines whether the main subject has been determined. If it is not determined that the main subject has been determined, the process proceeds to step S402. If it is determined that the main subject has been determined, the process proceeds to step S403.

  In step S <b> 402, the system control unit 201 (selection unit) selects a detection learning model 206 for subject detection before the main subject determination from among a plurality of learning models stored in the learning model storage unit 205, and performs subject detection processing. The parameter is set in the subject detection unit 204 as a parameter. The detection learning model 206 for detecting a subject before determining a main subject is a learning model for detecting a subject corresponding to a wide range of blur amounts, and has a wide range of blur amounts from a small blur amount to a large blur amount for a subject to be detected. It is a learning model obtained by machine learning using images. By using the detection learning model 206, it is possible to realize subject detection that can detect the subject to be detected even if the amount of blur is large.

  In step S <b> 403, the system control unit 201 (selection unit) selects a detection learning model 207 for subject detection after main subject determination from among a plurality of learning models stored in the learning model storage unit 205, and performs subject detection processing. The parameter is set in the subject detection unit 204 as a parameter. A detection learning model 207 for detecting a subject after the main subject is determined is a learning model for detecting a subject with a small amount of blur, and for a subject to be detected, by machine learning using an image with a small amount of blur as an emphasis. It is the obtained learning model. By using the detection learning model 207, subject detection capable of accurately detecting a subject to be detected with a small amount of blur can be realized.

  In steps S <b> 402 and S <b> 403, the system control unit 201 supplies image data generated by performing A / D conversion or noise reduction processing on the exposure control signal read in step S <b> 302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S404.

  In step S404, the subject detection unit 204 applies subject detection processing to the image data based on the exposure control signal using the detection learning model set in the step S402 or S403 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. The information indicating the detection result may include whether or not a subject has been detected (number of detections) and information (for example, position and size) regarding the detected subject region. The subject detection unit 204 uses CNN for subject detection. Details of CNN will be described later.

  In step S405, if one or more subjects are detected as a result of subject detection, the system control unit 201 determines a main subject from the detected subjects. The method for determining the main subject is as described above.

  Here, it has been described that, after the main subject is determined, subject detection is performed using the detection learning model 207 obtained by machine learning that places importance on an image of a subject with a small amount of blur. However, if the main subject is not detected for a predetermined period even after the main subject is determined, the detection learning model 206, which is a learning model for detecting the subject corresponding to a wide range of blurring, is used. You may comprise. This is because when the main subject is not detected (lost) for a predetermined period of time, the amount of blur of the main subject may be large.

  Further, even after the main subject is determined, the detection learning model 206 may be used for a period in which it is determined that the main subject is not in focus. This is because there is a possibility that the amount of blur of the main subject is large before focusing on the main subject. Alternatively, the detection learning model 206 may be used when the amount of blur of the main subject is greater than or equal to the threshold value. Alternatively, the subject detection may be performed using the detection learning model 206 even before the switch SW1 is turned on.

  When the detection learning model 206 is used, subject detection is performed with priority given to suppressing detection omission over erroneous detection. Therefore, in the case where the detection learning model 206 is used for subject detection, the detection reliability threshold necessary for determining the main subject is made stricter in S405 than in the case where the detection learning model 207 is used for subject detection. Can be set. In other words, the subject detection accuracy (detection reliability) required for the main subject region is greater than the subject region detected using the detection learning model 207 for the subject region detected using the detection learning model 207. Can be set high. Accordingly, it is possible to prevent the erroneously detected subject from being determined as the main subject and optimizing the focus adjustment and the exposure control in an inappropriate area.

(Details of subject detection unit)
Next, the subject detection unit 204 will be described. In the present embodiment, the subject detection unit 204 is composed of Neocognitron, which is a type of CNN (convolutional neural network). A basic configuration of the subject detection unit 204 will be described with reference to FIGS. 5 and 6. FIG. 5 shows a basic configuration of the CNN that detects a subject from the input two-dimensional image data. The processing flow takes the left end as input and proceeds in the right direction. The CNN includes two layers called a feature detection layer (S layer) and a feature integration layer (C layer) as one set, and is configured hierarchically. The S layer corresponds to the convolution layer described in the prior art, and the C layer corresponds to the pooling layer or the subsampling layer.

In the CNN, first, the next feature is detected based on the feature detected in the previous layer in the S layer. Further, it has a configuration in which the features detected in the S layer are integrated in the C layer and transmitted to the next layer as a detection result in that layer.
The S layer is composed of a feature detection cell surface, and detects different features for each feature detection cell surface. Further, the C layer is composed of a feature integrated cell surface, and the detection result on the feature detection cell surface of the previous layer is pooled or subsampled. Hereinafter, the feature detection cell surface and the feature integrated cell surface are collectively referred to as a feature surface unless it is particularly necessary to distinguish between them. In the present embodiment, the output layer (the nth layer), which is the last layer, is configured by only the S layer without using the C layer.

  Details of the feature detection process on the feature detection cell plane and the feature integration process on the feature integration cell plane will be described with reference to FIG. One feature detection cell surface is composed of a plurality of feature detection neurons, and each feature detection neuron is connected to the C layer of the previous layer with a predetermined structure. One feature-integrated cell surface is composed of a plurality of feature-integrated neurons, and each feature-integrated neuron is connected to the S layer of the same hierarchy with a predetermined structure.

と表記する。また、Ｌ階層目のＣ層のＭ番目の細胞面内において、位置(ξ, ζ)の特徴統合ニューロンの出力値を

と表記する。その時、それぞれのニューロンの結合係数を

とすると、各出力値は以下のように表すことができる。 The output value of the feature detection neuron at position (ξ, ζ) in the Mth cell plane of the Sth layer of the Lth layer shown in FIG.

Is written. Also, the output value of the feature integration neuron at the position (ξ, ζ) in the Mth cell plane of the Cth layer of the Lth layer

Is written. At that time, the coupling coefficient of each neuron

Then, each output value can be expressed as follows.

[数式２]

ここで、数式１におけるｆは活性化関数であり、例えばロジスティック関数や双曲正接関数などのシグモイド関数である。また、

は、Ｌ階層目のＳ層のＭ番目の細胞面における、位置(ξ, ζ)の特徴検出ニューロンの内部状態を表す。数式２は活性化関数を用いておらず、単純な線形和で表されている。 [Formula 1]

[Formula 2]

Here, f in Formula 1 is an activation function, for example, a sigmoid function such as a logistic function or a hyperbolic tangent function. Also,

Represents the internal state of the feature detection neuron at position (ξ, ζ) on the Mth cell surface of the Sth layer of the Lth layer. Equation 2 does not use an activation function and is represented by a simple linear sum.

と出力値

とは等しい。また、数式１の

を特徴検出ニューロンの結合先出力値と呼び、数式２の

を特徴統合ニューロンの結合先出力値と呼ぶ。 When the activation function is not used as in Equation 2, the internal state of the neuron

And output value

Is equal to In addition, Formula 1

Is called the connection destination output value of the feature detection neuron.

Is called the connection destination output value of the feature integration neuron.

が大きい場合、入力画像の画素位置(ξ, ζ)に、Ｌ階層目のＳ層のＭ番目の細胞面が検出する特徴が存在する可能性が高いことを意味する。またｎは数式１において、Ｌ−１階層目のＣ層のｎ番目の細胞面を意味しており、統合先特徴番号と呼ぶ。基本的にＬ−１階層目のＣ層に存在する全ての細胞面について積和演算を行う。（ｕ, ｖ）は、結合係数の相対位置座標であり、検出する特徴のサイズに応じて有限の範囲（ｕ, ｖ）において積和演算を行う。このような有限な（ｕ, ｖ）の範囲を受容野と呼ぶ。また受容野の大きさを、以下では受容野サイズと呼び、結合している範囲の横画素数×縦画素数で表す。 Here, ξ, ζ, u, v, and n in Equations 1 and 2 will be described. The position (ξ, ζ) corresponds to the position coordinate in the input image.

Is large, it means that there is a high possibility that the feature detected by the Mth cell surface of the Sth layer of the Lth layer exists at the pixel position (ξ, ζ) of the input image. Moreover, n means the nth cell surface of the C layer of the (L-1) th layer in Formula 1, and is called the integration destination feature number. Basically, the product-sum operation is performed on all the cell planes existing in the C layer of the (L-1) th layer. (U, v) is a relative position coordinate of the coupling coefficient, and a product-sum operation is performed in a finite range (u, v) according to the size of the feature to be detected. Such a finite range (u, v) is called a receptive field. The size of the receptive field is hereinafter referred to as a receptive field size, and is represented by the number of horizontal pixels × the number of vertical pixels in the combined range.

は、入力画像

である。ちなみにニューロンや画素の分布は離散的であり、結合先特徴番号も離散的なので、ξ，ζ，ｕ，ｖ，ｎは離散的な値をとる。ここでは、ξ，ζは非負整数、ｎは自然数、ｕ，ｖは整数とし、何れも有限な範囲を有する。 In Equation 1, L = 1, that is, in the S layer of the first hierarchy,

The input image

It is. Incidentally, since the distribution of neurons and pixels is discrete and the connection feature number is also discrete, ξ, ζ, u, v, and n take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, u and v are integers, and both have a finite range.

は、所定の特徴を検出するための結合係数であり、結合係数を適切な値に調整することによって、所定の特徴を検出可能になる。この結合係数の調整が学習であり、ＣＮＮの構築においては、さまざまなテストパターンを用いて、

が適切な出力値になるように、結合係数を繰り返し徐々に修正していくことで結合係数を調整する。 In Equation 1

Is a coupling coefficient for detecting a predetermined feature, and the predetermined feature can be detected by adjusting the coupling coefficient to an appropriate value. Adjustment of this coupling coefficient is learning, and in the construction of CNN, using various test patterns,

The coupling coefficient is adjusted by repeatedly and gradually correcting the coupling coefficient so that becomes an appropriate output value.

は、２次元のガウシアン関数を用いており、以下の数式３のように表すことができる。
[数式３]

ここでも、（ｕ，ｖ）は有限の範囲を有し、特徴検出ニューロンの場合と同様、範囲を受容野、範囲の大きさを受容野サイズと呼ぶ。ここではＬ階層目のＳ層のＭ番目の特徴のサイズに従って、受容野サイズの値を適宜設定することができる。数式３中のσは特徴サイズ因子であり、受容野サイズに応じて適宜定めることができる定数であってよい。例えば、受容野の一番外側の値がほぼ０とみなせるような値になるように特徴サイズ因子σを設定することができる。このように、本実施形態の被写体検出部２０４は、上述した演算を各階層で行い、最終階層（ｎ階層目）のＳ層において被写体検出を行うＣＮＮによって構成される。 Next, in Equation 2

Uses a two-dimensional Gaussian function and can be expressed as Equation 3 below.
[Formula 3]

Again, (u, v) has a finite range, and the range is called the receptive field and the size of the range is called the receptive field size, as in the case of the feature detection neuron. Here, the value of the receptive field size can be appropriately set according to the size of the Mth feature of the Sth layer of the Lth layer. Σ in Equation 3 is a feature size factor, and may be a constant that can be appropriately determined according to the receptive field size. For example, the feature size factor σ can be set so that the outermost value of the receptive field becomes a value that can be regarded as almost zero. As described above, the subject detection unit 204 of the present embodiment is configured by a CNN that performs the above-described calculation in each layer and performs subject detection in the S layer of the final layer (nth layer).

の具体的な調整（学習）方法について説明する。被写体検出部２０４に直接的に学習を行わせても良いし、被写体検出部２０４と等価なＣＮＮの機能を有するクラウドや外部のエッジコンピュータ上において学習を行わせるようにしてもよい。クラウドやエッジコンピュータ上で学習を行った場合には、ＤＳＬＲ１００が生成された学習モデルを無線通信、有線通信、あるいは、着脱可能な記憶メディアを介して取得し、学習モデル記憶部２０５に記憶させるようにすればよい。学習は、ＣＮＮに特定の入力画像（テストパターン）を与えて得られるニューロンの出力値と、教師信号（そのニューロンが出力すべき出力値）との関係に基づいて、結合係数

を修正することである。本実施形態の学習では、最終階層（ｎ階層目）の特徴検出層Ｓについては最小二乗法を用いて結合係数を修正する。また、他の階層（１〜ｎ−１階層目）の特徴検出層Ｓについては、誤差逆伝搬法を用いて結合係数を修正する。最小二乗法や誤差逆伝搬法を用いた結合係数の修正手法は例えば非特許文献１に記載されるような公知技術を用いることができるため、詳細についての説明は省略する。 (Subject detection learning method)
Coupling factor

A specific adjustment (learning) method will be described. The subject detection unit 204 may perform learning directly, or may perform learning on a cloud or an external edge computer having a CNN function equivalent to the subject detection unit 204. When learning is performed on a cloud or an edge computer, the learning model generated by the DSLR 100 is acquired via wireless communication, wired communication, or a removable storage medium, and stored in the learning model storage unit 205. You can do it. Learning is based on the relationship between the output value of a neuron obtained by giving a specific input image (test pattern) to the CNN and the teacher signal (the output value that the neuron should output).

Is to fix. In the learning of the present embodiment, for the feature detection layer S of the last layer (nth layer), the coupling coefficient is corrected using the least square method. For the feature detection layer S of the other layers (1st to (n-1) th layers), the coupling coefficient is corrected using the error back propagation method. Since a known technique as described in Non-Patent Document 1, for example, can be used as the coupling coefficient correction method using the least square method or the error back propagation method, a detailed description thereof will be omitted.

  A large number of patterns to be detected and patterns that should not be detected are prepared as test patterns for learning. Each test pattern has image data and a corresponding teacher signal. The image data corresponding to the pattern to be detected is a teacher signal such that the output of the neuron corresponding to the area where the pattern to be detected exists is 1 on the feature detection cell surface in the final hierarchy. On the other hand, for image data corresponding to a pattern that should not be detected, a teacher signal is given so that the output of the neuron corresponding to the region where the pattern that should not be detected exists is -1.

  In this embodiment, focusing on the fact that the characteristics of desirable subject detection differ depending on the timing of subject detection, the parameters used for subject detection (learning models generated by machine learning) are set according to the timing of subject detection. It was set as the structure switched. Therefore, a more appropriate subject detection result can be obtained as compared with a configuration using one learning model. Specifically, for a subject to be detected (for example, a human face), a subject before the main subject is determined by machine learning using a test pattern using image data of various blur amounts (from a small blur amount to a large blur amount). A detection learning model 206 for detection is prepared. In addition, a detection learning model 207 for subject detection after main subject determination is prepared for a subject to be detected by machine learning using a test pattern that preferentially uses image data with a small amount of blur.

  Then, the detection learning model 206 is used before the main subject is determined, and the detection learning model 207 is used after the determination, so that a suitable subject detection before the main subject is determined and a suitable subject detection after the main subject is determined. realizable. Therefore, it is possible to appropriately execute processing using a specific subject area among the detected subjects.

<< Second Embodiment >>
Next, a second embodiment of the present invention will be described. FIG. 7 is a block diagram illustrating a functional configuration example of the DSLR 100 according to the present embodiment. The same reference numerals as those in FIG. 2 are assigned to the same configurations as those in the first embodiment. However, the photometric sensor 108 differs from the first embodiment in that when generating an image signal, a gain is applied to the image signal so that the luminance value of the image signal is included in a predetermined range. The magnitude of the gain applied to the image signal by the photometric sensor 108 is notified from the photometric sensor 108 to the system control unit 201. The learning model storage unit 205 is different from the first embodiment in the type of the detected learning model stored.

  The position / orientation change acquisition unit 701 includes a position / orientation sensor such as a gyro, an acceleration sensor, or an electronic compass, for example, and outputs a signal representing the position / orientation change of the DSLR 100. The system control unit 201 stores the signal output from the position / orientation change acquisition unit 701 in the RAM 2012.

  The motion vector detection unit 702 detects a motion vector using image data for two frames. Information on the detected motion vector is output to the system control unit 201. The system control unit 201 stores motion vector information in the RAM 2012. Details of the vector detection process will be described later. Since other parts are the same as those in the first embodiment, description thereof is omitted.

を始点、それらに対応するフレーム_t上の点

を終点とする、ｉ個のベクトルｖ_iを以下のように算出する。

ただし、ベクトルの第一成分を画像の横方向、第二成分を画像の縦方向とする。 (Details of motion vector calculation processing)
Details of the motion vector calculation processing in the motion vector detection unit 702 will be described. FIG. 8 is a diagram schematically showing an image for two frames supplied to the motion vector detection unit 702. Here, two frames of the image is an image captured at different timings from the same viewpoint, a plurality of positions in the frame _t-1 to frame _t, 1 previous frame and the frame _t-1 to the current frame Is matched with the frame _t . And a point on frame _t-1

Starting point, corresponding point on frame _t

I vectors v _i are calculated as follows.

However, the first component of the vector is the horizontal direction of the image, and the second component is the vertical direction of the image.

(Dictionary switching for subject detection)
When attempting to detect a subject area in the current frame based on a motion vector between frames, the probability of false detection increases as the motion vector increases. This is because a subject with a large motion vector has a large image blur, and as in the case where the amount of blur is large, variations in the image similar to the subject increase and characteristics of the subject are lost.

  The subject detection unit 204 applies subject detection processing using processing parameters based on a learning model generated through machine learning in advance to an image. Specifically, the subject detection unit 204 includes a detection learning model 703 for small main subject motion vector and a detection learning model 704 for large main subject motion vector among the learning models stored in the learning model storage unit 205. refer. The detection learning model 703 for small main subject motion vectors is generated by machine learning that places importance on subject images with small subject blur amounts. The detection learning model 704 for large main subject motion vectors is generated by machine learning that places importance on subject images with a large subject blur amount.

  Then, the system control unit 201 selects either the detection learning model 703 for main subject motion vector small or the detection learning model 704 for large main subject motion vector depending on the magnitude of the motion vector of the main subject. Switch between using with. The system control unit 201 switches the characteristics of the subject detection process in the main subject detection unit 204 by switching the learning model used for the subject detection process according to the magnitude of the motion vector of the main subject. More specifically, the system control unit 201 switches the detection learning model 703 to use the detection learning model 704 if the main subject's motion vector is not determined to be large, and uses the detection learning model 704 if the main subject's motion vector is determined to be large. . The threshold for determining the magnitude of the motion vector can be experimentally determined in advance, for example.

(Subject detection process flow)
Next, details of subject detection processing performed by the DSLR 100 according to the present embodiment in a shooting operation will be described with reference to the flowchart shown in FIG. Since the shooting operation of the DSLR 100 of this embodiment is the same as that of the first embodiment, the subject detection process described below is performed in S303 of FIG. Here, the main subject has already been determined, and the motion vector detection unit 702 uses, for example, a frame image of a moving image shot in the shooting standby state, and a motion vector starting from a point in the main subject region in the previous frame. It is assumed that one or more are detected.

  In step S <b> 901, the system control unit 201 refers to the RAM 2012 and calculates the size of the motion vector of the main subject. Then, the system control unit 201 compares the calculated magnitude with a predetermined threshold, and determines that the motion vector is large if it is greater than the threshold. If it is determined that the motion vector of the main subject is large, the system control unit 201 proceeds to step S903, and if not, proceeds to step S902.

  In step S <b> 902, the system control unit 201 selects a detection learning model 703 for small motion vectors from among a plurality of learning models stored in the learning model storage unit 205, and sets the selected learning model 703 as a parameter for subject detection processing in the subject detection unit 204. To do. By using the detection learning model 703, it is possible to realize subject detection that can accurately detect a subject to be detected with small subject blur.

  In step S <b> 903, the system control unit 201 selects a detection learning model 704 for large motion vectors from among a plurality of learning models stored in the learning model storage unit 205, and sets the selected detection learning model 704 as a subject detection processing parameter in the subject detection unit 204. . By using the detection learning model 704, it is possible to realize subject detection that can detect a subject to be detected even if subject blurring is large.

  In steps S902 and S903, the system control unit 201 supplies the subject detection unit 204 with image data generated by performing A / D conversion, noise reduction processing, and the like on the exposure control signal read in step S302. Then, the system control unit 201 advances the process to S904.

  In step S904, the subject detection unit 204 applies subject detection processing to the image data based on the exposure control signal using the detection learning model set in step S902 or S903 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. Information representing the detection result and specific processing of subject detection using CNN may be the same as in the first embodiment.

  In step S905, if one or more subjects are detected as a result of subject detection, the system control unit 201 determines a main subject from the detected subjects. The method for determining the main subject may be the same as in the first embodiment.

  In S906, the motion vector detection unit 702 calculates the motion vector of the main subject determined in S905 by the method described above.

  In the present embodiment, focusing on the fact that the desired subject detection characteristics differ depending on the magnitude of the motion vector of the main subject, the parameters used for subject detection (the learning model generated by machine learning) are set to the magnitude of the motion vector of the main subject. It was set as the structure switched according to it. Therefore, a more appropriate subject detection result can be obtained as compared with a configuration using one learning model. Specifically, a detection learning model 703 for small motion vectors is prepared by subjecting a subject to be detected (for example, a human face) to machine learning using a test pattern that preferentially uses image data with a small subject blur amount. In addition, a detection learning model 704 for a large motion vector is prepared for a subject to be detected by machine learning using a test pattern that preferentially uses image data with a large subject blur amount.

  Then, when it is determined that the motion vector of the main subject is large, the detection learning model 704 is switched, and when it is not determined that the main subject is large, the detection learning model is switched so as to execute the subject detection processing. Thereby, subject detection suitable for the size of the motion vector of the main subject (that is, the amount of blur of the subject image) can be realized. Therefore, it is possible to appropriately execute processing using a specific subject area among the detected subjects.

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described. This embodiment can be carried out in the same manner as the second embodiment except for a learning model switching method used for subject detection. For this reason, the learning model switching method will be mainly described below.

(Dictionary switching for subject detection)
In the present embodiment, the learning model used for the subject detection process is switched according to the magnitude of the position / orientation change of the imaging device (DSLR 100). When the position and orientation change of the image pickup apparatus is large, camera shake becomes large, and erroneous detection of the subject area is likely to occur. This is because, when the change in the position and orientation of the imaging apparatus is large, variations in images similar to the subject increase and the characteristics of the subject are lost, as in the case where the amount of blur is large.

  The subject detection unit 204 of the present embodiment applies subject detection using processing parameters based on a learning model generated in advance through machine learning to an image. In this embodiment, the detection learning model 705 for small change in position and orientation and the detection learning model 706 for large change in position and orientation are referred to from the learning model storage unit 205. The detection learning model 705 for small position and orientation change is generated by machine learning that places importance on the subject image when the amount of camera shake is small. Further, the detection learning model 706 for large change in position and orientation is generated by machine learning that places importance on the subject image when the amount of camera shake is large.

  The system control unit 201 switches the subject detection characteristics in the subject detection unit 204 by switching the learning model used by the subject detection unit 204 according to the magnitude of the position and orientation change. More specifically, when it is not determined that the position and orientation change is large, the detection learning model 705 for small position and orientation change is detected. Switch to use the learning model 706. The threshold for determining the magnitude of the position and orientation change can be experimentally determined in advance, for example.

(Subject detection process flow)
Next, the details of subject detection processing performed by the DSLR 100 of the present embodiment in the shooting operation will be described using the flowchart shown in FIG. Since the shooting operation of the DSLR 100 of this embodiment is the same as that of the first embodiment, the subject detection process described below is performed in S303 of FIG.

  In step S <b> 1001, the system control unit 201 refers to the RAM 2012 and compares the position / orientation change obtained from the position / orientation change acquisition unit 701 with a predetermined threshold. If it is determined that the position and orientation change of the DSLR 100 is large, the system control unit 201 proceeds to S1003. If not determined, the system control unit 201 proceeds to S1002.

In step S <b> 1002, the system control unit 201 selects a detection learning model 705 for small change in position and orientation from among a plurality of learning models stored in the learning model storage unit 205, and supplies the subject detection unit 204 with a parameter for subject detection processing. Set. By using the detection learning model 705, it is possible to realize subject detection that can accurately detect a subject to be detected with small camera shake.
In step S <b> 1003, the system control unit 201 selects a detection learning model 706 for large change in position and orientation from among a plurality of learning models stored in the learning model storage unit 205, and sets the selected detection learning model 706 as a subject detection processing parameter in the subject detection unit 204. To do. By using the detection learning model 706, it is possible to realize subject detection that can detect a subject to be detected even if camera shake is large.

  In steps S1002 and S1003, the system control unit 201 supplies image data generated by performing A / D conversion, noise reduction processing, and the like on the exposure control signal read in step S302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S1004.

  In step S1004, the subject detection unit 204 applies subject detection processing to the image data based on the exposure control signal using the detection learning model set in the step S1002 or S1003 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. Information representing the detection result and specific processing of subject detection using CNN may be the same as in the first embodiment.

  In the present embodiment, focusing on the fact that desirable object detection characteristics differ according to the magnitude of the position and orientation change of the imaging apparatus, the parameters (learning model generated by machine learning) used for subject detection are set to the magnitude of the position and orientation change. It was set as the structure switched according to it. Therefore, a more appropriate subject detection result can be obtained as compared with a configuration using one learning model. Specifically, for a subject to be detected (for example, a human face), a detection learning model 705 for small change in position and orientation is obtained by machine learning using a test pattern that preferentially uses image data when the amount of camera shake is small. prepare. Also, a detection learning model 706 for large change in position and orientation is prepared by machine learning based on a test pattern that preferentially uses image data when the amount of camera shake is large for a subject to be detected.

  Then, the detection learning model 706 is switched using the detection learning model 706 when it is determined that the position and orientation change is large, and the detection learning model 705 is switched when it is not determined that the position and orientation change is large. Thereby, subject detection suitable for the magnitude of the position and orientation change (that is, the magnitude of the blur amount of the subject image) can be realized. Therefore, it is possible to appropriately execute processing using a specific subject area among the detected subjects.

<< 4th Embodiment >>
Next, a fourth embodiment of the present invention will be described. This embodiment can be carried out in the same manner as the second embodiment except for a learning model switching method used for subject detection. For this reason, the learning model switching method will be mainly described below.

(Dictionary switching for subject detection)
In this embodiment, the learning model used for the subject detection process is switched according to the amount of noise in the image data used for subject detection. More specifically, the learning model used for the subject detection process is set according to the magnitude of the gain (signal amplification factor) applied to the image data (image signal generated by the photometric sensor 108) supplied to the subject detection unit 204. To switch.

  When subject detection processing is applied to image data to which a large gain is applied, erroneous detection of the subject region is likely to occur. This is because by applying gain to the image data, noise is also amplified, the S / N ratio of the image is lowered, the variation of the image similar to the subject is increased, and the characteristics of the subject are lost.

  The subject detection unit 204 of the present embodiment applies subject detection using processing parameters based on a learning model generated in advance through machine learning to an image. In the present embodiment, the detection learning model 707 for small gain and the detection learning model 708 for large gain are referred to from the learning model storage unit 205. The small gain detection learning model 707 is generated by machine learning that places importance on subject images when noise is small. Further, the detection learning model 708 for increasing gain is generated by machine learning that places importance on a subject image when noise is large.

  The system control unit 201 switches the learning model used by the subject detection unit 204 according to the magnitude of the gain applied to the image data supplied to the subject detection unit 204, so that the subject in the main subject detection unit 204 Switch detection characteristics. More specifically, when it is determined that the gain applied to the image data is large, the detection learning model 708 for increasing the gain is set to a low gain when the gain applied to the image is not determined to be large. The detection learning model 707 is switched to use. The threshold for determining the magnitude of the gain can be experimentally determined in advance, for example. Note that when subject detection is performed on image data obtained by the image sensor 111, the system control unit 201 can determine the magnitude of the gain corresponding to the imaging sensitivity, for example.

(Subject detection process flow)
Next, details of subject detection processing performed by the DSLR 100 according to the present embodiment in a shooting operation will be described with reference to the flowchart shown in FIG. Since the shooting operation of the DSLR 100 of this embodiment is the same as that of the first embodiment, the subject detection process described below is performed in S303 of FIG.

  In step S1101, the system control unit 201 compares the magnitude of the gain applied to the image signal by the photometric sensor 108 with a predetermined threshold, and determines that the gain is large if it is greater than the threshold. If it is determined that the gain is large, the system control unit 201 proceeds to S1103. If not determined, the system control unit 201 proceeds to S1102.

  In step S1102, the system control unit 201 selects a low-gain detection learning model 707 from among a plurality of learning models stored in the learning model storage unit 205, and sets the selected learning model 707 as a parameter for subject detection processing in the subject detection unit 204. . By using the detection learning model 707, it is possible to realize subject detection that can accurately detect a subject to be detected when noise is small.

  In step S <b> 1103, the system control unit 201 selects a large-gain detection learning model 708 from among a plurality of learning models stored in the learning model storage unit 205, and sets the selected learning model 708 as a subject detection processing parameter in the subject detection unit 204. By using the detection learning model 708, it is possible to realize subject detection that can detect the subject to be detected even when the noise is large.

  In steps S1102 and S1103, the system control unit 201 supplies image data generated by performing A / D conversion, noise reduction processing, and the like on the exposure control signal read in step S302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S1104.

  In step S1104, the subject detection unit 204 applies subject detection processing to the image data based on the exposure control signal using the detection learning model set in step S1102 or S1103 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. Information representing the detection result and specific processing of subject detection using CNN may be the same as in the first embodiment.

  In this embodiment, focusing on the fact that desirable object detection characteristics differ depending on the amount of noise in the image data, the parameters used for object detection (the learning model generated by machine learning) are set to the gain applied to the image data. It was set as the structure switched according to a magnitude | size. Therefore, a more appropriate subject detection result can be obtained as compared with a configuration using one learning model. Specifically, for a subject to be detected (for example, a human face), a small gain detection learning model 707 is prepared by machine learning using a test pattern that preferentially uses image data with little noise. Also, a detection learning model 708 for increasing gain is prepared for a subject to be detected by machine learning based on a test pattern using image data with much noise.

  Then, when it is determined that the gain applied to the image data for subject detection is large, the detection learning model 708 is used, and when it is not determined that the gain is large, the subject detection processing is executed using the detection learning model 707. Switch detection learning model. Thereby, it is possible to realize subject detection suitable for a case where noise is low and subject detection suitable for a case where noise is high. Therefore, it is possible to appropriately execute processing using a specific subject area among the detected subjects.

(Other embodiments)
In the above-described embodiment, an example in which the present invention is applied to an imaging apparatus that optimizes exposure and focus adjustment in the region of the main subject has been described. However, the purpose of use of the main subject area is not limited to shooting. For example, it can be used for any other purpose such as when image processing is applied to the main subject area (or an area other than the main subject area). Therefore, in the present invention, a configuration relating to photographing is not essential. Moreover, although the above-mentioned embodiment demonstrated the structure which switches and uses two types of learning models, you may switch and use three or more types of learning models.

  As described above, examples of situations in which the amount of blur and sharpness of the subject to be subject to the subject detection process may vary include the degree of focus, the magnitude of the subject's movement, the magnitude of the position / orientation change of the imaging device, The embodiment focusing on the number has been described. However, the detection learning model may be switched according to other conditions.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a computer-readable storage medium, and one or more processors of the computer of the system or apparatus execute the program. It can also be realized by executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely specific examples for the purpose of assisting understanding of the present invention, and are not intended to limit the present invention to the above-described embodiments in any way. All embodiments that fall within the scope of the claims are encompassed by the present invention.

DESCRIPTION OF SYMBOLS 100 ... Digital single-lens reflex camera, 101 ... Main body, 102 ... Lens, 108 ... Photometry sensor, 111 ... Image sensor, 204 ... Subject detection part, 206 ... Detection learning model for subject detection processing before main subject determination, 207 ... Main Detection learning model for subject detection processing after subject determination

Claims (22)

Subject detection means for applying subject detection processing to image data using parameters generated based on machine learning;
Storage means for storing a plurality of parameters used for the subject detection processing;
A selection means for selecting a parameter to be used in the subject detection means from the parameters stored in the storage means;
Have
The image processing apparatus according to claim 1, wherein the selection unit selects different parameters according to a blur amount or a sharpness of a subject.   The image processing apparatus according to claim 1, wherein the selection unit selects different parameters depending on whether the main subject is determined or not.   The selection means selects a first parameter when the main subject is determined, selects a second parameter when the main subject is not determined, and the second parameter is: The image processing apparatus according to claim 2, wherein the image processing apparatus is a parameter suitable for detecting a subject having a larger amount of blur than the first parameter. The selection means selects a first parameter when the main subject is determined, and selects a second parameter when the main subject is not determined,
The first parameter is a parameter for performing subject detection processing that prioritizes suppression of subject false detection over suppression of subject detection omission,
The image processing apparatus according to claim 2, wherein the second parameter is a parameter for performing subject detection processing that prioritizes suppression of subject detection omission over suppression of subject erroneous detection.   The selection unit is configured to select the second parameter when the main subject is not detected continuously for a predetermined period of time even when the main subject is determined. 5. The image processing apparatus according to 4.   The selection means selects the second parameter when the blur amount of the main subject is equal to or greater than a threshold value even when the main subject is determined. The image processing apparatus according to claim 5.   The accuracy of subject detection required for the main subject region is set higher for the subject region detected using the second parameter than the subject region detected using the first parameter. The image processing apparatus according to claim 3, wherein the image processing apparatus is characterized.   The image processing apparatus according to claim 1, wherein the selection unit selects different parameters depending on whether the movement of the subject is determined to be large or not.   The selection means selects the first parameter when it is determined that the movement of the subject is large, selects the second parameter when it is not determined that the movement of the subject is large, and the first parameter The image processing apparatus according to claim 1, wherein the image processing apparatus is a parameter suitable for detecting a subject having a larger movement than the second parameter.   The image processing apparatus according to claim 1, wherein the selection unit selects different parameters depending on whether the position and orientation change of the image processing apparatus is determined to be large or not.   The selection unit selects the first parameter when it is determined that the position and orientation change of the image processing apparatus is large, and the second parameter when it is not determined that the position and orientation change of the image processing apparatus is large. The first parameter is a parameter suitable for detection of a subject when a change in position and orientation of the image processing apparatus is larger than the second parameter. An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the selection unit selects different parameters depending on whether the noise amount of the image data is determined to be large or not.   The selection means selects the first parameter when it is determined that the amount of noise of the image data is large, and selects the second parameter when it is not determined that the amount of noise of the image data is large. The image processing apparatus according to claim 1, wherein the first parameter is a parameter suitable for detecting a subject with respect to image data having a larger amount of noise than the second parameter.   The image processing apparatus according to claim 1, wherein the parameter is a coupling coefficient used in a convolutional neural network (CNN).   The image processing apparatus according to claim 1, further comprising an acquisition unit configured to acquire the parameter from outside.   The acquisition unit acquires the parameter from the outside through wireless communication, wired communication, or a removable storage medium, and the storage unit stores the parameter acquired by the acquisition unit. Item 15. The image processing apparatus according to Item 15. An image sensor;
An image processing apparatus comprising: the image processing apparatus according to claim 1,
Subject detection means of the image processing device applies subject detection processing to an image obtained by the image sensor;
An imaging apparatus characterized by that. Focus adjusting means for adjusting the focus so that the determined area of the main subject is in focus;
Determining means for determining an exposure condition such that the area of the main subject has an appropriate exposure;
The image pickup apparatus according to claim 17, further comprising at least one of the following.   The imaging apparatus according to claim 17 or 18, wherein the imaging element is an imaging element for acquiring an image for exposure control. An image processing method executed by an image processing apparatus,
A subject detection step of applying subject detection processing to an image using parameters generated based on machine learning;
A selection step of selecting parameters used in the subject detection step from a storage unit that stores a plurality of parameters used in the subject detection processing;
Have
In the selection step, an image processing method is characterized in that different parameters are selected according to the amount of blur or definition of a subject.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 16.   The program for functioning the computer which an imaging device has as an image processing apparatus which the imaging device of any one of Claims 17-19 has.

Images