JP2019186918A - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Abstract

To provide an image processing apparatus capable of improving subject detection accuracy for image signals.SOLUTION: The image processing apparatus uses a learning model generated on the basis of machine learning to apply subject detection processing to an image. By using a learning model selected according to image characteristics to which subject detection processing is applied from learning models stored in advance, the subject detection processing is applied.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus, and more particularly to a subject detection technique.

  A subject detection technique for automatically detecting a specific subject pattern from an image is very useful. Patent Document 1 discloses an imaging apparatus that detects a region corresponding to a specific subject pattern such as a human face from a photographed image and optimizes the focus and exposure of the detected region.

  It is also known to learn and recognize a subject in an image using a technique called deep learning (Non-Patent Document 1). The convolutional neural network (CNN) is a typical technique for deep learning. In general, CNN is a multi-layer structure in which a convolution layer that spatially integrates local features of an image, a pooling layer or sub-sampling layer that compresses features in the spatial direction, and a fully connected layer, an output layer, etc. Have Since the CNN can acquire a complicated feature expression through stepwise feature conversion with a multi-layer structure, the category recognition of the subject in the image and the subject detection can be performed with high accuracy based on the feature representation.

JP 2005-318554 A

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, “ImageNet classification with deep convolutional neural networks”, NIPS'12 Proceedings of the 25th International Conference on Neural Information Processing Systems-Volume 1, PP.1097-1105

  When machine learning is performed for features for detecting a subject from an image by supervised learning, a learning image signal and a pair of teacher signals are given to the apparatus. As a result of learning, a learning model used for subject detection is generated. An image signal obtained by shooting is affected by the characteristics of the optical system of the imaging apparatus, such as resolution, color tone, and degree of blur. For this reason, if the characteristics of the optical system are different between the time of learning and the time of subject detection using a learning result (learning model), the detection may fail.

  The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide an image processing apparatus, an image processing method, and an imaging apparatus capable of improving subject detection accuracy with respect to an image signal. To do.

  The above-described object is provided by subject detection means for applying subject detection processing to an image using parameters generated based on machine learning, storage means for storing a plurality of parameters used for subject detection processing, and storage means. This is achieved by an image processing apparatus comprising selection means for selecting a parameter to be used in the subject detection means in accordance with the characteristics of the image to which subject detection processing is applied from the stored parameters.

  According to the present invention, it is possible to provide an image processing device, an image processing method, and an imaging device capable of improving subject detection accuracy for an image signal.

1 is a schematic vertical sectional view of a digital single-lens reflex camera as an example of an image processing apparatus according to an embodiment of the present invention. 1 is a block diagram illustrating an example of a functional configuration of a digital single-lens reflex camera according to an embodiment. The flowchart regarding the outline | summary of imaging | photography operation | movement which concerns on embodiment. The flowchart regarding the still image shooting operation | movement which concerns on embodiment. The flowchart regarding the video recording operation | movement which concerns on embodiment. The schematic diagram which shows the structural example of CNN which the to-be-photographed object detection circuit which concerns on embodiment uses. The schematic diagram which shows the structure of a part of CNN of FIG.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, a case where the present invention is implemented by a digital single lens reflex camera (DSLR) will be described. However, the present invention can be implemented with any electronic device capable of handling image data, and a digital single-lens reflex camera is only an example of an image processing apparatus according to the present invention. Electronic devices that can implement the present invention include, but are not limited to, personal computers, smartphones, tablet terminals, game machines, robots, and the like.

● (Configuration of imaging device)
FIG. 1 is a vertical sectional view of a digital single-lens reflex camera (DSLR) 100 according to this embodiment. FIG. 2 is a block diagram showing a functional configuration example of the DSLR 100. Like reference numerals refer to like elements throughout the drawings.

  The DSLR 100 includes a main body 101 and a photographing lens 102 (interchangeable lens) that can be attached to and detached from the main body 101. Mount contact groups 115 are provided on the attachment and detachment portions (mounts) of the main body 101 and the photographing lens 102, respectively. When the photographic lens 102 is attached to the main body 101, the mount contact group 115 comes into contact, and electrical connection between the photographic lens 102 and the main body 101 is established.

  The system control circuit 201 includes one or more programmable processors, a ROM 2011, and a RAM 2012, and controls operations of the main body 101 and the photographing lens 102 by reading a program stored in the ROM 2011 into the RAM 2012 and executing it. The ROM 2011 stores various setting values, GUI data, and the like in addition to programs executed by the system control circuit 201.

  The photographing lens 102 is provided with a focus lens 113 that adjusts the focusing distance, and a diaphragm 114 (and a motor or actuator that drives them) that adjusts the amount of light incident on the main body 101. The driving of the focus lens 113 and the diaphragm 114 is controlled by the camera body 101 through the mount contact group 115.

  The main mirror 103 and the sub mirror 104 constitute a quick return mirror. A part of the main mirror 103 has a reflectance (transmittance) controlled to separate a light beam incident from the photographing lens 102 into a light beam traveling toward the finder optical system (upper side in the drawing) and a light beam traveling toward the sub mirror 104. .

  FIG. 1 shows a state when the optical viewfinder is used (when not photographing), and the main mirror 103 is positioned in the optical path of the light beam incident from the photographing lens 102. In this state, the reflected light of the main mirror 103 enters the finder optical system, and the light beam bent by the pentaprism 107 is emitted from the eyepiece 109. Therefore, the user can see the optical subject image by looking into the eyepiece 109.

  Further, the transmitted light of the main mirror 103 is reflected by the sub mirror 104 and enters the AF sensor 105 (first image sensor). The AF sensor 105 forms a secondary imaging surface of the photographing lens 102 on the line sensor, and generates a pair of image signals (focus detection signals) that can be used for focus detection by the phase difference detection method. The generated focus detection signal is transmitted to the system control circuit 201. The system control circuit 201 obtains the defocus amount of the focus lens 113 using the focus detection signal, and controls the drive direction and drive amount of the focus lens 113 based on the defocus amount.

  The focus plate 106 is disposed on the planned imaging plane of the photographing lens 102 in the finder optical system. A user looking into the eyepiece 109 observes the optical image formed on the focusing plate 106. In addition to the optical image, shooting information such as shutter speed and aperture value can also be provided.

  The photometric sensor 108 generates an image signal (exposure control signal) from the incident light beam and transmits it to the system control circuit 201. The system control circuit 201 performs automatic exposure control using the received exposure control signal or controls subject detection by a subject detection circuit 204 described later. The photometric sensor 108 is an image sensor in which pixels including a photoelectric conversion unit are two-dimensionally arranged.

  When the image sensor 111 is exposed, the main mirror 103 and the sub mirror 104 move out of the optical path of the light beam incident from the photographing lens 102. Further, a focal plane shutter 110 (hereinafter simply referred to as a shutter) is opened.

  In the image sensor 111 (second image sensor), pixels including a photoelectric conversion unit are two-dimensionally arranged, and a subject optical image formed by the photographing lens 102 is photoelectrically converted by each pixel, and an image signal is converted into a system. Transmit to the control circuit 201. The system control circuit 201 generates image data from the received image signal, stores it in the image storage memory 202, and displays it on a monitor 112 such as an LCD. The image data generated by the image sensor 111 is also supplied to the subject detection circuit 204 for subject detection. Note that the system control circuit 201 may perform focus detection by a contrast method using image data.

In this embodiment, each pixel of the image sensor 111 includes two photoelectric conversion units (referred to as a photoelectric conversion unit A and a photoelectric conversion unit B), and image signals can be read independently from the individual photoelectric conversion units. It shall have a configuration. In other words, the image sensor 111 can be exposed once.
An image signal (referred to as an A image) obtained from the photoelectric conversion unit A group;
An image signal (referred to as B image) obtained from the photoelectric conversion unit B group;
An image signal (A + B image) obtained by adding the image signal obtained from the photoelectric conversion unit A and the image signal obtained from the photoelectric conversion unit B for each pixel;
Can be generated.

  Since the A image and the B image are a pair of parallax images, focus detection by the phase difference detection method can be performed based on the A image and the B image. In the present embodiment, an A + B image is acquired at the time of still image shooting, and focus detection is performed using the AF sensor 105. On the other hand, since an image signal cannot be obtained from the AF sensor 105 during moving image shooting, an A + B image and an A image are acquired. The B image is generated by subtracting the A image from the A + B image. A B image may be acquired instead of the A image.

  The operation member 203 is an input device group that is provided in the main body 101 and the photographing lens 102 and can be operated by the user. A release button, a power switch, a direction key, a determination button, a menu button, an operation mode selection dial, and the like are specific examples of the input device included in the operation member 203, but are not limited thereto. The operation of the operation member 203 is detected by the system control circuit 201.

  For example, when a half-press operation of the release button is detected, the system control circuit 201 starts a still image shooting preparation operation. The shooting preparation operation is, for example, an operation related to automatic focus detection (AF) and automatic exposure control (AE). In addition, when the full pressing operation of the release button is detected, the system control circuit 201 performs still image shooting and recording operations. The system control circuit 201 displays an image obtained by shooting on the monitor 112 for a certain period of time.

  In addition, during moving image shooting (in shooting standby state or during moving image recording), the system control circuit 201 displays the moving image obtained by shooting on the monitor 112 in real time, thereby functioning the monitor 112 as an electronic viewfinder (EVF). Let A moving image displayed when the monitor 112 functions as an EVF and its frame image are referred to as a live view image or a through image. Which of the still image and the moving image is to be taken can be selected through the operation member 203, and the system control circuit 201 switches the control method of the camera body 101 and the taking lens 102 between the still image shooting and the moving image shooting.

  The subject detection circuit 204 is configured by a GPU (Graphic Processing Unit). The GPU is originally a processor for image processing, but has a plurality of product-sum operation units and is good at matrix calculation, so it is often used as a processor that performs processing for learning. In general, GPU is also used in the process of performing deep learning. For example, as the subject detection circuit 204, a Jetson TX2 module manufactured by NVIDIA can be used. Note that as the subject detection circuit 204, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like may be used.

  The subject detection circuit 204 applies subject detection processing to the supplied image data using one learning model selected by the system control circuit 201 among a plurality of learning models stored in the learning model memory 205. Details of the subject detection process will be described later. The learning model memory 205 may be a rewritable nonvolatile memory, for example, or may be a part of the ROM 2011. In the present embodiment, the learning model memory 205 stores learning models 206 and 207 prepared for each image sensor (image sensor) that generates an image signal that is a source of image data to be subjected to subject detection processing.

(Learning model switching for object detection)
The DSLR 100 according to the present embodiment applies subject detection to image data based on image signals generated by the photometric sensor 108 and the image sensor 111 through which light enters through different paths. Although details of subject detection will be described later, a learning model generated in advance through machine learning is used.

  Both the photometric sensor 108 and the image sensor 111 are common in that an optical image is photoelectrically converted by a plurality of two-dimensionally arranged pixels to generate an image signal, but the characteristics (image quality) of the generated image signal are different. The difference in image quality occurs because the optical path, sensor structure, signal processing, and the like differ between the photometric sensor 108 and the image sensor 111. Also, the processing when the image data is generated by the system control circuit 201 may be different. In general, the image signal generated by the photometric sensor 108 has lower resolution and color reproducibility than the image signal generated by the image sensor 111. This is largely due to the fact that the image sensor 111 is intended to generate an image signal for viewing purposes, whereas the photometric sensor 108 is intended to generate an image signal for exposure control. However, even if the photometric sensor 108 and the image sensor 111 use the same device, there is a difference in image quality due to an incident optical path, a difference in processing when generating image data, and the like.

  Therefore, when a learning model generated by machine learning based on the image signal generated by the photometric sensor 108 is used for subject detection for the image signal generated by the image sensor 111, the detection accuracy may decrease. The reverse is also true. Therefore, in the present embodiment, a different learning model is prepared for each image sensor (or an image having different characteristics) that generates an image signal. Then, subject detection processing is applied using a learning model corresponding to the sensor that generated the image signal to which subject detection processing is applied.

  Specifically, the subject detection circuit 204 uses a photometric sensor learning model 206 for image data based on the image signal generated by the photometric sensor 108. The subject detection circuit 204 uses the learning model 207 for the image sensor for image data based on the image signal generated by the image sensor 111.

(Shooting operation)
Next, with reference to FIGS. 3 to 5, the photographing operation of the DSLR 100 of the present embodiment will be described.
FIG. 3 is a flowchart relating to the outline of the photographing operation, and the processing of each step is realized by executing a program read from the ROM 2011 to the RAM 2012 by the programmable processor of the system control circuit 201.

  In step S301, the system control circuit 201 determines whether the power source of the main body 101 is ON. If it is not determined to be ON, the process ends. If it is determined to be ON, the process proceeds to step S302. The determination can be based on, for example, a reference to the state of the power switch of the operation member 203 or a flag indicating ON / OFF of the power source.

  In step S302, the system control circuit 201 determines the shooting mode. Here, it is determined whether the shooting mode is the still image shooting mode or the moving image shooting mode, but other shooting modes may be settable. The shooting mode can be changed by a user operation of the operation member 203. If it is determined that the still image shooting mode is set, the system control circuit 201 advances the process to step S303, and if it is determined that the moving image shooting mode is set, the process advances to step S304.

  In step S303, the system control circuit 201 performs still image shooting processing, and returns the processing to step S301. In step S304, the system control circuit 201 performs moving image shooting processing, and returns the processing to step S301. The still image shooting process will be described later with reference to FIG. 4, and the moving image shooting process will be described later with reference to FIG.

(Still image processing)
FIG. 4 is a flowchart regarding the details of the still image shooting process shown in S303 of FIG.
In step S401, the system control circuit 201 detects the state of the switch SW1 that is turned on when the release button is pressed halfway and the switch SW2 that is turned on when the release button is fully pressed. The system control circuit 201 advances the process to S402 if either of the switches SW1 and SW2 is on, and ends the process if both of the switches SW1 and SW2 are off.

  In step S <b> 402, the system control circuit 201 performs exposure processing (charge accumulation) of the photometric sensor 108. The exposure process of the photometric sensor 108 is realized by performing charge accumulation for a predetermined time using a so-called electronic shutter. The system control circuit 201 controls the operation of the photometric sensor 108, performs charge accumulation for a predetermined time, and reads an image signal (exposure control signal) from the photometric sensor 108. The system control circuit 201 also performs exposure processing (charge accumulation) on the AF sensor 105 and reads an image signal (focus detection signal).

  In step S <b> 403, the system control circuit 201 (selection unit) selects a learning model 206 for the photometric sensor from among a plurality of learning models stored in the learning model memory 205, and uses the subject detection circuit 204 as a parameter for subject detection processing. Set to. In addition, the system control circuit 201 supplies image data generated by performing A / D conversion, noise reduction processing, and the like on the exposure control signal read out in step S <b> 402 to the subject detection circuit 204.

  Here, it is assumed that the optical finder is being used at the time of still image shooting. However, for example, it may be determined whether the EVF (monitor 112) is being used or the optical finder is being used. During still image shooting without using the optical viewfinder, the system control circuit 201 selects a learning model 207 for the image sensor from among a plurality of learning models stored in the learning model memory 205, and uses it as a parameter for subject detection processing. Set in the subject detection circuit 204. Whether or not the optical viewfinder is being used can be determined by a known method such as a method using a proximity sensor provided in the vicinity of the eyepiece 109.

  In step S404, the subject detection circuit 204 applies subject detection processing to the image data based on the exposure control signal using the learning model 206 for the photometric sensor set in step S403. Details of the subject detection process will be described later. The subject detection circuit 204 supplies information representing the detection result to the system control circuit 201. The information indicating the detection result may include whether or not a subject has been detected (number of detections) and information (for example, position and size) regarding the detected subject region.

  In step S405, if one or more subjects are detected as a result of subject detection in step S404, the system control circuit 201 selects a focus detection region that is closest to the detected subject position. When a plurality of subjects are detected, the system control circuit 201 determines a representative subject based on, for example, the size and position of the subject region, and selects a focus detection region that is closest to the position of the representative subject. Then, the system control circuit 201 obtains the focus state (defocus amount and direction) for the selected focus detection region based on the focus detection signal.

  If no subject is detected in S404, the system control circuit 201 obtains the focus state (defocus amount and direction) for all selectable focus detection areas based on the focus detection signal. Then, the focus detection area where the subject is present at the closest distance is selected.

  In step S406, the system control circuit 201 adjusts the focus distance of the photographing lens 102 by controlling the position of the focus lens 113 based on the focus state of the focus detection area selected in step S405.

  In step S407, the system control circuit 201 determines shooting conditions (aperture value (AV value), shutter speed (TV value), ISO sensitivity (ISO value)) using the exposure control signal read in step S402. Although there is no particular limitation on the method for determining the shooting condition, here, the shooting condition corresponding to the luminance (Bv value) obtained based on the exposure control signal is determined with reference to a pre-stored program diagram. And Note that the shooting condition may be determined using the luminance of the subject area detected by the subject detection process.

  In step S408, the system control circuit 201 detects the state of the switch SW2. If the switch SW2 is on, the system control circuit 201 advances the process to step S409. If the switch SW2 is off, the system control circuit 201 ends the process.

  In step S409, the system control circuit 201 executes still image shooting processing. The system control circuit 201 moves the main mirror 103 and the sub mirror 104 to positions that do not intersect with the light flux from the photographic lens 102, and drives the shutter 110 according to the shutter speed determined in S407. Thereby, the image sensor 111 is exposed by the optical image formed by the photographing lens 102. The image sensor 111 generates an image signal obtained by converting the charge accumulated in each pixel during the exposure period into a voltage. The system control circuit 201 reads an image signal from the image sensor 111, and generates image data by applying predetermined image processing such as A / D conversion, noise reduction, white balance adjustment, and color interpolation. The system control circuit 201 saves the generated image data as an image data file in the image storage memory 202, or generates a display image signal based on the image data and displays it on the monitor 112.

(Movie shooting process)
Next, details of the moving image shooting processing in S304 of FIG. 3 will be described using the flowchart shown in FIG. The moving image shooting operation is executed at the time of shooting standby or in response to detection of a moving image recording start instruction. Note that moving image shooting during shooting standby aims to generate a through image for display, and therefore differs in resolution (number of pixels) and the like from moving image shooting for recording purposes. However, since the contents of the subject detection process are basically the same, the following description will be given without considering the purpose of capturing a moving image.

  In step S501, the system control circuit 201 executes processing for one frame of a moving image and generates image data. In moving image shooting, in order to continuously perform shooting at a predetermined frame rate, the shutter 110 is fully opened and the main mirror 103 and the sub mirror 104 are moved. The exposure time of the image sensor 111 is adjusted by controlling the charge accumulation time. The system control circuit 201 repeats charge accumulation, image signal readout, and accumulated charge reset every time one frame is shot. The system control circuit 201 generates image data by applying image processing to the image signals (A + B image and A image) read out from the image sensor 111, and stores the A + B image in the image storage memory 202. Further, a display image corresponding to the A + B image is generated and displayed on the monitor 112 as a through image. Further, the system control circuit 201 stores an A image and a B image generated from the A + B image and the A image in, for example, the RAM 2012 in order to perform focus detection.

  In step S502, the system control circuit 201 sets the learning model 207 for the image sensor in the subject detection circuit 204 as a parameter for subject detection processing. Further, the system control circuit 201 supplies the image data stored in the image storage memory 202 to the subject detection circuit 204.

  In step S503, the subject detection circuit 204 applies subject detection processing to the image data supplied from the system control circuit 201 using the learning model 207 for the image sensor set in step S502. Details of the subject detection process will be described later. The subject detection circuit 204 supplies information representing the detection result to the system control circuit 201. The information indicating the detection result may include whether or not a subject has been detected (number of detections) and information (for example, position and size) regarding the detected subject region.

  In step S504, if one or more subjects are detected as a result of the subject detection in step S503, the system control circuit 201 selects a focus detection region closest to the detected subject position. When a plurality of subjects are detected, the system control circuit 201 determines a representative subject based on, for example, the size and position of the subject region, and selects a focus detection region that is closest to the position of the representative subject.

  Then, the system control circuit 201 connects a plurality of pixel data included in the region corresponding to the selected focus detection region for each of the A image and the B image stored in the RAM 2012 to generate a pair of image signals (for focus detection). Signal). For example, when each pixel has two photoelectric conversion units arranged in the horizontal direction, the system control circuit 201 generates an image signal by connecting a plurality of pixel data arranged in the horizontal direction. The system control circuit 201 handles a pair of image signals generated from the A image and the B image in the same manner as a pair of image signals obtained from the AF sensor 105, and obtains a focus state (defocus amount and direction).

  In step S505, the system control circuit 201 adjusts the focus distance of the photographing lens 102 by controlling the position of the focus lens 113 according to the lens driving amount and driving direction corresponding to the defocus amount and defocus direction obtained in step S504. To do.

  In step S506, the system control circuit 201 determines shooting conditions (aperture value (AV value), shutter speed (TV value), ISO sensitivity (ISO value)) using the image signal (A + B image) read in step S501. Although there is no particular limitation on the method for determining the photographing condition, here, the photographing condition corresponding to the luminance (Bv value) obtained based on the image signal is determined with reference to a pre-stored program diagram. . Note that the shooting condition may be determined using the luminance of the subject area detected by the subject detection process.

  The processing from S502 to S506 is targeted for the processing of the next frame (next execution of S501). Until the power switch is determined not to be ON in S301 in FIG. 3, the processing from S501 to S505 is repeatedly executed in S304 during the period in which the shooting mode is determined to be the moving image shooting mode in S302.

(Details of subject detection)
Next, the subject detection circuit 204 and subject detection processing will be described. In this embodiment, the subject detection circuit 204 is configured by a neocognitron which is a kind of CNN (convolutional neural network). A basic configuration of the subject detection circuit 204 will be described with reference to FIGS. FIG. 6 shows a basic configuration of the CNN that detects the subject from the input two-dimensional image data. The processing flow takes the left end as input and proceeds in the right direction. The CNN includes two layers called a feature detection layer (S layer) and a feature integration layer (C layer) as one set, and is configured hierarchically. The S layer corresponds to the convolution layer described in the prior art, and the C layer corresponds to the pooling layer or the subsampling layer.

In the CNN, first, the next feature is detected based on the feature detected in the previous layer in the S layer. Further, it has a configuration in which the features detected in the S layer are integrated in the C layer and transmitted to the next layer as a detection result in that layer.
The S layer is composed of a feature detection cell surface, and detects different features for each feature detection cell surface. Further, the C layer is composed of a feature integrated cell surface, and the detection result on the feature detection cell surface of the previous layer is pooled or subsampled. Hereinafter, the feature detection cell surface and the feature integrated cell surface are collectively referred to as a feature surface unless it is particularly necessary to distinguish between them. In the present embodiment, the output layer (the nth layer), which is the last layer, is configured by only the S layer without using the C layer.

  Details of the feature detection process on the feature detection cell plane and the feature integration process on the feature integration cell plane will be described with reference to FIG. One feature detection cell surface is composed of a plurality of feature detection neurons, and each feature detection neuron is connected to the C layer of the previous layer with a predetermined structure. One feature-integrated cell surface is composed of a plurality of feature-integrated neurons, and each feature-integrated neuron is connected to the S layer of the same hierarchy with a predetermined structure.

と表記する。また、Ｌ階層目のＣ層のＭ番目の細胞面内において、位置(ξ, ζ)の特徴統合ニューロンの出力値を

と表記する。その時、それぞれのニューロンの結合係数を

とすると、各出力値は以下のように表すことができる。 The output value of the feature detection neuron at the position (ξ, ζ) in the Mth cell plane of the Sth layer of the Lth layer shown in FIG.

Is written. Also, the output value of the feature integration neuron at the position (ξ, ζ) in the Mth cell plane of the Cth layer of the Lth layer

Is written. At that time, the coupling coefficient of each neuron

Then, each output value can be expressed as follows.

[数式２]

ここで、数式１におけるｆは活性化関数であり、例えばロジスティック関数や双曲正接関数などのシグモイド関数である。また、

は、Ｌ階層目のＳ層のＭ番目の細胞面における、位置(ξ, ζ)の特徴検出ニューロンの内部状態を表す。数式２は活性化関数を用いておらず、単純な線形和で表されている。 [Formula 1]

[Formula 2]

Here, f in Formula 1 is an activation function, for example, a sigmoid function such as a logistic function or a hyperbolic tangent function. Also,

Represents the internal state of the feature detection neuron at position (ξ, ζ) on the Mth cell surface of the Sth layer of the Lth layer. Equation 2 does not use an activation function and is represented by a simple linear sum.

と出力値

とは等しい。また、数式１の

を特徴検出ニューロンの結合先出力値と呼び、数式２の

を特徴統合ニューロンの結合先出力値と呼ぶ。 When the activation function is not used as in Equation 2, the internal state of the neuron

And output value

Is equal to In addition, Formula 1

Is called the connection destination output value of the feature detection neuron.

Is called the connection destination output value of the feature integration neuron.

が大きい場合、入力画像の画素位置(ξ, ζ)に、Ｌ階層目のＳ層のＭ番目の細胞面が検出する特徴が存在する可能性が高いことを意味する。またｎは数式１において、Ｌ−１階層目のＣ層のｎ番目の細胞面を意味しており、統合先特徴番号と呼ぶ。基本的にＬ−１階層目のＣ層に存在する全ての細胞面について積和演算を行う。（ｕ, ｖ）は、結合係数の相対位置座標であり、検出する特徴のサイズに応じて有限の範囲（ｕ, ｖ）において積和演算を行う。このような有限な（ｕ, ｖ）の範囲を受容野と呼ぶ。また受容野の大きさを、以下では受容野サイズと呼び、結合している範囲の横画素数×縦画素数で表す。 Here, ξ, ζ, u, v, and n in Equations 1 and 2 will be described. The position (ξ, ζ) corresponds to the position coordinate in the input image.

Is large, it means that there is a high possibility that the feature detected by the Mth cell surface of the Sth layer of the Lth layer exists at the pixel position (ξ, ζ) of the input image. Moreover, n means the nth cell surface of the C layer of the (L-1) th layer in Formula 1, and is called the integration destination feature number. Basically, the product-sum operation is performed on all the cell planes existing in the C layer of the (L-1) th layer. (U, v) is a relative position coordinate of the coupling coefficient, and a product-sum operation is performed in a finite range (u, v) according to the size of the feature to be detected. Such a finite range (u, v) is called a receptive field. The size of the receptive field is hereinafter referred to as a receptive field size, and is represented by the number of horizontal pixels × the number of vertical pixels in the combined range.

は、入力画像

である。ちなみにニューロンや画素の分布は離散的であり、結合先特徴番号も離散的なので、ξ，ζ，ｕ，ｖ，ｎは離散的な値をとる。ここでは、ξ，ζは非負整数、ｎは自然数、ｕ，ｖは整数とし、何れも有限な範囲を有する。 In Equation 1, L = 1, that is, in the S layer of the first hierarchy,

The input image

It is. Incidentally, since the distribution of neurons and pixels is discrete and the connection feature number is also discrete, ξ, ζ, u, v, and n take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, u and v are integers, and both have a finite range.

は、所定の特徴を検出するための結合係数であり、結合係数を適切な値に調整することによって、所定の特徴を検出可能になる。この結合係数の調整が学習であり、ＣＮＮの構築においては、さまざまなテストパターンを用いて、

が適切な出力値になるように、結合係数を繰り返し徐々に修正していくことで結合係数を調整する。 In Equation 1

Is a coupling coefficient for detecting a predetermined feature, and the predetermined feature can be detected by adjusting the coupling coefficient to an appropriate value. Adjustment of this coupling coefficient is learning, and in the construction of CNN, using various test patterns,

The coupling coefficient is adjusted by repeatedly and gradually correcting the coupling coefficient so that becomes an appropriate output value.

は、２次元のガウシアン関数を用いており、以下の数式３のように表すことができる。
[数式３]

ここでも、（ｕ，ｖ）は有限の範囲を有し、特徴検出ニューロンの場合と同様、範囲を受容野、範囲の大きさを受容野サイズと呼ぶ。ここではＬ階層目のＳ層のＭ番目の特徴のサイズに従って、受容野サイズの値を適宜設定することができる。数式３中のσは特徴サイズ因子であり、受容野サイズに応じて適宜定めることができる定数であってよい。例えば、受容野の一番外側の値がほぼ０とみなせるような値になるように特徴サイズ因子σを設定することができる。このように、本実施形態の被写体検出回路２０４は、上述した演算を各階層で行い、最終階層（ｎ階層目）のＳ層において被写体検出を行うＣＮＮによって構成される。 Next, in Equation 2

Uses a two-dimensional Gaussian function and can be expressed as Equation 3 below.
[Formula 3]

Again, (u, v) has a finite range, and the range is called the receptive field and the size of the range is called the receptive field size, as in the case of the feature detection neuron. Here, the value of the receptive field size can be appropriately set according to the size of the Mth feature of the Sth layer of the Lth layer. Σ in Equation 3 is a feature size factor, and may be a constant that can be appropriately determined according to the receptive field size. For example, the feature size factor σ can be set so that the outermost value of the receptive field becomes a value that can be regarded as almost zero. As described above, the subject detection circuit 204 of the present embodiment is configured by a CNN that performs the above-described calculation in each layer and performs subject detection in the S layer of the final layer (nth layer).

の具体的な調整（学習）方法について説明する。学習は、ＣＮＮに特定の入力画像（テストパターン）を与えて得られるニューロンの出力値と、教師信号（そのニューロンが出力すべき出力値）との関係に基づいて、結合係数

を修正することである。本実施形態の学習では、最終階層（ｎ階層目）の特徴検出層Ｓについては最小二乗法を用いて結合係数を修正する。また、他の階層（１〜ｎ−１階層目）の特徴検出層Ｓについては、誤差逆伝搬法を用いて結合係数を修正する。最小二乗法や誤差逆伝搬法を用いた結合係数の修正手法は例えば非特許文献１に記載されるような公知技術を用いることができるため、詳細についての説明は省略する。 (Subject detection learning method)
Coupling factor

A specific adjustment (learning) method will be described. Learning is based on the relationship between the output value of a neuron obtained by giving a specific input image (test pattern) to the CNN and the teacher signal (the output value that the neuron should output).

Is to fix. In the learning of the present embodiment, for the feature detection layer S of the last layer (nth layer), the coupling coefficient is corrected using the least square method. For the feature detection layer S of the other layers (1st to (n-1) th layers), the coupling coefficient is corrected using the error back propagation method. Since a known technique as described in Non-Patent Document 1, for example, can be used as the coupling coefficient correction method using the least square method or the error back propagation method, a detailed description thereof will be omitted.

  A large number of patterns to be detected and patterns that should not be detected are prepared as test patterns for learning. Each test pattern has image data and a corresponding teacher signal. The image data corresponding to the pattern to be detected is a teacher signal such that the output of the neuron corresponding to the area where the pattern to be detected exists is 1 on the feature detection cell surface in the final hierarchy. On the other hand, for image data corresponding to a pattern that should not be detected, a teacher signal is given so that the output of the neuron corresponding to the region where the pattern that should not be detected exists is -1.

  In the present embodiment, a learning model 206 for the photometric sensor is prepared by learning with a test pattern using image data based on the image signal obtained by the photometric sensor 108. Further, a learning model 207 for the image sensor is prepared by learning with a test pattern using image data based on the image signal obtained by the image sensor 111. In this way, by separately performing learning with the image signal obtained by the photometric sensor 108 and learning with the image signal obtained by the image sensor 111, differences in optical path, element, image processing, etc. were reflected. A learning model suitable for the image signal of each image sensor can be generated.

  Note that image data for generating a learning model for the image sensor 111 can be easily obtained by executing a still image shooting process or a moving image shooting process, while generating a learning model for the photometric sensor 108. Therefore, it is not always easy to obtain image data. This is because the image data obtained by the photometric sensor 108 is not stored in the image storage memory 202.

  Therefore, image data corresponding to image data based on the image signal obtained by the photometric sensor 108 may be generated from the image signal obtained by the image sensor 111. For example, based on image data generated by photographing the same subject with the image sensor 111 and the photometric sensor 108, differences in optical paths, elements, image processing, etc. reflected in the image data are detected. The system control circuit 201 applies the correction corresponding to the detected difference to the image data based on the image signal obtained by the image sensor 111, thereby corresponding to the image data based on the image signal obtained by the photometric sensor 108. Image data can be generated. The correction method is not limited. For example, the sharpness difference can be realized by applying a low-pass filter or contrast correction, and the color difference can be realized by color conversion by applying a lookup table. These pieces of information necessary for correction can be stored in the ROM 2011, for example. Thereby, the image data for generating the learning model for the photometric sensor 108 can be acquired in the same manner as the image data for generating the learning model for the image sensor 111. It should be noted that the learning model can be generated by another device.

  As described above, according to the present embodiment, in an apparatus that can perform subject detection on image signals obtained by different image sensors, subject detection is performed by using a subject detection parameter in accordance with image characteristics. Accuracy can be improved.

(Other embodiments)
In the above-described embodiment, when one image pickup apparatus has two image pickup elements (photometric sensor 108 and image pickup element 111) having different optical paths, a learning model for subject detection is used for each image pickup element used at the time of subject detection. An example of a configuration for switching is shown. However, the essence of the present invention is that for object detection processing that takes into account the characteristics of an imaging optical system (such as optical path and lens aberration) reflected in an image signal or image data for subject detection, an image sensor, and signal processing. It is to use parameters. Therefore, in an imaging apparatus having one imaging element, a configuration using different subject detection parameters depending on a shooting lens used for shooting and a configuration using different subject detection parameters depending on the imaging device are also included in the present invention. included.

  For example, there are smartphones and tablet terminals that include a plurality of image sensors with different light receiving sensitivities, such as RGB image sensors and infrared sensors. Alternatively, there are smartphones and tablet terminals that include a plurality of imaging optical systems with different optical magnifications such as standard, wide-angle, and telephoto. The present invention can also be applied to these smartphones and tablet terminals.

  A smartphone or tablet terminal downloads or updates a learning model for subject detection processing that takes into account characteristics such as the imaging optical system, imaging device, or signal processing from the network via wireless communication or wired communication. It is good also as a structure. At this time, the smartphone or tablet terminal obtains a plurality of learning models for the same subject for each characteristic of the image based on the photographing optical system, the image sensor, or signal processing.

  Alternatively, the server or the edge computer may be configured to include a subject detection circuit and a plurality of learning models for subject detection processing in consideration of image characteristics. A server or edge computer receives an image transmitted from an imaging device, a smartphone, etc., selects a learning model according to the characteristics of the received image, performs subject detection processing, and captures the detection result as the image transmitted You may make it transmit to an apparatus or a smart phone.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a computer-readable storage medium, and one or more processors of the computer of the system or apparatus execute the program. It can also be realized by executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely specific examples for the purpose of assisting understanding of the present invention, and are not intended to limit the present invention to the above-described embodiments in any way. All embodiments that fall within the scope of the claims are encompassed by the present invention.

DESCRIPTION OF SYMBOLS 100 ... Digital single-lens reflex camera, 101 ... Main body, 102 ... Sensor, 108 ... Photometric sensor, 111 ... Image sensor, 204 ... Subject detection circuit, 206 ... Learning model for photometric sensor, 207 ... Learning model for image sensor

Claims (19)

Subject detection means for applying subject detection processing to an image using parameters generated based on machine learning;
Storage means for storing a plurality of parameters used for the subject detection processing;
Selection means for selecting parameters used by the subject detection means from parameters stored in the storage means according to the characteristics of the image to which subject detection processing is applied;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the selection unit selects a learning model to be used by the subject detection unit according to an imaging element that has generated the image. The first learning model used when the subject detection processing is applied to the image generated by the first image sensor is machine-learned using an image corresponding to the first image sensor. Learning model,
The second learning model used when the subject detection processing is applied to the image generated by the second image sensor is machine-learned using an image corresponding to the second image sensor. The image processing apparatus according to claim 2, wherein the image processing apparatus is a learning model. The first learning model is a learning model in which machine learning is performed using an image generated by the first image sensor,
The image processing apparatus according to claim 3, wherein the second learning model is a learning model in which machine learning is performed using an image generated by the second image sensor. The first learning model is a learning model in which machine learning is performed using an image generated by the first image sensor,
The image processing apparatus according to claim 3, wherein the second learning model is a learning model in which machine learning is performed using an image obtained by correcting an image generated by the first image sensor. .   The image processing apparatus according to claim 1, wherein the selection unit selects a parameter to be used by the subject detection unit in accordance with an optical system used for capturing the image. A first learning model used when applying the subject detection process to an image photographed using the first optical system performs machine learning using an image corresponding to the first optical system. Learning model,
The second learning model used when the subject detection process is applied to an image photographed using the second optical system performs machine learning using an image corresponding to the second optical system. The image processing apparatus according to claim 6, wherein the learning model is a broken learning model. The first learning model is a learning model in which machine learning is performed using an image photographed using the first optical system,
The image processing apparatus according to claim 7, wherein the second learning model is a learning model in which machine learning is performed using an image photographed using the second optical system. The first learning model is a learning model in which machine learning is performed using an image photographed using the first optical system,
The image according to claim 7, wherein the second learning model is a learning model in which machine learning is performed using an image obtained by correcting an image photographed using the first optical system. Processing equipment.   The image processing apparatus according to claim 1, further comprising a communication unit that acquires a learning model used by the subject detection unit from the storage unit via a network.   The image processing apparatus according to claim 1, wherein the machine learning uses a convolutional neural network (CNN). A first image sensor;
A second imaging device;
An image processing apparatus comprising: the image processing apparatus according to any one of claims 1 to 11,
The imaging apparatus according to claim 1, wherein the selection unit selects a learning model to be used by the subject detection unit according to a shooting mode.   The imaging apparatus according to claim 12, wherein the shooting mode is either a moving image shooting mode or a still image shooting mode.   The shooting mode is any of a shooting mode using the first image sensor and not using the second image sensor, and a shooting mode using the second image sensor and not using the first image sensor. The imaging apparatus according to claim 12 or 13, wherein The shooting mode using the first image sensor and not using the second image sensor is a shooting mode using the optical viewfinder,
The shooting mode that uses the second image sensor and does not use the first image sensor is a shooting mode that does not use an optical viewfinder.
The imaging apparatus according to claim 14.   The image pickup apparatus according to claim 12, wherein the first image pickup element is an image pickup element for acquiring an image for exposure control. An image processing method executed by an image processing apparatus,
A subject detection step of applying subject detection processing to an image using a learning model generated based on machine learning;
A selection step of selecting a learning model to be used in the subject detection step from storage means for storing a plurality of learning models to be used for the subject detection processing, according to the characteristics of the image to which the subject detection processing is applied;
An image processing method comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 11.   The program for functioning the computer which an imaging device has as an image processing apparatus which the imaging device of any one of Claim 12 to 16.

Images