JP2020008899A - Image processing device, image processing method, program and storage medium - Google Patents

Abstract

To improve accuracy of detecting a subject in machine learning.SOLUTION: An image processing device comprises: a detection unit of detecting a target subject by referring to dictionary data acquired in machine learning corresponding to the target subject to be detected; a selection unit of selecting any piece of dictionary data from among a plurality of pieces of dictionary data with respect to the target subject; and a control unit of controlling the detection unit so as to detect the target subject using the selected dictionary data and different dictionary data from the selected dictionary data in the case where a detection evaluation value at detection of the subject using the dictionary data selected by the selection unit is lower than a predetermined value.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image processing device having a subject detection function.

  An image processing method of automatically detecting a specific subject pattern from an image is a very useful technique, for example, such that a human face region can be specified from an image. As disclosed in Patent Literature 1, an imaging apparatus such as a digital camera or a digital video camera detects an area of a specific subject pattern such as a face area of a person from a captured image, and focuses and exposes the detected area. Optimization has been done.

  Further, there is a technique called deep learning as disclosed in Non-Patent Document 1 in order to learn and recognize a subject in an image. As a typical technique of deep learning, there is a technique called a convolutional neural network (hereinafter, referred to as CNN). A general CNN consists of multi-stage calculations. At each stage of the CNN, a convolution operation is performed to spatially integrate the local features of the image and input to the next-stage neuron in the intermediate layer. In addition, an operation called "pooling" or "subsampling" is performed to compress the feature in the spatial direction. The CNN can acquire a complex feature expression through such multi-stage feature conversion. Therefore, category recognition and subject detection of the subject in the image can be performed with high accuracy based on the feature amount. In machine learning represented by CNN, an image signal and a teacher signal are learned as a set. As a result of the learning, dictionary data, which is a processing parameter for subject detection, is generated.

JP 2005-318554 A JP-A-2005-5237

Alex Krizhevsky, Ilya Sutskever, Geoffrey E .; Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems 25 (NIPS'12), 2012.

  When photographing, subject characteristics may differ depending on the shooting scene. Here, the subject characteristic is a difference in how the person looks when the subject is a person, and is a characteristic that affects the detection difficulty of the posture of the person, the overlap of the person, and the like. By learning dictionary data for each subject characteristic and using dictionary data specialized for a predetermined subject in the detection processing, detection accuracy can be improved.

  Therefore, in order to improve the detection accuracy of the subject, a method of switching and using dictionary data having appropriate subject characteristics depending on the situation is conceivable. In Patent Literature 2, for a plurality of distance ranges set in accordance with the distance from an imaging device to a subject, learning feature amounts for each of the distance ranges are stored, and the learning feature amount is compared with the feature amount of the subject. Is detected. Although there is a known technique for switching the dictionary for subject detection, it does not disclose switching of dictionary data having different subject characteristics. Further, in the method using the dedicated dictionary data corresponding to the subject characteristics, when a unique subject characteristic occurs at the time of shooting, the detection accuracy may be lower than that of general-purpose dictionary data.

  The present invention has been made in view of the above-described problem, and an object of the present invention is to improve the accuracy of subject detection by machine learning.

  The image processing apparatus according to the present invention includes a detection unit that detects the target object by referring to dictionary data acquired by machine learning corresponding to the target object to be detected from the acquired image. Selecting means for selecting any dictionary data from a plurality of dictionary data for the target subject; and a detection evaluation value when the subject is detected using the dictionary data selected by the selecting means, is greater than a predetermined value. Also, when low, the selected dictionary data, and control means for controlling the detection means, so as to detect the target object using dictionary data different from the selected dictionary data, Wherein the plurality of dictionary data includes general-purpose dictionary data and a plurality of dedicated dictionary data, and each of the plurality of dedicated dictionary data is When the subject to be read is under the condition corresponding to each of the plurality of special dictionary data, the general dictionary data has a higher probability of being able to detect the target subject than the general dictionary data. The data is dictionary data capable of detecting the target subject under more conditions than each of the plurality of dedicated dictionary data.

  According to the present invention, it is possible to improve the accuracy of subject detection by machine learning.

FIG. 1 is a side sectional view of a digital single-lens reflex camera which is an embodiment of an image processing apparatus according to the present invention. FIG. 2 is a diagram illustrating a block configuration of a digital single-lens reflex camera. FIG. 4 is a diagram illustrating an example of dictionary data according to subject characteristics. 5 is a flowchart illustrating a procedure of an imaging operation of the digital single-lens reflex camera. 5 is a flowchart illustrating a procedure of detecting a subject of the digital single-lens reflex camera. FIG. 6 is a state transition diagram of dictionary data used for subject detection. FIG. 2 is a schematic diagram illustrating an example of the entire configuration of a CNN. FIG. 2 is a schematic diagram illustrating an example of a partial configuration of a CNN.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments of the present invention show preferred embodiments of the present invention, and do not limit the scope of the present invention. The following embodiments will be described using an example of a digital single-lens reflex camera, but may be a mirrorless camera, a video camera, a surveillance camera, a smartphone with a camera function, or the like having a function of detecting a target subject. . Also, the present invention can be applied to a case where a personal computer, a cloud computer, or an edge computer that receives a moving image captured by these cameras performs a process of detecting a target subject. is there.

(Configuration of imaging device)
FIG. 1 is a side sectional view of a digital single-lens reflex camera 100 which is an embodiment of the image processing apparatus of the present invention, and FIG. 2 is a diagram showing a block configuration of the digital single-lens reflex camera 100.

  In FIG. 1, a digital single-lens reflex camera 100 includes a camera body 101 and a photographing lens 102 detachably mounted on the camera body 101. 2, the camera main body 101 includes a system control unit 201 that controls the entire digital single-lens reflex camera 100. The system control unit 201 is connected to a quick return mirror 103, a focus detection sensor 105, a photometric sensor 108, a focal plane shutter 110, an image sensor 111, a display unit 112, and a mount contact group 115, which will be described later. The system control unit 201 is further connected to an image storage unit 202, an operation unit 203, a subject detection unit 204, and a storage unit 210 that stores dictionary data, which will be described later. The system control unit 201 includes a multi-core CPU, a RAM, and a ROM that can perform a plurality of tasks in parallel, and controls each unit of the camera body 101 and the photographing lens 102.

  Hereinafter, the configuration of each unit of the digital single-lens reflex camera 100 will be described with reference to FIGS. 1 and 2. The taking lens 102 is replaceable, and the camera body 101 and the taking lens 102 are also electrically connected via a mount contact group 115. A focusing lens 113 and an aperture shutter 114 are arranged in the taking lens 102, and are configured so that the amount of light taken into the camera and the focus can be adjusted by control via a mount contact group 115.

  The quick return mirror 103 includes a main mirror 103a and a sub mirror 103b. The main mirror 103a is configured by a half mirror. The main mirror 103a is disposed obliquely on the photographing optical path in the viewfinder observation state, and reflects a light beam incident from the photographing lens 102 to the viewfinder optical system. On the other hand, the transmitted light enters the focus detection sensor 105 via the sub mirror 103b.

  The focus detection sensor 105 has a focus detection line sensor disposed on the secondary imaging plane of the photographing lens 102, and generates an AF signal (automatic focus control signal) indicating the focus state of the photographing lens 102 by a phase difference detection method. I do. The generated AF signal is transmitted to the system control unit 201, and the system control unit 201 detects the focus state of the focusing lens 113 based on the AF signal. Further, the system control unit 201 performs focus adjustment by controlling the driving of the focusing lens 113 based on the result of focus detection.

  A focusing plate 106 is arranged on a predetermined image forming plane of the photographing lens 102 in the finder optical system. The light path of the light passing through the focus plate 106 is changed by the pentaprism 107 and guided to the eyepiece 109. The photographer can check the photographing screen and photographing information by observing the focus plate 106 via the eyepiece 109.

  A photometric sensor 108 is arranged beside the eyepiece 109. The photometric sensor 108 photoelectrically converts the irradiated light, and generates image data having a luminance signal and a color difference signal. The photometric sensor 108 also generates an AE signal (automatic exposure control signal) based on the generated image data, and transmits the signal to the system control unit 201. The system control unit 201 performs exposure control using the received AE signal. The subject detection unit 204 performs subject detection based on the AE signal. The system control unit 201 optimizes focus adjustment and exposure control based on the subject detected by the subject detection unit 204.

  Behind the quick return mirror 103, a focal plane shutter 110 and an image sensor 111 are arranged. When performing exposure, the main mirror 103a and the sub-mirror 103b are retracted from the imaging optical path, and the focal plane shutter 110 is opened, so that the image sensor 111 is exposed. The focal plane shutter 110 shields the image sensor 111 when no image capturing is performed, and opens to guide a subject light beam to the image sensor 111 during image capturing.

  The imaging element 111 is configured by a CCD, a CMOS sensor, or the like, and includes an infrared cut filter, a low-pass filter, and the like. The image sensor 111 photoelectrically converts a subject image formed by passing through the imaging optical system of the imaging lens 102, generates an image signal, and transmits the image signal to the system control unit 201. The system control unit 201 generates image data from the received image signal, stores it in the image storage unit 202, and displays the image data on the display unit 112 such as an LCD.

  The operation unit 203 detects a user operation performed via a release button (not shown), a switch, a connected device, or the like, and transmits a signal corresponding to the operation content to the system control unit 201. When the release button is half-pressed, the release switch SW1 is turned on, and a shooting preparation operation such as AF (auto focus) or AE (auto exposure control) is performed. When the release button is fully pressed, the release switch SW2 is turned on, and a still image capturing operation is performed. A still image taken immediately before is displayed on the display unit 112 for a certain period of time so that the user can confirm the photographing result.

  Next, the subject detection operation of the digital single-lens reflex camera configured as described above will be described.

(Dictionary switching for subject detection)
The subject detection unit 204 detects a subject from the AE signal described above. In the subject detection unit 204, processing parameters for detecting a subject are determined using dictionary data based on machine learning stored in the storage unit 210. The characteristics of the subject may vary depending on the shooting scene. Therefore, dictionary data is prepared for each subject characteristic, and the detection accuracy of the subject can be improved by using the dictionary data according to the scene. That is, it has a plurality of dictionary data, and selects and uses the dictionary data according to the situation.

  As illustrated in FIG. 2, the storage unit 210 stores general-purpose dictionary data 205 and two or more dedicated dictionary data 206-1 to 206-N (N is an integer of 2 or more) corresponding to subject characteristics. I have. As a method for selecting dictionary data, there is a method in which the user selects the dictionary data via the operation unit 203 according to the purpose. Alternatively, a method in which the system control unit 201 selects appropriate dictionary data according to the situation may be used.

  An example of dictionary data and characteristics of a subject will be described with reference to FIG. FIG. 3 shows a table in which the subject is a person, the ID is defined in the first column, the definition of dictionary data is defined in the second column, and the example of the subject is described in the third column. ID1 is general-purpose dictionary data, which is dictionary data obtained by machine learning from image data of general human subjects and teacher data. ID2 to ID5 are dedicated dictionary data, and are dictionary data specialized and learned for a specific person. The dedicated dictionary data can be said to be, for example, dictionary data divided by at least one element of the posture of the subject, the number of the subjects, the overlap of the subjects, the presence / absence of a decoration for the subject, and the type. ID2 indicates a state in which the human subject has a peculiar posture, ID3 indicates a state in which the human subject overlaps, ID4 indicates a state in which many human subjects exist, and ID5 indicates a state in which there is a decoration on the head or the like of the human subject. That is, ID1 to ID5 are dictionary data for detecting a common specific subject (here, a person). If ID2 to ID4 are used, the subject can be detected with a higher probability than the general-purpose dictionary data of ID1 if the subject satisfies the specific condition corresponding to each. The probability that the subject can be detected is lower than that of the general-purpose dictionary data. If the general dictionary data of ID1 is used, the subject can be detected under a plurality of conditions or more conditions than each of the dedicated dictionary data. However, under the conditions corresponding to any of ID2 to ID, ID2 can be detected. The probability that the subject can be detected is lower than that of any of the dictionary data of ID4 to ID4. Here, the subject is described as a person, but is not limited to this, and the detection target is a part of a person (for example, a head), a specific individual, a specific animal, a specific object, or a specific object. Scenes and the like.

  By setting appropriate dictionary data according to the characteristics of the subject in the shooting scene, highly accurate subject detection can be performed. However, although the dedicated dictionary data has high detection accuracy for a specific subject, generalization ability is lost. Therefore, when peculiar subject characteristics occur at the time of shooting, the detection accuracy of the dedicated dictionary data may be lower than that of the general dictionary data. Therefore, if the set dictionary data is dedicated dictionary data and the detection evaluation value is low, a plurality of dictionary data is used, such as using both general-purpose dictionary data and the set dedicated dictionary data. For example, by alternately using general-purpose dictionary data and special-purpose dictionary data, it is possible to avoid a situation where the detection accuracy continuously decreases.

(Processing flow of imaging device)
Next, an imaging operation of the digital single-lens reflex camera having the above configuration will be described with reference to FIGS. FIG. 4 is a flowchart illustrating a procedure of an imaging operation of the digital single-lens reflex camera 100. The operation of this flowchart is realized by the system control unit 201 expanding the program stored in the ROM into the RAM and executing the program.

  In step S401, the user sets dictionary data in the subject detection unit 204 via the operation unit 203. One of the general dictionary data 205 and the special dictionary data 206-1 to 206-N is set. If there is no setting by the user, the initial setting is the general dictionary data 205. Here, a series of processing from step S402 to step S409 described below corresponds to one frame of the camera.

  In step S402, the system control unit 201 detects the states of the release switch SW1 and the release switch SW2, and if either of them is on, advances the frame by one and advances to step S403. If both the release switches SW1 and SW2 are off, the process ends.

  In step S403, the system control unit 201 causes the photometric sensor 108 to perform charge accumulation, and reads out the generated image signal as an AE signal. Further, the system control unit 201 causes the focus detection sensor 105 to perform charge accumulation, and reads out the generated image signal as an AF signal.

  In step S404, the subject detection unit 204 performs subject detection using the AE signal read in step S403 as an input image. Details of the subject detection processing will be described later. In step S405, the system control unit 201 selects the focus detection area closest to the position of the subject detected in step S404, and detects the focus state of the selected focus detection area using the AF signal acquired in step S403. I do. If no subject is detected in step S404, focus detection is performed for all focus detection areas, and then a focus detection area having a focus closest to the camera is selected.

  In step S406, the system control unit 201 adjusts the focus position of the focusing lens 113 based on the focus state of the focus detection area selected in step S405. In step S407, the system control unit 201 performs an automatic exposure calculation by a known method using the AE signal read in step S403, and sets an aperture value (AV value), a shutter speed (TV value), and an ISO sensitivity (ISO value). To determine. Here, the AV value, the TV value, and the ISO value are determined using a program diagram stored in advance.

  In step S408, the system control unit 201 detects the state of the release switch SW2, and proceeds to step S409 if the release switch SW2 is on. On the other hand, if the release switch SW2 is off, the process returns to step S402.

  In step S409, the system control unit 201 raises the main mirror 103a and the sub-mirror 103b to retreat from the optical path, and exposes the image sensor 111. The exposed image sensor 111 generates an image signal and transmits the image signal to the system control unit 201. Then, the system control unit 201 generates image data based on the image signal received from the imaging element 111, stores the image data in the image storage unit 202, and displays the image data on the display unit 112. The above is the operation procedure of the digital single-lens reflex camera in the present embodiment.

(Processing flow of subject detection)
Next, the flow of the subject detection process in step S404 in FIG. 4 will be described with reference to FIG.

  In step S501, the system control unit 201 determines the type of dictionary data used by the subject detection unit 204. If the dictionary data is the general dictionary data 205 (NO in step S501), the process proceeds to step S504. In step S504, the subject detection unit 204 performs subject detection from an image signal based on the general-purpose dictionary data using a CNN method described below. If the dictionary data is any of the dedicated dictionary data 206-1 to 206-N (determined as YES in step S501), the process proceeds to step S502.

  In step S502, the subject detection unit 204 performs subject detection from the image signal based on the set dedicated dictionary data. Next, in step S503, the system control unit 201 determines whether the detection evaluation value of the subject detected in step S502 is lower than a predetermined value. If the detection evaluation value is equal to or greater than the predetermined value (NO in step S503), it is determined that the target subject has been detected, and the detection process ends. If the detection evaluation value is lower than the predetermined value (YES in step S503), it is determined that the target subject has not been detected. In this case, there is a possibility that a special situation occurs, and a special dictionary having a low generalization ability is in a state where detection is poor. Therefore, in step S504, the subject detection unit 204 causes the subject detection processing to be performed again using the general-purpose dictionary data. The processing ends as the result of subject detection.

  In the above description, it has been described that if the evaluation value of the subject detection by the dedicated dictionary data is low, the subject detection using the general dictionary data is performed. In this case, subject detection is performed twice per frame, so that the processing load is high and the delay of the imaging process is large. Therefore, if the evaluation value of subject detection by the dedicated dictionary data is low, a method of processing while switching between the dedicated dictionary data and the general-purpose dictionary data in the time direction can be considered. In this case, since the number of detections per frame is one, the delay of the imaging process does not increase. In this method of processing while switching between the dedicated dictionary data and the general-purpose dictionary data, when the detection evaluation value of the dedicated dictionary data increases, the process shifts to a detection process using only the dedicated dictionary data. As a result, even if a special situation where the dedicated dictionary data is not good temporarily occurs, it is possible to return to the detection process utilizing the characteristics of the dedicated dictionary.

  The use state of the dictionary data of the subject detection unit 204 will be described with reference to the state transition diagram of FIG. States 601, 602, and 603 indicate the use states of dictionary data. State 601 is a state in which general-purpose dictionary data is used. State 602 is a state in which special-purpose dictionary data is used. Indicates the state of use. The state 601 is an initial state, and arrows between the states indicate conditions for state transition. In the state 601, the state transits to the state 602 if the user specifies the dedicated dictionary data via the operation unit 203. In the state 602, the state transits to the state 601 if the user specifies general-purpose dictionary data via the operation unit 203. Also, in the state 602, if the evaluation value of subject detection by the dedicated dictionary data is low, the state transits to the state 603. In the state 603, the state transits to the state 601 if the user specifies general-purpose dictionary data via the operation unit 203. Also, in the state 603, if the evaluation value of subject detection by the dedicated dictionary data is high, the state transits to the state 602. Subject detection processing is performed by the above state transition.

Here, when the dedicated dictionary data is selected, the general dictionary data is used together when the subject detection evaluation value is low. However, when the general dictionary data is selected, the dedicated dictionary data is used even if the subject detection evaluation value is low. The reason why data is not used together will be described. If the evaluation value of subject detection is low even though the user has selected the dedicated dictionary data, it is highly likely that the subject to be detected is not under the conditions corresponding to the selected dedicated dictionary data. Therefore, the probability of detecting a subject can be increased by using general-purpose dictionary data in combination with the general-purpose dictionary data in order to detect a subject that does not correspond to the dedicated dictionary data. Conversely, the fact that the user has not selected the dedicated dictionary data suggests that it is highly likely that the subject to be detected is not under conditions corresponding to the dedicated dictionary data. For this reason, even if the evaluation value of subject detection in the case of using general-purpose dictionary data is low, the probability of detecting a subject cannot be expected to increase even if dedicated dictionary data is used in combination. Therefore, when the general dictionary data is selected, the state does not transition to the state in which the dedicated dictionary data is used even if the evaluation value of the subject detection is low. (Detailed description of CNN)
In the present embodiment, the subject detection unit 204 is configured by a CNN (Convolutional Neural Network). The basic configuration of the CNN will be described with reference to FIGS. FIG. 7 is a diagram showing a basic configuration of a CNN for detecting a subject from input two-dimensional image data. In the processing flow, the left end is input and the processing proceeds rightward. The CNN is a set of two layers called a feature detection layer (S layer) and a feature integration layer (C layer), which are hierarchically configured.

  The CNN first detects the next feature in the S layer based on the feature detected in the previous stage layer. In addition, features detected in the S layer are integrated in the C layer, and the detection result in that layer is sent to the next layer.

  The S layer includes a feature detection cell surface, and detects a different feature for each feature detection cell surface. The C layer is composed of a feature-integrated cell surface, and pools detection results on the preceding feature detection cell surface. Hereinafter, the feature detection cell surface and the feature integrated cell surface are collectively referred to as a feature surface, unless it is particularly necessary to distinguish them. In the present embodiment, the output layer, which is the final stage layer, includes only the S layer without using the C layer.

The details of the feature detection process on the feature detection cell surface and the feature integration process on the feature integration cell surface will be described with reference to FIG. The feature detection cell surface is composed of a plurality of feature detection neurons, and the feature detection neurons are connected to the C layer of the previous stage in a predetermined structure. The feature-integrated cell surface is composed of a plurality of feature-integrated neurons, and the feature-integrated neurons are connected to the same layer of the S layer by a predetermined structure. In the Mth cell plane of the L-th layer S layer shown in FIG. 8, the output value of the feature detection neuron at the position (ξ, ζ) is represented by y ^LS _M (ξ, ζ), th in the cell surface, the position (xi], zeta) the output value of the feature integration neuron y ^LC _M (ξ, ζ) and denoted. At that time, the coupling coefficient w ^LS _M of each neuron (n, u, v), w LC M (u, v) and when each output value can be expressed as follows.

…（１）

…（２）
式（１）のｆは、活性化関数であり、ロジスティック関数や双曲正接関数などのシグモイド関数であれば何でもよい。ｕ^LS _M(ξ,ζ)は、Ｌ階層目Ｓ層のＭ番目細胞面における、位置(ξ,ζ)の特徴検出ニューロンの内部状態である。式（２）は活性化関数を用いず単純な線形和をとっている。式（２）のように活性化関数を用いない場合は、ニューロンの内部状態ｕ^LC _M(ξ,ζ)と出力値ｙ^LC _M(ξ,ζ)は等しい。また、式（１）のｙ^L-1C _n(ξ+u,ζ+v)、式（２）のｙ^LS _M(ξ+u,ζ+v)をそれぞれ特徴検出ニューロン、特徴統合ニューロンの結合先出力値と呼ぶ。

… (1)

… (2)
F in Expression (1) is an activation function, and any sigmoid function such as a logistic function or a hyperbolic tangent function may be used. u ^LS _M (ξ, で) is the internal state of the feature detection neuron at the position (ζ, ζ) on the M-th cell surface of the L-th hierarchical S layer. Equation (2) takes a simple linear sum without using an activation function. If without activation function as in Equation (2), the internal state of the neuron u ^LC _M (ξ, ζ) and the output value y ^LC _M (ξ, ζ) are equal. In addition, y ^L-1C _n (ξ + u, ζ + v) in equation (1) and y ^LS _M (ξ + u, ζ + v) in equation (2) are respectively connected to the feature detection neuron and the feature integration neuron. It is called the first output value.

Ξ, ζ, u, v, and n in Equations (1) and (2) will be described. The position (ξ, ζ) corresponds to the position coordinates in the input image. For example, if y ^LS _M (ξ, ζ) has a high output value, the pixel position (ξ, ζ) of the input image is placed on the L level. It means that there is a high possibility that the feature to be detected in the Mth cell surface of the eye S layer exists. Further, n in Expression (1) means the n-th cell surface of the C layer of the (L-1) th layer, and is referred to as an integration destination feature number. Basically, a product-sum operation is performed for all cell planes existing in the C layer of the (L-1) th layer. (U, v) is a relative position coordinate of the coupling coefficient, and performs a product-sum operation in a finite range (u, v) according to the size of the feature to be detected. Such a finite (u, v) range is called a receptive field. The size of the receptive field is hereinafter referred to as the receptive field size, and is represented by the number of horizontal pixels × the number of vertical pixels in the combined range.

In equation (1), in L = 1, that is, in the first S layer, y ^L-1C _n (ξ + u, ζ + v) in equation (1) is equivalent to the input image y ^in-image (ξ + u, ζ + v). Incidentally, since the distribution of neurons and pixels is discrete and the connection destination feature number is also discrete, ξ, ζ, u, v, and n are not continuous variables but take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, both of which have a finite range.

W ^LS _M (n, u, v) in the equation (1) is a coupling coefficient distribution for detecting a predetermined feature, and adjusting this to an appropriate value to detect the predetermined feature. Becomes possible. A learning adjustment of the coupling coefficient distribution, in the construction of CNN, presents a variety of test _{^{patterns, y LS M (ξ, ζ}} ) as is appropriate output value gradually repeatedly coupling coefficient The correction is performed to adjust the coupling coefficient.

Then, w ^LC _M in formula (2) (u, v) is using a two-dimensional Gaussian function can be expressed as the following equation (3).

…（３）
ここでも、（ｕ，ｖ）は有限の範囲としているので、特徴検出ニューロンの説明と同様に、有限の範囲を受容野といい、範囲の大きさを受容野サイズと呼ぶ。この受容野サイズは、ここではＬ階層目Ｓ層のＭ番目特徴のサイズに応じて適当な値に設定すればよい。式（３）数中の、σは特徴サイズ因子であり、受容野サイズに応じて適当な定数に設定しておけばよい。具体的には、受容野の一番外側の値がほぼ０とみなせるような値になるように設定するのがよい。上述のような演算を各階層で行うことにより、最終階層のＳ層において、被写体検出を行うのが、本実施形態におけるＣＮＮの構成である。

… (3)
Here, since (u, v) is a finite range, the finite range is called a receptive field, and the size of the range is called a receptive field size, as in the description of the feature detection neuron. Here, the receptive field size may be set to an appropriate value according to the size of the M-th feature in the L-th layer and the S-layer. In the equation (3), σ is a feature size factor, and may be set to an appropriate constant according to the size of the receptive field. Specifically, it is preferable that the outermost value of the receptive field be set to a value that can be regarded as substantially zero. The configuration of the CNN according to the present embodiment is such that the above-described calculation is performed in each layer, and the subject is detected in the S layer of the last layer.

(CNN learning method)
Specific coupling coefficient _{^{w LS M (n, u,}} v) adjusting method will be described. Adjustment of the coupling coefficient, that is, a learning method will be described. In the learning, actually obtains the output value of the neuron giving test pattern, the coupling coefficient from the relationship between the output value and the teacher signal (desired output value to be output by the _{^{neuron) w LS M (n, u}} , v) of Make corrections. In the learning of the present embodiment, the least squares method is used for the feature detection layer of the last layer, and the coupling coefficient is corrected for the feature detection layer of the intermediate layer using the backpropagation method. Since the details of the correction method of the coupling coefficient such as the least squares method and the error back propagation method are described in Non-Patent Document 1, the detailed description is omitted here.

  A large number of specific patterns to be detected and many patterns not to be detected are prepared as test patterns for learning. Each test pattern includes an image signal and a teacher signal as one set. When a specific pattern to be detected is presented, a teacher signal is given to neurons in an area where the specific pattern exists on the feature detection cell surface of the last layer so that the output becomes 1. Conversely, when a pattern that should not be detected is presented, a teacher signal is given to neurons in the area of the pattern so that the output becomes -1.

  As described above, according to the present embodiment, by using the dictionary data corresponding to the subject characteristics, it is possible to improve the accuracy of subject detection and to suppress the possibility of deteriorating the detection accuracy in a unique situation. it can.

In the above embodiment, by switching the dictionary, detection processing parameters are obtained by a machine learning, that has been described to switch the coupling coefficient _{^{w LS M (n, u,}} v). The present invention is not limited to this, and switching including the network configuration of the CNN may be performed by switching dictionary data. Changing the network configuration of the CNN means changing the number of feature detection layers, the receptive field size of each layer, the type of activation function, and the like.

(Other embodiments)
In addition, the present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read the program. It can also be realized by executing processing. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

101: camera body, 102: shooting lens, 201: system control unit, 203: operation unit, 204: subject detection unit, 210: storage unit

The image processing apparatus according to the present invention includes an analysis unit that selects any one of the plurality of dictionary data and analyzes the acquired image using the selected dictionary data. Includes at least a first dictionary data and a second dictionary data, wherein the analyzing means has a detection score of a subject using the first dictionary data lower than a threshold value, or the first dictionary data Even when the object cannot be detected using the data, the image is analyzed again using the first dictionary data, and the detection score of the object using the second dictionary data is smaller than a threshold. Low, or when the object cannot be detected using the second dictionary data, the image is analyzed again using dictionary data different from the second dictionary data. That.
Further, the image processing apparatus according to the present invention detects the target object by referring to dictionary data acquired by machine learning corresponding to the target object to be detected from the acquired image. Means for selecting any dictionary data from the plurality of dictionary data for the target subject; and a detection evaluation value when the subject is detected using the dictionary data selected by the selecting means is predetermined. Control means for controlling the detection means so as to detect the target subject using the selected dictionary data and dictionary data different from the selected dictionary data when the value is lower than the value. Wherein the plurality of dictionary data includes general-purpose dictionary data and a plurality of dedicated dictionary data, and each of the plurality of dedicated dictionary data is When the target subject is under the conditions corresponding to each of the plurality of dedicated dictionary data, the dictionary data has a higher probability of being able to detect the target subject than the general-purpose dictionary data, The general-purpose dictionary data is dictionary data capable of detecting the target subject under more conditions than each of the plurality of dedicated dictionary data.

Claims (12)

From the acquired image, corresponding to the target subject to be detected, by referring to the dictionary data acquired by machine learning, detecting means for detecting the target subject,
Selecting means for selecting any dictionary data from a plurality of dictionary data for the target subject,
When the detection evaluation value when the subject is detected using the dictionary data selected by the selection unit is lower than a predetermined value, the selected dictionary data and a dictionary different from the selected dictionary data Control means for controlling the detection means, so as to detect the target object using data,
With
The plurality of dictionary data includes general-purpose dictionary data and a plurality of dedicated dictionary data,
Each of the plurality of dedicated dictionary data, when the target object is under a condition corresponding to each of the plurality of dedicated dictionary data, detecting the target object than the general-purpose dictionary data. The general-purpose dictionary data is dictionary data capable of detecting the target subject under more conditions than each of the plurality of dedicated dictionary data. Image processing device.   The image processing apparatus according to claim 1, wherein the dictionary data is data that defines a detection processing parameter obtained by machine learning.   The image processing apparatus according to claim 1, wherein the selection unit selects one of the plurality of dedicated dictionary data.   When the detection evaluation value when the subject is detected using one of the plurality of dedicated dictionary data selected by the selection unit is lower than a predetermined value, the control unit selects the selected dedicated dictionary. The image processing apparatus according to claim 3, wherein the detection unit is controlled to detect the target subject using the dictionary data and the general-purpose dictionary data.   The image processing apparatus according to claim 4, wherein the control unit controls the detection unit such that the selected dedicated dictionary data and the general-purpose dictionary data are used alternately for each frame. .   When the detection evaluation value when the subject is detected using the general-purpose dictionary data is lower than a predetermined value, the control unit does not use any of the plurality of special-purpose dictionary data, and uses the general-purpose dictionary data. The image processing apparatus according to claim 4, wherein the detection unit is controlled so as to detect the target subject using the image processing.   7. The dictionary data according to claim 1, wherein the dedicated dictionary data is dictionary data divided by at least one element of a posture of a subject, the number of subjects, an overlap of subjects, presence / absence of a decoration for the subject, and a type. The image processing device according to claim 1.   When the detection evaluation value when the subject is detected by using the dictionary data selected by the selection unit is equal to or more than a predetermined value, the control unit uses the selected dictionary data for the object. The image processing apparatus according to claim 1, wherein the detection unit is controlled to detect a subject. An image processing apparatus according to claim 1,
Imaging means for imaging a subject image;
An imaging device comprising: From the acquired image, corresponding to the target subject to be detected, by referring to the dictionary data acquired by machine learning, a detection step of detecting the target subject,
A selecting step of selecting any dictionary data from a plurality of dictionary data for the target subject,
When the detection evaluation value when the subject is detected using the dictionary data selected in the selection step is lower than a predetermined value, the selected dictionary data and a different dictionary from the selected dictionary data A control step of controlling the detection step so as to detect the target object using data;
Has,
The plurality of dictionary data includes general-purpose dictionary data and a plurality of dedicated dictionary data,
Each of the plurality of dedicated dictionary data, when the target object is under a condition corresponding to each of the plurality of dedicated dictionary data, detecting the target object than the general-purpose dictionary data. The general-purpose dictionary data is dictionary data capable of detecting the target subject under more conditions than each of the plurality of dedicated dictionary data. Image processing method.   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images