JP7286330B2 - Image processing device and its control method, program, storage medium - Google Patents

Description

The present invention relates to a technique for determining a main subject area in an image processing device.

2. Description of the Related Art Conventionally, in an imaging device or an image processing device, a function is known in which the device identifies an image and automatically selects a main subject without any operation by the user. In this function, the developer prepares a group of assumed images in advance, sets the correct answer to be the main subject in each image, and adjusts the parameters of the determiner based on them. For example, Patent Literature 1 discloses a method of determining an object using a neural network using a training data group prepared in advance.

In addition to the pre-learned decider, a method of performing additional learning using information obtained from the decision area at the time of decision has also been proposed. For example, Patent Literature 2 discloses a method of determining an object using a learned classifier obtained by adding a pre-learned fixed classifier and information obtained from the decision area of the fixed classifier to dictionary data.

JP 2016-157219 A Japanese Unexamined Patent Application Publication No. 2010-170201

S. Haykin, "Neural Networks A Comprehensive Foundation 2nd Edition", Prentice Hall, pp. 156-255, July 1998

Here, images that are frequently shot by users are generally different for each user. Further, even for the same image, the correct answer to be the main subject differs depending on the user. Therefore, the function of automatically selecting the main subject should be adjusted according to each user's preference.

However, in the above-mentioned Patent Document 1, only learning is performed from a group of learning data prepared in advance, and in the above-mentioned Patent Document 2, information obtained from the decision region of the fixed classifier is added to the dictionary data. Only. Therefore, it cannot be said that any technique is adjusted according to the user's preference.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus capable of appropriately selecting a subject desired by a user when selecting a main subject from an image.

The image processing apparatus according to the present invention uses, as data to give a negative reward in learning for the determining means, an image in which the main subject has been determined using the determining means for determining the main subject from the acquired image. A selection means is provided for selecting an image based on user input, and the selection means selects the first main subject determined by the determination means when there is a user input indicating a photographing instruction. and, if the subject is different from the second main subject determined using the determining means before the user's input indicating the photographing instruction, the image in which the second main subject is determined is It is characterized in that it is sorted out as data that gives a negative reward .

According to the present invention, it is possible to provide an image processing apparatus capable of appropriately selecting a subject desired by a user when selecting a main subject from an image.

1 is a block diagram showing the configuration of an imaging device according to one embodiment of the present invention; FIG. 4 is a flow chart showing the flow of the overall operation of the imaging device of one embodiment. 4A and 4B are schematic diagrams showing examples of main subject determination results; FIG. The schematic diagram which shows the example of the whole structure of CNN. Schematic diagram showing an example of a partial configuration of a CNN. 4 is a flow chart showing a data set sorting procedure in one embodiment. A flowchart illustrating another data set filtering procedure in one embodiment. The figure which shows the system which consists of a smart phone and a server.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(Configuration of imaging device)
FIG. 1 is a block diagram showing the configuration of an imaging device 100 as an image processing device according to one embodiment of the present invention.

In FIG. 1, an imaging device 100 is a digital still camera, a video camera, or the like that can photograph a subject and record moving and still image data on various media such as tapes, solid-state memories, optical disks, and magnetic disks. However, the present invention is not limited to these, and can be applied to other devices having a photographing function, such as camera-equipped mobile phones and tablet terminals. Each unit in the imaging device 100 is connected via a bus 160 . Each unit is controlled by a CPU 151 (central processing unit).

The lens unit 101 comprises a fixed 1st group lens 102 , a zoom lens 111 , an aperture 103 , a fixed 3rd group lens 121 and a focus lens 131 . The diaphragm control circuit 105 drives the diaphragm 103 through the diaphragm motor 104 (AM) in accordance with a command from the CPU 151, thereby adjusting the aperture diameter of the diaphragm 103 and adjusting the amount of light during photographing. A zoom control circuit 113 changes the focal length by driving the zoom lens 111 via a zoom motor 112 (ZM). The focus control circuit 133 determines the drive amount for driving the focus motor 132 (FM) based on the amount of focus shift of the lens unit 101 . In addition, the focus control circuit 133 controls the focus adjustment state by driving the focus lens 131 via the focus motor 132 (FM). AF control is realized by movement control of the focus lens 131 by the focus control circuit 133 and the focus motor 132 . The focus lens 131 is a lens for focus adjustment, and although it is simply shown as a single lens in FIG. 1, it is usually composed of a plurality of lenses.

A subject image formed on the image sensor 141 through the lens unit 101 is converted into an electric signal by the image sensor 141 . The imaging element 141 is a photoelectric conversion element that converts a subject image (optical image) into an electrical signal. The image sensor 141 has m pixels in the horizontal direction and n pixels in the vertical direction. An image formed on the imaging element 141 and photoelectrically converted is arranged as an image signal (image data) by an imaging signal processing circuit 142 .

Image data output from the imaging signal processing circuit 142 is sent to an imaging control circuit 143 and temporarily stored in a RAM (random access memory) 154 . The image data accumulated in the RAM 154 is recorded in the image recording medium 157 after being compressed in the image compression/decompression circuit 153 . In parallel with this, the image data accumulated in the RAM 154 is sent to the image processing circuit 152 . The image processing circuit 152 processes an image signal, performs reduction/enlargement processing to an optimum size for image data, similarity calculation processing between image data, and the like. By appropriately sending the image data processed to the optimum size to the monitor display 150 and displaying it, preview image display and through image display can be performed. Also, the main subject determination result of the main subject determination circuit 162 can be superimposed on the image data. By using the RAM 154 as a ring buffer, it is possible to buffer a plurality of image data captured within a predetermined period and the determination result of the main subject determination circuit 162 corresponding to each image data. Similarly, image data used for learning of the main subject determination circuit 162 and main subject determination results corresponding to the image data can be buffered.

The operation switch 156 is an input interface including a touch panel, buttons, and the like, and various operations can be performed by selecting and operating various function icons displayed on the monitor display 150 . For example, while viewing the through image displayed on the monitor display 150, the user can manually specify the position of the main subject or cancel the already specified main subject.

The CPU 151 determines the accumulation time of the image sensor 141 and image signal processing from the image sensor 141 based on the user's instruction input from the operation switch 156 or the magnitude of the pixel signal of the image data temporarily stored in the RAM 154 . A gain setting value and the like for outputting a signal to the circuit 142 are determined. The imaging control circuit 143 receives instructions of the accumulation time and gain setting values from the CPU 151 and controls the imaging device 141 .

A main subject determination circuit 162 uses the image signal to determine the area where the main subject exists. Main subject determination processing in the main subject determination circuit 162 is realized by feature extraction processing by CNN (Convolutinal Neural Networks). The main subject determination circuit 162 is composed of a GPU (Graphic Processing Unit). A GPU is originally a processor for image processing, but since it has a plurality of sum-of-products arithmetic units and excels at matrix calculation, it is often used as a processor for performing processing for learning. Also in the deep learning process, a GPU is generally used, but a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like may also be used.

A focus control circuit 133 performs AF control for a specific subject area. A diaphragm control circuit 105 performs exposure control using the luminance value of a specific subject area. The image processing circuit 152 performs gamma correction, white balance processing, etc. based on the subject area. A monitor display 150 displays an image and a main subject determination result in a rectangle or the like. The battery 159 is appropriately managed by the power management circuit 158 to stably supply power to the entire imaging apparatus 100 .

The flash memory 155 stores control programs necessary for the operation of the imaging apparatus 100, parameters used for the operation of each unit, and the like. When the imaging apparatus 100 is activated by the user's operation (shifted from the power OFF state to the power ON state), the control program and parameters stored in the flash memory 155 are read into part of the RAM 154 . The CPU 151 controls the operation of the imaging device 100 according to the control program and constants loaded in the RAM 154 .

(Overall processing flow)
FIG. 2 is a flow chart showing the overall operation flow of the imaging apparatus 100 of this embodiment. The flowchart shown in FIG. 2 is repeated, for example, every one frame period of the live view while the imaging device is displaying the live view.

When the imaging apparatus 100 starts live view display, first, in S<b>201 , the imaging control circuit 143 supplies an input image obtained using the lens unit 101 and the imaging element 141 to each section of the imaging apparatus 100 .

In S202, the main subject determination circuit 162 determines the main subject for the input image. Details of the main subject determination process will be described later. Further, it is assumed that the main subject determination circuit 162 has previously undergone necessary learning, and the details of the learning process will be described later.

In S203, the CPU 151 determines whether or not the main subject determination circuit 162 has output the main subject determination result in S202. If it is output, the process proceeds to S204, and if it is not output, the process proceeds to S205.

In S204, the monitor display 150 displays the input image and superimposes the main subject determination result. In S205, monitor display 150 displays only the input image.

In S206, the CPU 151 buffers the input image live-view-displayed in S204, the main subject determination result, and the display time in the RAM 154 as a set of data sets. In S<b>207 , the CPU 151 selects an input image for use in positive learning or negative learning from the data set buffered in the RAM 154 . This processing will be described later.

In S208, the CPU 151 determines whether or not the user has given a shooting instruction. If there is a shooting instruction, the process proceeds to S209, and if there is no shooting instruction, the process proceeds to S211. Note that the shooting instruction is an instruction to start shooting a still image for recording or an instruction to start shooting a moving image for recording. The user can give a photographing instruction by fully pressing the release button or by operating the touch panel.

In S209, the CPU 151 performs still image or moving image shooting processing, and when the shooting ends, the process proceeds to S210.

In S<b>210 , the CPU 151 selects a data set to be used for positive learning or negative learning from the data sets buffered in the RAM 154 . This processing will be described later.

In S211, the CPU 151 determines whether or not the number of input images selected in S207 and S210 is equal to or greater than a threshold.

In S212, the main subject determination circuit 162 performs additional learning processing. Details of the additional learning process will be described later.

In S<b>213 , the CPU 151 determines whether or not there is an end instruction from the operation switch 156 . If there is an end instruction, the process is terminated, and if there is no end instruction, the process returns to S201 and repeats the series of processes.

(Example of main subject determination result)
FIG. 3 is a diagram showing an example of the main subject determination result displayed in S204 of FIG. FIG. 3A shows an example of the main subject determination result when additional learning is not performed in S207. FIG. 3B shows an example of the main subject determination result after additional learning is performed in S207 by the operation of a user who likes flowers and frequently shoots them. FIG. 3(c) shows an example of the main subject determination result after additional learning in S207 by the operation of a user who likes birds and frequently shoots them.

(Description of main subject determination circuit)
In this embodiment, the main subject determination circuit 162 is configured by CNN (Convolutinal Neural Networks). A basic configuration of the CNN will be described with reference to FIGS. 4 and 5. FIG.

FIG. 4 is a diagram showing the basic configuration of a CNN that determines a main subject from input two-dimensional image data and a position map. In the flow of processing, the left end is the input and the processing proceeds to the right. The CNN has a set of two layers called a feature detection layer (S layer) and a feature integration layer (C layer), which are hierarchically configured.

In CNN, first, the following features are detected in the S layer based on the features detected in the previous layer. In addition, the features detected in the S layer are integrated in the C layer, and the results of detection in that layer are sent to the next layer. Various information can be considered as information input to this CNN. For example, an RGB image, a pixel-by-pixel captured image signal (RAW image) before development processing, image depth information, an object detection score map by an object detector, a contrast map obtained from variance values in a local area of an image, etc. is mentioned.

The S layer consists of feature detection cell planes, and detects different features for each feature detection cell plane. Also, the C layer consists of a feature integration cell plane, and pools the detection results of the feature detection cell plane in the preceding stage. Hereinafter, the feature detection cell plane and the feature integration cell plane will be collectively referred to as feature planes when there is no particular need to distinguish them. In this embodiment, the output layer, which is the final stage layer, is composed only of the S layer without using the C layer.

FIG. 5 is a diagram for explaining feature detection processing in the feature detection cell plane and feature integration processing in the feature integration cell plane.

The feature detection cell surface is composed of a plurality of feature detection neurons, and the feature detection neurons are connected to the C layer of the pre-stage layer in a predetermined structure. The feature-integrating cell surface is composed of a plurality of feature-integrating neurons, and the feature-integrating neurons are connected to the S layer of the same hierarchy in a predetermined structure. In the M-th cell plane of the L-th layer S layer shown ⁱⁿ FIG _. In the cell plane, the output value of the feature integration neuron at position (ξ, ζ) is expressed as y _M ^LC (ξ, ζ). In that case, if the coupling coefficients of the neurons are w _M ^LS (n, u, v) and w _M ^LC (u, v), each output value can be expressed as follows.

…（１）

…（２）
式（１）のｆは活性化関数であり、ロジスティック関数や双曲正接関数などのシグモイド関数であれば何でもよく、例えばtanh関数で実現してよい。ｕ_M ^LS(ξ,ζ)は、Ｌ階層目Ｓ層のＭ番目細胞面における、位置(ξ,ζ)の特徴検出ニューロンの内部状態である。式（２）は活性化関数を用いず単純な線形和をとっている。式（２）のように活性化関数を用いない場合は、ニューロンの内部状態ｕ_M ^LC(ξ,ζ)と出力値ｙ_M ^LC(ξ,ζ)は等しい。また、式（１）のｙ_n ^(L-1C)(ξ+u,ζ+v)、式（３）のｙ_M ^LS(ξ+u,ζ+v)をそれぞれ特徴検出ニューロン、特徴統合ニューロンの結合先出力値と呼ぶ。

…(1)

…(2)
f in Equation (1) is an activation function, which may be any sigmoid function such as a logistic function or a hyperbolic tangent function, and may be realized by, for example, a tanh function. u _M ^LS (ξ, ζ) is the internal state of the feature detection neuron at position (ξ, ζ) on the M-th cell plane of the L-th layer S layer. Equation (2) takes a simple linear sum without using an activation function. When no activation function is used as in Equation (2), the neuron's internal state u _M ^LC (ξ, ζ) and output value y _M ^LC (ξ, ζ) are equal. Also, y _n ^(L−1C) (ξ+u, ζ+v) in equation (1) and y _M ^LS (ξ+u, ζ+v) in equation (3) are represented by feature detection neurons and feature integration neurons, respectively. is called the destination output value of

Here, ξ, ζ, u, v, and n in equations (1) and (2) will be explained. The position (ξ, ζ) corresponds to the position coordinates in the input image. For example, if y _M ^LS (ξ, ζ) is a high output value, then L This means that there is a high probability that the features detected in the M-th cell surface of the S-th layer are present. Also, n in the formula (2) means the n-th cell surface of the C layer of the L-1 layer, and is called the integrated feature number. Basically, sum-of-products operations are performed for all cell planes present in the L-1 layer C layer. (u, v) are the relative position coordinates of the coupling coefficient, and the sum-of-products operation is performed in a finite range (u, v) according to the size of the feature to be detected. Such a finite range of (u, v) is called a receptive field. The size of the receptive field is hereinafter referred to as the size of the receptive field, and is represented by the number of horizontal pixels×the number of vertical pixels in the combined range.

Also, in equation (1), y _n ^(L-1C) (ξ+u, ζ+v) is the input image y ^in_image (ξ+u, ζ+v) Alternatively, it becomes the input position map y ^in_posi_map (ξ+u, ζ+v). Incidentally, the distribution of neurons and pixels is discrete, and the connection destination feature number is also discrete, so ξ, ζ, u, v, and n are not continuous variables but discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have a finite range.

w _M ^LS (n, u, v) in equation (1) is a coupling coefficient distribution for detecting a predetermined feature, and by adjusting this to an appropriate value, the predetermined feature can be detected. becomes possible. This adjustment of the coupling coefficient distribution is learning, and in constructing the CNN, various test patterns are presented, and the coupling coefficients are repeatedly adjusted so that y _M ^LS (ξ, ζ) becomes an appropriate output value. Coupling coefficients are adjusted by correcting them.

Next, w _M ^LC (u,v) in Equation (2) uses a two-dimensional Gaussian function and can be expressed as in Equation (3) below.

…（３）
ここでも、（ｕ，ｖ）は有限の範囲としているので、特徴検出ニューロンの説明と同様に、有限の範囲を受容野といい、範囲の大きさを受容野サイズと呼ぶ。この受容野サイズは、ここではＬ階層目Ｓ層のＭ番目特徴のサイズに応じて適当な値に設定すればよい。式（３）中の、σは特徴サイズ因子であり、受容野サイズに応じて適当な定数に設定しておけばよい。具体的には、受容野の一番外側の値がほぼ０とみなせるような値になるように設定するのがよい。

…(3)
Since (u, v) has a finite range here as well, the finite range is called the receptive field, and the size of the range is called the receptive field size, as in the description of the feature detection neuron. This receptive field size may be set to an appropriate value according to the size of the M-th feature of the L-th layer S layer. In Equation (3), σ is a feature size factor, which may be set to an appropriate constant according to the size of the receptive field. Specifically, it is preferable to set the value so that the outermost value of the receptive field can be regarded as approximately zero.

The configuration of the CNN in this embodiment is to determine the main subject in the S layer, which is the final layer, by performing the above-described calculations in each layer.

(Flow of additional learning process)
FIG. 6 is a flowchart showing the flow of image selection processing in S207 of FIG. Note that hereinafter, learning a predetermined area of an input image as a test pattern for determining a main subject is referred to as positive learning. Also, learning a predetermined area as a test pattern for which the main subject should not be determined is referred to as negative learning. Details of each learning will be described later.

First, in S<b>601 , the CPU 151 determines whether or not there is a manual instruction from the operation switch 156 . If there is a manual instruction, the process proceeds to S602, and if there is no manual instruction, the process proceeds to S605. Here, the manual instruction refers to an operation (selection instruction) in which the user manually operates the operation switch 156 to specify the main subject area for the image. The manual instruction in S601 means that the main subject determined in S202 of FIG. 2 is different from the main subject desired by the user, so the process proceeds to S602 and subsequent steps.

In S602, the subject manually designated by the user in S601 is considered to be the correct main subject. Therefore, the CPU 151 selects the input image acquired in S201 of FIG. 2 and the main subject area manually instructed in S601 as data sets for positive learning.

In S603, the CPU 151 determines whether the main subject area determined in S202 of FIG. 2 and the main subject area manually specified in S601 are different subjects. For example, if the main subject area determined in S202 in FIG. 2 and the main subject area manually designated in S601 are separated by a predetermined distance or more, it is determined that the two areas are different subjects. If the distance is equal to or longer than the predetermined distance, the process proceeds to S604, and if not, the flow chart ends. The predetermined interval here may be anything as long as it is sufficiently large, but may be determined, for example, as a ratio to the angle of view of the input image. More specifically, when the input image is an image of 640×480 pixels, for example, the predetermined interval may be 64 pixels, which is 10% of the horizontal size of 640 pixels. If it is clear from the result of face detection, human body detection, or object detection that the main subject area determined in S202 and the main subject area manually designated in S601 are different subjects, It is not necessary to determine the distance between

In S604, the CPU 151 selects the input image acquired in S201 of FIG. 2 and the main subject area determined in S202 as data sets for negative learning. Then, this flowchart ends.

In S<b>605 , the CPU 151 determines whether or not there is a cancel instruction from the operation switch 156 . If there is a cancel instruction, the process advances to S606, and if there is no cancel instruction, this flowchart ends. Here, the cancel instruction refers to an operation to delete the main subject determination result output in S204 of FIG. 2 and place the imaging apparatus 100 in a state in which the main subject has not been determined. For example, if a function is provided that allows the user to instruct to redo determination of the main subject area, that instruction corresponds to a cancel instruction. Alternatively, when the user redoes the half-pressing of the release button, which is an operation for determining the subject to be focused, this redoing of the half-pressing corresponds to a cancel instruction. The fact that there is a cancel instruction in S605 means that the main subject determined in S202 of FIG. 2 is different from the main subject desired by the user, so the process proceeds to S606.

In S606, the CPU 151 selects the input image acquired in S201 of FIG. 2 and the main subject area determined in S202 as data sets for negative learning. Then, this flowchart ends.

FIG. 7 is a flow chart showing the flow of image selection processing in S210 of FIG.

In S<b>701 , the CPU 151 determines whether or not the same subject as when the shooting instruction was given is the data set determined as the main subject for each data set group buffered in the RAM 154 . If the data set has the same subject determined as the main subject when the shooting instruction was given, the process proceeds to S702; otherwise, the process proceeds to S703. If a subject other than the subject that was determined as the main subject at the time of the shooting instruction was determined as the main subject prior to the shooting instruction, the previous determination of the main subject is an error. Probability is high. Therefore, a data set in which another subject has been determined as the main subject is effective as a data set for negative learning.

In S702, the CPU 151 selects a data set that satisfies a positive learning condition from the data set group in which the same subject as when the shooting instruction was given is determined as the main subject. The positive learning condition here is that the difference between the display time in the data set and the execution time of this step is less than a predetermined time. Note that the predetermined time here is based on the time lag from when the user confirms the determination result of the main subject output to the monitor display 150 in S204 of FIG. may be determined by More specifically, for example, the predetermined time is set to 2 seconds. Also, when a plurality of data sets satisfy the conditions at once, the degree of similarity between the input images included in each data set is calculated. Exclude sets from screening. In the exclusion process, data with a large difference between the display time included in the data set and the execution time of this step may be preferentially excluded. A SAD value, a histogram difference, or the like can be used as the degree of similarity here.

In S703, the CPU 151 selects a data set that satisfies a negative learning condition from a data set group in which a subject different from that at the time of the photographing instruction is determined as the main subject. Also, if a plurality of data sets satisfy the condition at once, exclusion processing is performed in the same manner as in S608. When at least one of S702 and S703 is performed, the CPU 151 ends this flowchart.

Then, in S212 of FIG. 2, the main subject determination circuit 162 performs reinforcement learning using the data sets selected in S210 and S212.

In S212, the CPU 151 deletes all data sets buffered in the RAM 154 in S206 of FIG.

As described above, by performing additional learning based on the user's operation, it is possible to adjust the parameters of the main subject determination circuit 162 according to the user's preference. In addition, since data sets with high similarity are excluded during data set selection, over-learning for specific scenes can be suppressed.

(learning method)
Next, a specific learning method will be described. In this embodiment, the coupling coefficient is adjusted by supervised learning. In supervised learning, a test pattern is given to actually obtain the output value of a neuron, and the coupling coefficient w _M ^LS (n, u, v) should be corrected. In the learning of this embodiment, the last feature detection layer uses the least squares method, and the intermediate feature detection layer uses the error backpropagation method to correct the coupling coefficients (the least squares method, error backpropagation See Non-Patent Document 1 for details of the method of modifying the coupling coefficient, such as the modulus.

In this embodiment, when learning in advance, a large number of specific patterns to be detected and patterns not to be detected are prepared as test patterns for learning. Select a test pattern from the buffer. Each test pattern is a set of images and teacher signals. The data set selected for positive learning is used as the specific pattern to be detected, and the data set selected for negative learning is used as the pattern not to be detected.

When the tanh function is used as the activation function, when a specific pattern to be detected is presented, the neuron in the area where the specific pattern exists in the feature detection cell surface of the final layer is supervised so that the output becomes 1. give a signal. That is, give a positive reward. Conversely, when a pattern that should not be detected is presented, a teacher signal is given to the neuron in the region of that pattern so that the output becomes -1. i.e. give a negative reward.

A CNN for determining a main subject from a two-dimensional image is constructed by the above method. In actual determination, computation is performed using the coupling coefficients w _M ^LS (n, u, v) constructed by learning. It is determined that the main subject exists in

In the present embodiment, the imaging apparatus 100 has the main subject determination circuit 162, and the configuration for selecting data sets for learning and performing reinforcement learning has been described as an example, but the present invention is limited to this. isn't it.

For example, the present invention can be applied to a system in which a smartphone or tablet terminal having a camera function is connected to an external device such as a server or an edge computer by wireless communication.

FIG. 8 shows a system consisting of a smart phone and a server. The smartphone 801 transmits the data set buffered in the internal RAM to the server 802 , and the server 802 performs reinforcement learning to obtain the coupling coefficient of the main subject determination circuit inside the smartphone 801 . Then, the server 802 transmits the obtained coupling coefficient to the smartphone 801, and the smartphone 801 sets the received coupling coefficient in the main subject determination circuit. At this time, either the smart phone 801 or the server 802 may perform the process of selecting data sets for positive learning and negative learning from the data sets buffered in the RAM of the smart phone 801 . If the smartphone 801 sorts out the datasets, the sorted datasets may be transmitted from the smartphone 801 to the server 802 . If the server 802 sorts out the datasets, the smartphone 801 transmits to the server 802 all the datasets buffered in the RAM and the user operation information stored in association with the datasets. . It is assumed that the images included in the data set and the information of the user's operation are associated with each other on the time axis. With such a configuration, the smartphone 801 transmits data sets and user operation information to the server 802, so that the server 802 can select data sets.

Alternatively, the smartphone 801 may not include the main subject determination circuit, and the server 802 may include the main subject determination circuit. The smartphone 801 transmits the input image to the server in real time, and the server 802 determines the main subject and transmits the result to the smartphone 801 . In this case, the server 802 performs reinforcement learning to update the coupling coefficients of the main subject determination circuit of the server. In this case as well, the process of selecting data sets for positive learning and negative learning from the data sets buffered in the RAM of the smart phone 801 may be performed by either the smart phone 801 or the server 802 .

Furthermore, as for the data set for positive learning, an input image obtained by shooting may be used instead of the data set buffered in the RAM 154 before the shooting instruction. This is because there is a high possibility that the main subject has been correctly selected at the time of photographing. In this case, the data set for positive learning is the data set containing the images recorded by shooting, whereas the data set for negative learning is the data set buffered before the shooting instruction. Become.

As described above, according to the present embodiment, by repeating normal camera operations, it is possible to determine the main subject according to the user's preference. Furthermore, by determining the relationship between the main subject determination result before the shooting instruction and the user's operation at that time, it becomes possible to identify that the main subject determination result is unintended by the user. It is possible to select data for negative learning.

(Other embodiments)
In addition, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads the program. It can also be realized by executing processing. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: imaging device, 101: lens unit, 141: imaging element, 142: imaging signal processing circuit, 143: imaging control circuit, 151: CPU, 152: image processing circuit, 162: main subject determination circuit

Claims (16)

An image to be used as data for giving a negative reward in learning for the determining means is selected from among the images for which the main subject has been determined using the determining means for determining the main subject from the acquired images, based on the user's input. Having sorting means for sorting,
The selecting means uses the first main subject determined by the determining means at the time of the user's input indicating the photographing instruction, and the determining means before the user's input indicating the photographing instruction. and selecting an image in which the second main subject is determined as the data for giving the negative reward when the subject is different from the second main subject that has been determined by the method. . The selection means selects an image to be used as data that gives a positive reward in learning for the determination means from among the images for which the main subject has been determined using the determination means for determining the main subject from the acquired images, 2. The image processing apparatus according to claim 1, wherein the selection is made based on the user's input. An image to be used as data for giving a negative reward in learning for the determining means is selected from among the images for which the main subject has been determined using the determining means for determining the main subject from the acquired images, based on the user's input. Having sorting means for sorting,
The selection means selects an image to be used as data that gives a positive reward in learning for the determination means from among the images for which the main subject has been determined using the determination means for determining the main subject from the acquired images, filtering based on said user input;
The selecting means uses the first main subject determined by the determining means at the time of the user's input indicating the photographing instruction, and the determining means before the user's input indicating the photographing instruction. selecting the image in which the second main subject is determined as the data for giving the negative reward, and selecting the image in which the second main subject is determined as the data that gives the negative reward, and 2. An image processing apparatus, wherein, when the second main subject is the same subject, an image in which the second main subject is determined is selected as data for giving the positive reward. When the first main subject specified based on the user's input and the second main subject determined using the determining means are different subjects, the selecting means selects the second main subject. 4. The image processing apparatus according to any one of claims 1 to 3, wherein an image for which is determined is selected as the data for giving the negative reward. When the first main subject is designated based on the user's input, the selection means selects an image in which the first main subject is determined as the data for giving the positive reward. 3. The image processing apparatus according to claim 2. When the user's input for determining the subject to be focused is redone, the selection means converts the image in which the main subject has been determined using the determination means before the redo to the negative image. 6. The image processing apparatus according to any one of claims 1 to 5, wherein the data is selected as reward data. Further comprising calculating means for calculating the similarity of the plurality of images,
The selection means selects only an image selected from a plurality of images having a degree of similarity calculated by the calculation means equal to or higher than a predetermined value as data for giving the negative reward. 7. The image processing apparatus according to any one of claims 1 to 6 . 8. An image processing apparatus according to claim 1, further comprising said determining means. Having learning means for learning for the determining means,
9. The image processing apparatus according to any one of claims 1 to 8 , wherein the learning means performs learning using the data selected by the selecting means and giving the negative reward. The determining means uses a learning result obtained by wireless communication from a learning means of an external device that performs learning for the determining means,
9. The image processing apparatus according to claim 8 , wherein the learning means performs learning using the data selected by the selecting means and giving the negative reward. An external device capable of wireless communication with the image processing device has the determining means and a learning means for performing learning for the determining means,
8. The image processing apparatus according to any one of claims 1 to 7 , wherein the learning means performs learning using the data selected by the selecting means and giving the negative reward. 12. The image processing apparatus according to any one of claims 1 to 11 , further comprising imaging means for imaging an image in which the main subject has been determined. An image to be used as data for giving a negative reward in learning for the determining means is selected from among the images for which the main subject has been determined using the determining means for determining the main subject from the acquired images, based on the user's input. Having a sorting process for sorting,
In the selecting step, the first main subject determined by the determining means when the user input indicating the photographing instruction is received, and the determining means is used before the user input indicating the photographing instruction. and selecting an image in which the second main subject is determined as the data for giving the negative reward when the subject is different from the second main subject that has been determined by the method. control method. An image to be used as data for giving a negative reward in learning for the determining means is selected from among the images for which the main subject has been determined using the determining means for determining the main subject from the acquired images, based on the user's input. Having a sorting process for sorting,
In the selecting step, an image to be used as data for giving a positive reward in learning for the determining means, from among the images for which the main subject has been determined using the determining means for determining the main subject from the acquired images, filtering based on said user input;
In the selecting step, the first main subject determined by the determining means when the user input indicating the photographing instruction is received, and the determining means is used before the user input indicating the photographing instruction. selecting the image in which the second main subject is determined as the data for giving the negative reward, and selecting the image in which the second main subject is determined as the data that gives the negative reward, and and selecting an image in which the second main subject is determined as the data giving the positive reward when the second main subject is the same subject. . A program for causing a computer to execute the steps of the control method according to claim 13 or 14. A computer-readable storage medium storing a program for causing a computer to execute the steps of the control method according to claim 13 or 14.

Images