JP7054603B2 - Judgment device, judgment method, and judgment program - Google Patents

Description

The present invention relates to a determination device, a determination method, and a determination program.

Conventionally, a technique for extracting image features by a neural network has been provided. For example, a convolutional neural network provides a technique for generating an image saliency map. Further, a neural network provides a technique for identifying a predetermined object included in an image.

Min Lin, Qiang Chen, Shuicheng Yan, "Network In Network", arXiv preprint arXiv: 1312.4400 (2013) & ICLR (International Conference on Learning Representations)-2014 Jianming Zhang, Shugao Ma, Mehrnoosh Sameki, Stan Sclaroff, Margrit Betke, Zhe Lin, Xiaohui Shen, Brian Price, Radomir Mech "Salient Object Subitizing", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4045 -4054

However, in the above-mentioned conventional technique, the number of objects included in the image is not always properly determined. For example, it is not always possible to appropriately determine the number of objects included in an image simply by identifying a predetermined object included in the image.

The present application has been made in view of the above, and an object thereof is to provide a determination device, a determination method, and a determination program for appropriately determining the number of objects included in an image by using information in a neural network. ..

The determination device according to the present application is a plurality of feature information based on an input image input to a neural network that identifies the number of objects in an image, and a plurality of feature information corresponding to each number identified by the neural network. It is characterized by including an acquisition unit to be acquired and a determination unit for determining the number of objects included in the input image based on the plurality of feature information acquired by the acquisition unit.

According to one aspect of the embodiment, there is an effect that the number of objects included in the image can be appropriately determined by using the information in the neural network.

FIG. 1 is a diagram showing an example of a determination process according to an embodiment. FIG. 2 is a diagram showing a configuration example of the determination device according to the embodiment. FIG. 3 is a diagram showing an example of a learning information storage unit according to an embodiment. FIG. 4 is a diagram showing an example of an image information storage unit according to an embodiment. FIG. 5 is a flowchart showing an example of determination of the number of objects according to the embodiment. FIG. 6 is a diagram showing an example of generating a processed image according to the embodiment. FIG. 7 is a flowchart showing an example of generating a processed image according to the embodiment. FIG. 8 is a diagram showing an example of a learning process according to an embodiment. FIG. 9 is a diagram showing an example of the learning process according to the embodiment. FIG. 10 is a flowchart showing an example of the learning process according to the embodiment. FIG. 11 is a diagram showing an example of the determination process according to the embodiment. FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the function of the determination device.

Hereinafter, a determination device according to the present application, a determination method, and an embodiment for implementing the determination program (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the determination device, determination method, and determination program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

(Embodiment)
[1. Determination process〕
An example of the determination process according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of a determination process according to an embodiment. The determination device 100 shown in FIG. 1 determines the number of objects included in the image by using the feature information corresponding to each number acquired from the learning device LE that recognizes the number of objects included in the image. Specifically, the determination device 100 calculates a score for each feature information corresponding to each number acquired from the learning device LE, and determines the number of objects included in the image based on the score. The learning device LE shown in FIG. 1 identifies whether the number of objects included in the image is classified into five types (classes) of 0, 1, 2, 3, or 4 or more. .. In this case, the learner LE classifies the images into five classes (numbers).

In the example shown in FIG. 1, for the learner LE that recognizes the number of objects included in the image, various conventional techniques such as detection of the number by NIN (Network In Network) as shown in Non-Patent Document 1 are appropriately applied. It may be a learning device generated by using. For example, the learner LE in FIG. 1 may be a learner in which the fully connected layer of the convolutional neural network is replaced with the convolutional layer. In the following, the convolutional neural network may be referred to as CNN. For example, the learner LE in FIG. 1 may be a CNN that does not include a fully connected layer, that is, an FCN (Fully Convolutional Neural network). For example, in a normal CNN, the spatial information of the image is lost by passing through the fully connected layer. On the other hand, in FCN, by replacing the fully connected layer with a convolutional layer, it is possible to perform estimation while maintaining spatial information (also referred to as “receptive field”). The learning device LE in FIG. 1 may be any learning device as long as the feature information corresponding to each class (number) can be acquired. For example, the learning device LE in FIG. 1 may be one in which NIN is trained to identify the number of objects.

Further, the learning device LE in FIG. 1 identifies the number of human faces included in the image. For example, the learner LE in FIG. 1 classifies an image into a plurality of classes (numbers) according to the number of human faces included in the image. For example, the learner LE in FIG. 1 classifies images into classes corresponding to each number according to the number of human faces included in the image. Further, in the example shown in FIG. 1, a case where the target for determining the number is a human face is shown, but the target is not limited to the human face, but other organisms such as dogs and cats, plants, cars, and the like. It may be various objects or the like. Further, the object referred to here may include various objects as long as it can be identified, and may include various phenomena such as fire and sea waves.

As described above, in FIG. 1, the determination device 100 uses a learning device LE, which is a discriminator (model) that outputs information regarding the number of human faces included in the image. In the example of FIG. 1, the determination device 100 uses a learning device LE that has been generated by a predetermined learning process described later. The determination device 100 may use any learning device as long as it can determine the number of predetermined objects. Further, when the learning device LE is generated (learning), a predetermined loss function, correct answer information, and the like are used, but the details will be described later.

From here, with reference to FIG. 1, the determination process of the number of objects included in the image by the determination device 100 will be described. As shown in FIG. 1, the image IM 11 is input to the determination device 100 (step S10). For example, the determination device 100 acquires an image IM 11 in which five people, that is, five faces, are captured as targets. The determination device 100 that has acquired the image IM 11 inputs the image IM 11 to a predetermined learning device (step S11). For example, in FIG. 1, the determination device 100 inputs the image IM 11 to the learner LE, which is a discriminator (model) that identifies the number of objects included in the image.

For example, the learner LE to which the image IM 11 is input performs a process of identifying the number of objects included in the image IM 11. For example, the learner LE generates feature information corresponding to each class (number) in the process of identifying the number of objects. In the example of FIG. 1, feature information corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more is generated by the learner LE.

Therefore, the determination device 100 acquires each feature information generated in the process of identifying the number of human faces included in the image by the learning device LE. In the example of FIG. 1, the determination device 100 acquires the feature information FM10 corresponding to the class in which the number of human faces is 0 from the learning device LE (step S12-0).

For example, the feature information FM10 indicates the feature amount of each pixel in the image IM11. The feature amount referred to here is, for example, a numerical value indicating the feature amount. Specifically, the positions of the points (pixels) constituting the feature information FM10 correspond to the overlapping positions in the image IM11 when superimposed on the image IM11, and the feature information FM10 corresponds to the characteristics of the corresponding pixels in the image IM11. Indicates the amount. In the feature information FM10 in FIG. 1, the region showing the feature is shown in a dark color mode. That is, in the feature information FM10, the larger the feature amount, the darker the color is displayed. Specifically, in the feature information FM10 in FIG. 1, the region where the human face is located is shown in the image IM11 in a dark mode. The same applies to other feature information.

Further, in the example of FIG. 1, the determination device 100 acquires the feature information FM11 corresponding to the class in which the number of human faces is one from the learning device LE (step S12-1). Further, in the example of FIG. 1, the determination device 100 acquires the feature information FM12 corresponding to the class in which the number of human faces is two from the learning device LE (step S12-2). Further, in the example of FIG. 1, the determination device 100 acquires the feature information FM13 corresponding to the class in which the number of human faces is three from the learning device LE (step S12-3). Further, in the example of FIG. 1, the determination device 100 acquires the feature information FM14 corresponding to the class in which the number of human faces is 4 or more from the learning device LE (step S12-4). In this way, the determination device 100 acquires the feature information FM10 to FM14 corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more from the learner LE. .. In the above example, for the sake of simplicity, an example of acquiring five feature information from one learning device LE is shown, but the number is 0, 1, 2, 3, or 4 or more. Feature information corresponding to each number may be acquired from five learning devices that identify each person's face. Hereinafter, when steps S12-0 to S12-4 are described without distinction, they may be described as step S12.

After that, the determination device 100 calculates a score for each of the feature information FM10 to FM14 acquired in step S12. For example, the determination device 100 may calculate the average of the feature amounts in each of the feature information FM10 to FM14 as a score. Further, the determination device 100 may calculate as a score a value obtained by multiplying the average of the feature amounts in each of the feature information FM10 to FM14 by a predetermined coefficient. Further, the determination device 100 may calculate the total of the feature amounts in each of the feature information FM10 to FM14 as a score. Further, the determination device 100 may calculate as a score a value obtained by multiplying the total feature amount in each of the feature information FM10 to FM14 by a predetermined coefficient.

For example, the determination device 100 calculates the score of the feature information FM10 based on the feature amount in the feature information FM10 (step S13-0). In the example of FIG. 1, the determination device 100 calculates the score corresponding to the number of human faces "0" as "0.12" as shown in the score information SC10. Further, for example, the determination device 100 calculates the score of the feature information FM11 based on the feature amount in the feature information FM11 (step S13-1). In the example of FIG. 1, the determination device 100 calculates the score corresponding to the number of human faces "1" as "0.01" as shown in the score information SC11. Further, for example, the determination device 100 calculates the score of the feature information FM12 based on the feature amount in the feature information FM12 (step S13-2). In the example of FIG. 1, the determination device 100 calculates the score corresponding to the number of human faces "2" as "0.03" as shown in the score information SC12. Further, for example, the determination device 100 calculates the score of the feature information FM13 based on the feature amount in the feature information FM13 (step S13-3). In the example of FIG. 1, the determination device 100 calculates the score corresponding to the number of human faces "3" as "0.06" as shown in the score information SC13. Further, for example, the determination device 100 calculates the score of the feature information FM14 based on the feature amount in the feature information FM14 (step S13-4). In the example of FIG. 1, the determination device 100 calculates the score corresponding to the number of human faces "4 or more" as "0.75" as shown in the score information SC14. Hereinafter, when steps S13-0 to S13-4 are described without distinction, they may be described as step S13.

After that, the determination device 100 determines the number of human faces included in the image IM 11 (step S14). In the example of FIG. 1, the determination device 100 determines the number of human faces included in the image IM 11 based on the scores of the feature information FM10 to FM14 acquired in step S13. For example, the determination device 100 includes the class (number) corresponding to the feature information FM14, which is the maximum score among the scores of the feature information FM10 to FM14 acquired in step S13, as the number of human faces included in the image IM11. judge. Specifically, the determination device 100 determines that the number of human faces included in the image IM 11 is 4 or more, as shown in the number determination information AN11.

Further, the determination device 100 determines the position of the human face included in the image IM 11 (step S15). In the example of FIG. 1, the determination device 100 determines the position of the human face included in the image IM 11 based on the feature information FM 14 which is the maximum score. For example, the determination device 100 determines that the human face is included in the region where the feature amount equal to or higher than a predetermined threshold value is located. In the example of FIG. 1, the determination device 100 determines in the image IM11 that the region corresponding to the region AR11 of the feature information FM14 includes a human face. Specifically, as shown in the position determination information AP11, the determination device 100 determines that the position of the human face included in the image IM11 is the upper part of the image IM11. When the learning device LE is FCN, the determination device 100 may estimate the position of the human face without using the feature information. Further, the processing of the image based on the position of the determined human face will be described later.

As described above, the determination device 100 appropriately determines the number of objects included in the image by using the information in the neural network (FCN in FIG. 1). In FIG. 1, the determination device 100 acquires feature information FM10 to FM14 corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more from the learner LE. .. Then, the determination device 100 determines the number of human faces included in the image IM 11 based on the scores of the acquired feature information FM10 to FM14. As a result, the determination device 100 can appropriately determine the number of objects included in the image. Then, the determination device 100 determines the position of the human face included in the image IM 11 based on the feature information FM 14 corresponding to the number of determined human faces. As a result, the determination device 100 can appropriately determine the position of the target included in the image.

In the above example, the learning device LE shows a case where the number of objects included in the image is classified into one of five types (classes) of 0, 1, 2, 3, or 4 or more. However, the classes classified by the learner LE are not limited to the above, and may be various classes depending on the purpose. For example, the class classified by the learner LE may be 11 classes for identifying whether the number of objects included in the image is 0 to 9 each, or 10 or more. Further, for example, the class classified by the learner LE may be two classes for identifying whether the number is less than 5, or 5 or more.

[2. Judgment device configuration]
Next, the configuration of the determination device 100 according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the determination device 100 according to the embodiment. As shown in FIG. 2, the determination device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The determination device 100 has an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from the administrator of the determination device 100, and a display unit (for example, a liquid crystal display, etc.) for displaying various information. You may.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 110 is connected to the network by wire or wirelessly, and transmits / receives information to / from, for example, a terminal device.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. As shown in FIG. 2, the storage unit 120 according to the embodiment has a learning information storage unit 121 and an image information storage unit 122.

(Learning information storage unit 121)
The learning information storage unit 121 according to the embodiment stores various information related to learning. For example, in FIG. 3, the learning information storage unit 121 stores learning information (model) related to the learning device LE generated by a predetermined learning process. FIG. 3 shows an example of the learning information storage unit 121 according to the embodiment. The learning information storage unit 121 shown in FIG. 3 stores a “weight ( _wij )”.

For example, in the example shown in FIG. 3, it is shown that the “weight (w ₁₁ )” is “0.2” and the “weight (w ₁₂ )” is “−0.3”. Further, in the example shown in FIG. 3, it is shown that the “weight (w ₂₁ )” is “0.5” and the “weight (w ₂₂ )” is “1.3”.

The "weight ( _wij )" may be, for example, a synaptic connection coefficient from the neuron y _i to the neuron x _j in the learner LE. Further, the learning information storage unit 121 is not limited to the above, and may store various information depending on the purpose.

(Image information storage unit 122)
The image information storage unit 122 according to the embodiment stores various information related to the image. FIG. 4 shows an example of the image information storage unit 122 according to the embodiment. The image information storage unit 122 shown in FIG. 4 has items such as an “image ID” and an “image”.

The "image ID" indicates identification information for identifying an image. "Image" indicates image information. Specifically, the "image" indicates an image input to the learner LE. In FIG. 4, an image identified by an image ID is shown for explanation, but as the “image”, a file path name or the like indicating a storage location of the image may be stored.

The image information storage unit 122 is not limited to the above, and may store various information depending on the purpose. For example, the image information storage unit 122 may store information regarding the date and time when the image was generated. Further, for example, the image information storage unit 122 may store information about an object included in the image. In FIG. 4, information indicating that the image identified by the image ID “IM11” includes the faces of five people may be stored. Further, for example, the image information storage unit 122 may store the acquired original image.

(Control unit 130)
Returning to the description of FIG. 2, the control unit 130 is a controller, and is stored in a storage device inside the determination device 100 by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). It is realized by executing various programs (corresponding to an example of a determination program) using the RAM as a work area. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 2, the control unit 130 includes an acquisition unit 131, a learning unit 132, a calculation unit 133, a determination unit 134, a generation unit 135, and a provision unit 136, and information described below. Realize or execute the function or action of processing. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 2, and may be any other configuration as long as it is configured to perform information processing described later.

(Acquisition unit 131)
The acquisition unit 131 acquires various types of information. The acquisition unit 131 acquires an image. For example, the acquisition unit 131 acquires an image from an external information processing device. In FIG. 1, the acquisition unit 131 acquires the image IM 11 from an external information processing device. Further, the acquisition unit 131 may acquire an image (for example, an image IM 11) from the image information storage unit 122.

Further, the acquisition unit 131 acquires a plurality of feature information based on the input image input to the neural network that identifies the number of objects in the image, and acquires a plurality of feature information corresponding to each number identified by the neural network. do. For example, the acquisition unit 131 acquires a plurality of feature information based on the input image input to the neural network that identifies the number of human faces in the image. Further, for example, the acquisition unit 131 acquires a plurality of feature information corresponding to each number identified by the neural network that performs the convolution process and the pooling process. Further, for example, the acquisition unit 131 acquires a plurality of feature information corresponding to each number identified by the neural network that does not include the fully connected layer. Further, for example, the acquisition unit 131 acquires a plurality of feature information corresponding to each number identified by the FCN.

In the example of FIG. 1, the acquisition unit 131 acquires a plurality of feature information corresponding to each number identified by the learner LE. For example, the acquisition unit 131 acquires the feature information FM10 to FM14 corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more from the learning device LE. Further, for example, the acquisition unit 131 may acquire the feature information FM10 to FM14 from an external information processing device.

(Learning Department 132)
The learning unit 132 learns various information. Further, the learning unit 132 generates various information by learning. For example, the learning unit 132 learns a learning device (model). In other words, the learning unit 132 generates a learning device (model) by performing learning. For example, the learning unit 132 learns the learning device LE. For example, the learning unit 132 learns the learning device by combining an image and the number of objects included in the image. Further, the learning unit 132 learns the learning device so as to minimize a predetermined evaluation function. The details of the learning process performed by the learning unit 132 will be described later. The determination device 100 does not have to have the learning unit 132 when acquiring a plurality of feature information corresponding to each number from an external information processing device.

(Calculation unit 133)
The calculation unit 133 calculates various information. For example, the calculation unit 133 calculates the score based on a plurality of feature information. For example, the calculation unit 133 calculates the score corresponding to each number based on the feature information corresponding to each number (class).

In the example of FIG. 1, the calculation unit 133 calculates a score for each of the feature information FM10 to FM14. For example, the calculation unit 133 calculates the average of the feature amounts in each of the feature information FM10 to FM14 as a score. For example, the calculation unit 133 calculates the score corresponding to the number of human faces "0" as "0.12" based on the feature amount in the feature information FM10. Further, for example, the calculation unit 133 calculates the score corresponding to the number of human faces "1" as "0.01" based on the feature amount in the feature information FM11. Further, for example, the calculation unit 133 calculates the score corresponding to the number of human faces "2" as "0.03" based on the feature amount in the feature information FM12. Further, for example, the calculation unit 133 calculates the score corresponding to the number of human faces "3" as "0.06" based on the feature amount in the feature information FM13. Further, for example, the calculation unit 133 calculates the score corresponding to the number of human faces "4 or more" as "0.75" based on the feature amount in the feature information FM14.

(Judgment unit 134)
The determination unit 134 determines various information. For example, the determination unit 134 determines the number of objects included in the input image based on the plurality of feature information acquired by the acquisition unit 131. For example, the determination unit 134 determines the number of objects included in the input image based on the score calculated by the calculation unit 133. The determination unit 134 determines the number of objects included in the input image based on the score corresponding to the number calculated by the calculation unit 133.

In the example of FIG. 1, the determination unit 134 determines the number of human faces included in the image IM 11. For example, the determination unit 134 determines the number of human faces included in the image IM 11 based on the scores of the feature information FM10 to FM14. For example, the determination unit 134 determines that the class (number) corresponding to the feature information FM14, which is the maximum score among the scores of the feature information FM10 to FM14, is the number of human faces included in the image IM11. For example, the determination unit 134 determines that the number of human faces included in the image IM 11 is 4 or more.

Further, the determination unit 134 determines the position of the target in the input image based on the information regarding the feature amount in the feature information corresponding to the number of the determined targets. The determination unit 134 determines the position of the human face included in the image IM 11. In the example of FIG. 1, the determination unit 134 determines the position of the human face included in the image IM 11 based on the feature information FM 14 which is the maximum score. For example, the determination unit 134 determines that the human face is included in the region where the feature amount equal to or higher than the predetermined threshold value is located. For example, the determination unit 134 determines that the human face is included in the region corresponding to the region AR11 (see FIG. 1) of the feature information FM14 in the image IM11. For example, as shown in the position determination information AP11, the determination unit 134 determines that the position of the human face included in the image IM11 is the upper part of the image IM11.

Further, for example, the determination unit 134 determines the position of the target included in the input image based on the information regarding the feature amount of each of the plurality of regions included in each feature information. Details of this case will be described later.

(Generator 135)
The generation unit 135 generates various information. For example, the generation unit 135 generates an image based on a plurality of feature information acquired by the acquisition unit 131, the number and positions of objects determined by the determination unit 134, and the like. For example, the generation unit 135 generates a processed image (hereinafter, also referred to as “cropping image”) by cropping the image IM11. The cropping here means a process of cutting a predetermined area from an image. Further, the image generated by the generation unit 135 by cropping is delivered as, for example, an image of predetermined content, and the details will be described later. When the determination device 100 only determines the number and position of objects in the image, it does not have to have the generation unit 135.

(Providing Department 136)
The providing unit 136 provides (transmits) various information to an external information processing device. For example, the providing unit 136 provides the processed image generated by the generating unit 135 to an external information processing device. Further, the providing unit 136 provides processed images IM12 to IM14 and the like (see FIG. 6). Further, the providing unit 136 provides information regarding the number and positions of the objects determined by the determining unit 134 to the external information processing device.

[3. Flow of judgment processing of the number of objects included in the image]
Here, with reference to FIG. 5, a procedure for determining the number of objects included in the image by the determination device 100 according to the embodiment will be described. FIG. 5 is a flowchart showing an example of determination of the number of objects according to the embodiment.

As shown in FIG. 5, the determination device 100 acquires an image (step S101). In FIG. 1, the determination device 100 acquires the image IM 11. After that, the determination device 100 inputs the image acquired in step S101 to the learner (step S102). In FIG. 1, the determination device 100 inputs the acquired image IM 11 to the learner LE.

After that, the determination device 100 acquires the feature information corresponding to each of the plurality of classes (numbers) (step S103). In FIG. 1, the determination device 100 acquires feature information FM10 to FM14 corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more from the learner LE. ..

After that, the determination device 100 calculates the score corresponding to each class (number) based on the feature information corresponding to each class (number) (step S104). In FIG. 1, the determination device 100 assigns a score corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more based on each of the feature information FM10 to FM14. calculate.

After that, the determination device 100 determines the number of objects included in the image based on the score corresponding to each class (number) (step S105). In FIG. 1, the determination device 100 determines the number of human faces included in the image IM 11 based on the scores of the feature information FM10 to FM14.

[4. Generation of processed images]
Here, the generation of the processed image by the determination device 100 will be described with reference to FIG. FIG. 6 is a diagram showing an example of generating a processed image according to the embodiment. In the example of FIG. 6, an example of cropping an image based on the position of a person's face determined by the determination unit 134 of the determination device 100 is shown. For example, such image cropping is performed by the generation unit 135 of the determination device 100.

The image IM11 shown in FIG. 6 is assumed to be the same as the image IM11 shown in FIG. That is, the image IM11 shown in FIG. 6 is determined to include a human face in the region (hereinafter, also referred to as “feature region”) corresponding to the region AR11 (see FIG. 1) of the feature information FM14, and is included in the image IM11. It is assumed that the position of the face of the person is determined to be the upper part of the image IM11.

In the example of FIG. 6, the case where the determination device 100 generates the processed images IM12 to IM14 corresponding to the three types of aspect ratios from the image IM11 is shown. For example, when the aspect ratio is "1: 1", the determination device 100 crops the image IM 11 to generate a processed image IM 12 having an aspect ratio of "1: 1" and including a feature region of the image IM 11. do. Further, for example, when the aspect ratio of the determination device 100 is "2: 1", the aspect ratio is "2: 1" by cropping the image IM11, and the processed image IM13 including the characteristic region of the image IM11 is included. To generate. Further, for example, when the aspect ratio of the determination device 100 is "4: 3", the aspect ratio is "4: 3" by cropping the image IM11, and the processed image IM14 including the feature area of the image IM11 is included. To generate. In this way, the determination device 100 can generate processed images IM12 to IM14 and the like according to various aspect ratios. For example, when the aspect ratio is specified, the determination device 100 may generate a processed image satisfying the specified aspect ratio from the image IM 11.

The processed images IM12 to IM14 corresponding to each aspect ratio shown in FIG. 6 are examples, and the determination device 100 may generate any processed image as long as the aspect ratio is satisfied. For example, when the aspect ratio is specified, the determination device 100 expands or contracts the cutting frame satisfying the aspect ratio, or moves the cutting frame in the feature information FM14 to feature the area in the cutting frame in the feature information FM14. By cropping the region where the average amount is maximum, a processed image satisfying the specified aspect ratio may be generated. Further, for example, the determination device 100 may generate any processed image as long as it includes the feature region of the image IM 11 and satisfies the aspect ratio.

Further, in the example shown in FIG. 6, a case where the processed image IM 13 having an aspect ratio of “2: 1” generated by the determination device 100 is displayed on the terminal device 10 used by the user U1 is shown. In the example shown in FIG. 6, the terminal device 10 displays the contents CT11 to CT14 distributed from the predetermined content distribution device. When the determination device 100 distributes the content, the content CT11 to CT14 may be transmitted from the determination device 100 to the terminal device 10.

In the example shown in FIG. 6, the processed image IM13 is used for the image of the content CT14. As described above, the processed image IM 13 may be used as an image of the content displayed side by side in the scroll direction in the terminal device 10. In this way, the processed image IM 13 generated by the determination device 100 is displayed on various terminal devices 10 such as smartphones. As described above, since the determination device 100 can generate the processed image according to various aspect ratios, it is possible to generate an appropriate processed image regardless of the type of the terminal device 10.

[5. Image processing flow]
Next, the procedure of image processing by the determination device 100 according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of generating a processed image according to the embodiment.

As shown in FIG. 7, the determination device 100 acquires the image and the feature information corresponding to the number of objects in the image (step S201). In FIG. 1, the determination device 100 acquires the feature information FM14 corresponding to the number of objects (4 or more) in the image IM11 and the image IM11.

After that, the determination device 100 acquires the aspect ratio (step S202). For example, the determination device 100 acquires the aspect ratio "2: 1". After that, the determination device 100 generates a cropping image so as to include an object in the image based on the aspect ratio and the feature information (step S203). For example, the determination device 100 generates a processed image IM 13 including a feature region of the image IM 11 based on the aspect ratio “2: 1” and the feature information FM14.

[6. Learning process]
Here, the learning process in the learning unit 132 of the determination device 100 will be described with reference to FIGS. 8 and 9. 8 and 9 are diagrams showing an example of the learning process according to the embodiment.

First, the learning device LE used by the determination device 100 will be briefly described. The learning device LE used by the determination device 100 is, for example, a learning device in which a plurality of nodes that output calculation results for the input data are connected in multiple layers, and the features of the image abstracted by supervised learning are learned. It is a learning device. For example, the learner LE is a neural network in which layers having a plurality of nodes are connected in multiple stages, and may be a DNN (Deep Neural Network) realized by a so-called deep learning technique. In addition, the features of the image are not only the specific features that appear in the image, such as the presence or absence of characters contained in the image, the color, and the composition, but also what the imaged object is and what kind of user the image is. It may be a concept that also includes the features of an abstracted (meta-ized) image, such as the atmosphere of the image.

For example, the learning device LE is generated by the following learning method by the technique of deep learning. For example, in the learner, the connection coefficient between each node is initialized, and an image having various characteristics is input. Then, the learner is generated by processing such as backpropagation (error back propagation method) that corrects a parameter (connection coefficient) so that an error between the output in the learner and the input image is reduced. For example, the learner is generated by performing processing such as backpropagation so as to minimize a predetermined loss function such as an error function. By repeating the process as described above, the learner can output an output that can better reproduce the input image, that is, a feature of the input image.

The learning method of the learning device is not limited to the above-mentioned method, and any known technique can be applied. Further, as the information used when learning the learner, data sets of various images such as an image and the number of objects included in the image may be used. The information used when learning the learner includes an image containing one target and a set of information indicating that the target is one, and an image containing a plurality of targets (for example, two) and a plurality of targets. You may use a set of information indicating that (for example, two), an image that does not include the target, a set of information that indicates that the target is not included (0), and the like. In addition, any method can be applied to the method of inputting an image to the learning device, the format of the data output by the learning device, the content of the feature to be explicitly learned by the learning device, and the like. That is, the determination device 100 can use any learning device as long as it can calculate the feature amount indicating the abstracted feature from the image.

In FIG. 1, the determination device 100 uses a learner LE by a so-called convolutional neural network (CNN) that repeats convolution and pooling of a local region of an input image. For example, the learning device LE by CNN has an output invariance with respect to a positional variation such as a character included in an image or an image pickup target, in addition to a function of extracting features from an image and outputting the feature. Therefore, the learner LE can accurately calculate the abstracted features of the image.

First, it will be described with reference to FIG. In the example shown in FIG. 8, the determination device 100 has an image IM21 including two human faces and an information RO21 indicating the number of faces included in the image IM21 (hereinafter, may be referred to as “correct answer information RO21”). The combination with and is acquired as teacher data (step S21). In FIG. 8, the correct answer information RO21 includes information indicating that the number of faces included in the image IM21 is two. Specifically, since the correct answer information RO21 has two faces included in the image IM21, the probability that the number of faces included in the image IM21 is two is "1 (100%)". Contains information indicating.

Then, an image IM21 including the faces of two people is input to the learner LE (step S22). After that, information indicating the probability of each number of objects as shown in the output information OC21-1 is output from the learner LE (step S23). In FIG. 8, the probability that the number of faces included in the image IM21 is 0 is “0.05 (5%)”, the probability that the number is 1 is “0.3 (30%)”, and 2 The probability of having 3 pieces is "0.5 (50%)", the probability of having 3 pieces is "0.1 (10%)", and the probability of having 4 or more pieces is "0.05 (5%)". The output information OC21-1 indicating that the above is output from the learner LE.

As described above, for example, the learning unit 132 learns and generates the learning device LE by the technique of deep learning. For example, the learning unit 132 uses a combination of an image and a predetermined number of objects in the image as teacher data. For example, the learning unit 132 corrects a parameter (connection coefficient) so that the error between the output in the learner LE and the probability of each number of predetermined objects included in the teacher data is small, backpropagation (error back propagation). The learner LE is learned by performing processing such as method). For example, the learning unit 132 generates the learning device LE by performing processing such as backpropagation so as to minimize a predetermined error (loss) function.

For example, the learning unit 132 uses an error function L as shown in the following equation (1). As shown in the following equation (1), the learning unit 132 uses the cross entropy as an error function, for example, in the case of the N-class classification problem. The error function L may be any function as long as it represents the certainty of the discrimination result. For example, the error function L may be the entropy obtained from the discrimination probability. Further, for example, the error function L may be any function as long as it indicates the accuracy of recognition of the learner LE.

Here, "x" in the above formula (1) and the following formulas (2) to (3) indicates an image. For example, in the example shown in FIG. 8, "x" in the above formula (1) and the following formulas (2) to (3) corresponds to the image IM21. Further, 0 to N assigned to the variable "n" correspond to each class identified (classified) by the learner LE. For example, the learner LE corresponding to the above equation (1) indicates that N classes are identified. For example, each class corresponds to "1", "2", etc., which indicate the number of objects.

Further, "t _n (x)" in the above equation (1) and the following equation (3) indicates the probability of the number of objects corresponding to the class n (any of 0 to N) in the image IM21. For example, "t _n (x)" in the above equation (1) indicates the probability of the number of objects corresponding to the class n as shown in the correct answer information RO21. In this case, for example, when the number of objects corresponding to class 0 is "0", "t ₀ (x)" becomes "0 (0%)". Further, for example, when the number of objects corresponding to class 2 is "2", "t ₂ (x)" becomes "1 (100%)".

Further, " _pn (x)" in the above equation (1) and the following equations (2) and (3) is the number of objects corresponding to the class n (any of 0 to N) in the image IM21. The probability based on the output of the learner LE is shown. For example, " _pn (x)" in the above equation (1) indicates the probability of the number of objects corresponding to the class n output by the learner LE as shown in the output information OC21-1. In this case, for example, when the number of objects corresponding to class 1 is "1", "p ₁ (x)" becomes "0.3 (30%)".

Further, " _pn (x)" in the above equation (1) is defined by the Softmax function as shown in the following equation (2) with a probability of class n with respect to x.

The function "f _n " in the above equation (2) is a score of the class n output by the CNN (learner LE). "Θ" is a parameter of CNN (learner LE). Further, the function "exp" is an exponential function. In this case, the gradient of the error function L (1) shown in the above equation (1) is calculated by the following equation (3).

As shown in the above equation (3), when _pn (x) = t _n (x) in all the classes from 1 to N, the gradient of the error function L (x) becomes 0 and becomes an extreme value. .. For example, the learning unit 132 performs feedback processing so that the gradient of the error function L (x) becomes 0 (step S24). For example, when the learning unit 132 repeats the above-mentioned processing, the learning device LE can appropriately output information regarding the number of objects in the input image. Note that FIG. 8 is for visually showing that the process is repeated so as to minimize the error function L and the like in order to bring the output of the learner LE closer to the correct answer information RO21. It may be done automatically.

Next, it will be described with reference to FIG. In the example of FIG. 9, a case where the determination device 100 uses the image IM31 including the faces of two people including the whole and the face of one person including half as the teacher data is shown. That is, in the example of FIG. 9, the determination device 100 uses the image IM 31 including 2.5 human faces as teacher data.

As described above, in the example of FIG. 9, the determination device 100 includes an image IM 31 including 2.5 human faces and an information RO 31 indicating the number of faces included in the image IM 31 (hereinafter referred to as “correct answer information RO 31”). The combination with (may be described) is acquired as teacher data (step S31). In FIG. 8, the correct answer information RO31 includes information indicating that the number of faces included in the image IM31 is 2.5. Specifically, since the correct answer information RO31 has 2.5 faces included in the image IM31, the probability that the number of faces included in the image IM31 is 2 is "0.5 (50%)). , And includes information indicating that the probability that the number of faces included in the image IM31 is three is "0.5 (50%)".

For example, as shown in the above-mentioned correct answer information RO31, the probability that the number of faces included in the image IM31 is 2 is "0.5 (50%)", and the number of faces included in the image IM31 is 3. The information indicating that the probability of being "0.5 (50%)" satisfies "2.5 (pieces) = 2 (pieces) x 0.5 + 3 (pieces) x 0.5". In this way, when learning is performed, the determination device 100 is included in the image using the probability of the class corresponding to each number and each number even when the human face is partially included. The number of objects can be estimated appropriately.

Then, an image IM 31 including the faces of two people is input to the learner LE (step S32). After that, information indicating the probability of each number of objects as shown in the output information OC31-1 is output from the learner LE (step S33). In FIG. 9, the probability that the number of faces included in the image IM31 is 0 is "0 (0%)", the probability that the number is 1 is "0.1 (10%)", and the number is 2. A certain probability is "0.4 (40%)", a probability of 3 is "0.4 (40%)", and a probability of 4 or more is "0.1 (10%)". The output information OC31-1 indicating that there is is output from the learner LE.

In the example shown in FIG. 9, similarly to the example shown in FIG. 8, the learning unit 132 uses the above equations (1) to (3) so that the gradient of the error function L (x) becomes 0. Feedback processing is performed (step S34). For example, by repeating the above-mentioned processing by the learning unit 132, the learning device LE can appropriately output the number of objects included in the input image even when there are a plurality of objects. Note that FIG. 9 is for visually showing that the process is repeated so as to minimize the error function L and the like in order to bring the output of the learner LE closer to the correct answer information RO31. It may be done automatically.

[7. Learning process flow]
Here, the procedure of the learning process by the determination device 100 according to the embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing an example of the learning process according to the embodiment.

As shown in FIG. 10, the determination device 100 acquires the image and the correct answer information regarding the number of objects in the image (step S301). In FIG. 8, the determination device 100 acquires an image IM 21 including two human faces and a correct answer information RO 21 indicating the number of human faces included in the image. After that, the determination device 100 inputs the image acquired in step S301 into the learner (step S302). In FIG. 8, the determination device 100 inputs the acquired image IM21 to the learner LE.

After that, the determination device 100 learns so that the error between the probability of each number of human faces based on the output of the learner and the probability of each number of human faces in the correct answer information becomes small (step S303). In FIG. 8, the determination device 100 is based on the probability of each number of human faces shown in the output information OC21-1 based on the output of the learning device LE and the probability of each number of human faces shown in the correct answer information RO21. learn.

After that, when the predetermined condition is satisfied (step S304: Yes), the determination device 100 ends the process. For example, the determination device 100 starts learning when the error between the probability of each number of human faces based on the output of the learning device and the probability of each number of human faces in the correct answer information is within a predetermined threshold value. When a predetermined time has elapsed since then, the process may be terminated on the assumption that the predetermined condition is satisfied. Further, when the determination device 100 does not satisfy the predetermined condition (step S304: No), the determination device 100 repeats the process of step S303. For example, the determination device 100 starts learning when the error between the probability of each number of human faces based on the output of the learning device and the probability of each number of human faces in the correct answer information is larger than a predetermined threshold value. If the predetermined condition is not satisfied when the predetermined time has not elapsed, the process of step S303 may be repeated. The above learning process is an example, and the determination device 100 may perform learning by various procedures.

[8. Image split]
In the above example, an example in which the determination device 100 performs processing on the entire image is shown, but the determination device 100 may divide the image into a plurality of ranges and perform processing. This point will be described with reference to FIG. FIG. 11 is a diagram showing an example of the determination process according to the embodiment.

Hereinafter, the determination process of the number of objects included in the image by the determination device 100 will be described with reference to FIG. As shown in FIG. 11, the image IM 11 is input to the determination device 100 (step S40). For example, the determination device 100 acquires an image IM 11 in which five people, that is, five faces, are captured as targets. The determination device 100 that has acquired the image IM 11 inputs the image IM 11 to a predetermined learning device (step S41).

Here, the determination device 100 divides the image IM 11 into four ranges (four ranges) and inputs the image IM 11 to the learner. In the example of FIG. 11, the determination device 100 includes the image IM11-1 corresponding to the upper left range, the image IM11-2 corresponding to the upper right range, the image IM11-3 corresponding to the lower left range, and the lower right range. The image IM11 is divided into four ranges (4 ranges) of the corresponding image IM11-4 and input to the learner LE. In the example of FIG. 11, although shown as step S41, the images IM11-1 to IM11-4 may be individually input to the learner LE.

For example, the learner LE to which the images IM11-1 to IM11-4 are input performs a process of identifying the number of objects included in each of the images IM11-1 to IM11-4. For example, the learner LE generates feature information corresponding to each class (number) in the process of identifying the number of objects. In the example of FIG. 11, the feature information corresponding to the number (class) of five faces of 0, 1, 2, 3, or 4 or more is each of the images IM11-1 to IM11-4. Is generated by the learner LE.

Therefore, the determination device 100 acquires each feature information generated in the process of identifying the number of human faces included in the image by the learning device LE for each of the images IM11-1 to IM11-4. In the example of FIG. 11, the determination device 100 acquires the feature information FM40-1 corresponding to the class in which the number of human faces is 0 from the learning device LE for the image IM11-1. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM40-2 corresponding to the class in which the number of human faces is 0 from the learning device LE for the image IM11-2. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM40-3 corresponding to the class in which the number of human faces is 0 from the learning device LE for the image IM11-3. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM40-4 corresponding to the class in which the number of human faces is 0 from the learning device LE for the image IM11-4. In the following, the feature information FM40-1 to FM40-4 may be collectively referred to as the feature information FM40. As described above, the determination device 100 acquires the feature information FM40 corresponding to the class in which the number of human faces is 0 from the learning device LE for the image IM11 (step S42-0).

For example, the feature information FM40 indicates the feature amount of each pixel in the image IM11. The feature amount referred to here is, for example, a numerical value indicating the feature amount. Specifically, the positions of the points (pixels) constituting the feature information FM 40 correspond to the overlapping positions in the image IM 11 when superimposed on the image IM 11, and the feature information FM 40 corresponds to the characteristics of the corresponding pixels in the image IM 11. Indicates the amount. In the feature information FM40 in FIG. 11, the region showing the feature is shown in a dark color mode. That is, in the feature information FM40, the larger the feature amount, the darker the color is displayed. Specifically, in the feature information FM40 in FIG. 11, the region where the human face is located is shown in the image IM11 in a dark mode. The same applies to other feature information.

Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM41-1 corresponding to the class in which the number of human faces is one from the learning device LE for the image IM11-1. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM41-2 corresponding to the class in which the number of human faces is one from the learning device LE for the image IM11-2. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM41-3 corresponding to the class in which the number of human faces is one from the learning device LE for the image IM11-3. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM41-4 corresponding to the class in which the number of human faces is one from the learning device LE for the image IM11-4. In the following, the feature information FM41-1 to FM41-4 may be collectively referred to as the feature information FM41. As described above, the determination device 100 acquires the feature information FM41 corresponding to the class in which the number of human faces is one from the learning device LE for the image IM11 (step S42-1).

Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM42-1 corresponding to the class in which the number of human faces is two from the learning device LE for the image IM11-1. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM42-2 corresponding to the class in which the number of human faces is two from the learning device LE for the image IM11-2. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM42-3 corresponding to the class in which the number of human faces is two from the learning device LE for the image IM11-3. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM42-4 corresponding to the class in which the number of human faces is two from the learning device LE for the image IM11-4. In the following, the feature information FM42-1 to FM42-4 may be collectively referred to as the feature information FM42. As described above, the determination device 100 acquires the feature information FM42 corresponding to the class in which the number of human faces is two from the learning device LE for the image IM11 (step S42-2).

Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM43-1 corresponding to the class in which the number of human faces is three from the learning device LE for the image IM11-1. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM43-2 corresponding to the class in which the number of human faces is three from the learning device LE for the image IM11-2. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM43-3 corresponding to the class in which the number of human faces is three from the learning device LE for the image IM11-3. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM43-4 corresponding to the class in which the number of human faces is three from the learning device LE for the image IM11-4. In the following, the feature information FM43-1 to FM43-4 may be collectively referred to as the feature information FM43. As described above, the determination device 100 acquires the feature information FM43 corresponding to the class in which the number of human faces is three from the learning device LE for the image IM11 (step S42-3).

Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM44-1 corresponding to the class in which the number of human faces is 4 or more from the learning device LE for the image IM11-1. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM44-2 corresponding to the class in which the number of human faces is 4 or more from the learning device LE for the image IM11-2. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM44-3 corresponding to the class in which the number of human faces is 4 or more from the learning device LE for the image IM11-3. Further, in the example of FIG. 11, the determination device 100 acquires the feature information FM44-4 corresponding to the class in which the number of human faces is 4 or more from the learning device LE for the image IM11-4. In the following, the feature information FM44-1 to FM44-4 may be collectively referred to as the feature information FM44. As described above, the determination device 100 acquires the feature information FM44 corresponding to the class in which the number of human faces is 4 or more from the learning device LE for the image IM11 (step S42-4). Hereinafter, when steps S42-0 to S42-4 are described without distinction, they may be described as step S42.

After that, the determination device 100 calculates a score for each of the feature information FM40 to FM44 acquired in step S42. For example, the determination device 100 calculates the scores of the feature information FM40-1 to FM40-4 for the feature information FM40. For example, the determination device 100 calculates the score of the feature information FM40-1 based on the feature amount in the feature information FM40-1. Further, for example, the determination device 100 calculates the score of the feature information FM40-2 based on the feature amount in the feature information FM40-2. For example, the determination device 100 calculates the score of the feature information FM40-3 based on the feature amount in the feature information FM40-3. Further, for example, the determination device 100 calculates the score of the feature information FM40-4 based on the feature amount in the feature information FM40-4. As described above, the determination device 100 calculates the score of each of the feature information FM40-1 to FM40-4 based on the feature amount in each of the feature information FM40-1 to FM40-4 (step S43-0).

In the example of FIG. 11, as shown in the score information SC40, the determination device 100 sets the score of the feature information FM40-1 to "0" and the score of the feature information FM40-2 for the number of human faces "0". The score of "0", the score of the feature information FM40-3 is calculated as "0.9", and the score of the feature information FM40-4 is calculated as "0.8".

Further, for example, the determination device 100 calculates the score of each of the feature information FM41-1 to FM41-4 based on the feature amount in each of the feature information FM41-1 to FM41-4 (step S43-1). In the example of FIG. 11, as shown in the score information SC41, the determination device 100 sets the score of the feature information FM41-1 to "0" and the score of the feature information FM41-2 for the number of human faces "1". The score of "0", the score of the feature information FM41-3 is calculated as "0.05", and the score of the feature information FM41-4 is calculated as "0.1".

Further, for example, the determination device 100 calculates the score of each of the feature information FM42-1 to FM42-4 based on the feature amount in each of the feature information FM42-1 to FM42-4 (step S43-2). In the example of FIG. 11, as shown in the score information SC42, the determination device 100 sets the score of the feature information FM42-1 to "0.2" and the score of the feature information FM42-2 for the number of human faces "2". The score is calculated as "0.8", the score of the feature information FM42-3 is calculated as "0", and the score of the feature information FM42-4 is calculated as "0".

Further, for example, the determination device 100 calculates the score of each of the feature information FM43-1 to FM43-4 based on the feature amount in each of the feature information FM43-1 to FM43-4 (step S43-3). In the example of FIG. 11, as shown in the score information SC43, the determination device 100 sets the score of the feature information FM43-1 to "0.8" and the score of the feature information FM43-2 for the number of human faces "3". The score is calculated as "0.2", the score of the feature information FM43-3 is calculated as "0", and the score of the feature information FM43-4 is calculated as "0".

Further, for example, the determination device 100 calculates the score of each of the feature information FM44-1 to FM44-4 based on the feature amount in each of the feature information FM44-1 to FM44-4 (step S43-4). In the example of FIG. 11, as shown in the score information SC44, the determination device 100 sets the score of the feature information FM44-1 to "0" and the score of the feature information FM44-2 for the number of human faces "4 or more". Is calculated as “0”, the score of the feature information FM44-3 is calculated as “0”, and the score of the feature information FM44-4 is calculated as “0”.

After that, the determination device 100 determines the number of human faces included in each range by the following equation (4).

In the above formula (4), "Num" indicates the number of faces of the person to be determined. Further, " _pc " in the right side corresponds to each number of scores (probabilities). Further, " _nc " in the right side corresponds to each number (class). “C” corresponds to the number (class) of five faces of 0, 1, 2, 3, or 4 or more.

Here, the determination of the number of human faces included in the image IM11-1 corresponding to the upper left range in the image IM11 will be described. The number of human faces included in the image IM11-1 is calculated by the following formula (5) using the above formula (4).

2.8 = 0 x 0 + 0 x 1 + 0.2 x 2 + 0.8 x 3 + 0 x 4 ... (5)

The first term in the right-hand side of the above equation (5) corresponds to 0 classes and is a term obtained by multiplying the number of faces "0" by the score "0". Further, the second term in the right side of the above equation (5) corresponds to one class and is a term obtained by multiplying the number of faces "1" and the score "0". Further, the third term in the right side of the above equation (5) corresponds to two classes and is a term obtained by multiplying the number of faces "2" and the score "0.2". Further, the fourth term in the right side of the above equation (5) corresponds to three classes and is a term obtained by multiplying the number of faces "3" and the score "0.8". Further, the fifth term in the right side of the above equation (5) corresponds to four or more classes, and is a term obtained by multiplying the number of faces "4" and the score "0".

According to the above formula (5), the determination device 100 determines that the number of human faces included in the image IM11-1 is "2.8".

Further, the determination of the number of human faces included in the image IM11-2 corresponding to the upper right range in the image IM11 will be described. The number of human faces included in the image IM11-2 is calculated by the following formula (6) using the above formula (4).

2.2 = 0 x 0 + 0 x 1 + 0.8 x 2 + 0.2 x 3 + 0 x 4 ... (6)

According to the above formula (6), the determination device 100 determines that the number of human faces included in the image IM11-2 is "2.2". In the image IM11-3 and the image IM11-4, it is assumed that the score of the feature information corresponding to the case where the number of human faces is one or more is less than a predetermined threshold value, and the number is determined by the above equation (4). It is not necessary to perform (calculation).

As described above, in the example of FIG. 11, the determination device 100 includes 2.8 human faces in the upper left range of the image IM11 and 2.2 human faces in the upper right range of the image IM11. Judged to be included. Therefore, the determination device 100 can appropriately determine the position including the human face in the image IM11.

Further, the determination device 100 adds up the number of faces "2.8" calculated by the above formula (5) and the number of faces "2.2" calculated by the above formula (6). Therefore, it is determined that the number of human faces included in the entire image IM11 is "5". As described above, the determination device 100 can determine the number of human faces included in the entire image IM11 based on the number of human faces determined for the images IM11-1 to IM11-4 corresponding to each range. can.

In the example of FIG. 11, the image is divided into four areas of upper left, upper right, lower left, and lower right, but the image may be divided into various areas such as 9 areas and 16 areas.

[9. effect〕
As described above, the determination device 100 according to the embodiment has an acquisition unit 131 and a determination unit 134. The acquisition unit 131 is used for an input image (“image IM11”; the same applies hereinafter) input to a neural network (“learner LE”; the same applies hereinafter in the embodiment) that identifies the number of objects in the image. It is a plurality of feature information based on, and a plurality of feature information corresponding to each number identified by the neural network is acquired. The determination unit 134 determines the number of objects included in the input image based on the plurality of feature information acquired by the acquisition unit 131.

Thereby, the determination device 100 according to the embodiment can appropriately determine the number of objects included in the image by using the information in the neural network.

Further, in the determination device 100 according to the embodiment, the determination unit 134 determines the position of the target in the input image based on the information regarding the feature amount in the feature information corresponding to the number of the determined targets.

As a result, the determination device 100 according to the embodiment determines the position of the human face included in the input image based on the feature information corresponding to the number of determined target faces. Therefore, the determination device 100 can appropriately determine the position of the target included in the image.

Further, in the determination device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of feature information based on the input image input to the neural network that identifies the number of human faces in the image.

Thereby, the determination device 100 according to the embodiment can appropriately determine the number of human faces included in the image by using the information in the neural network.

Further, the determination device 100 according to the embodiment has a calculation unit 133. The calculation unit 133 calculates the score based on a plurality of feature information. The determination unit 134 determines the number of objects included in the input image based on the score calculated by the calculation unit 133.

As a result, the determination device 100 according to the embodiment determines the number of objects included in the image based on the calculated score, and appropriately determines the number of objects included in the image using the information in the neural network. be able to.

Further, in the determination device 100 according to the embodiment, the calculation unit 133 calculates the score corresponding to each number based on the feature information corresponding to each number. The determination unit 134 determines the number of objects included in the input image based on the score corresponding to each number calculated by the calculation unit 133.

As a result, the determination device 100 according to the embodiment determines the number of objects included in the image based on the score corresponding to each calculated number, thereby using the information in the neural network to determine the objects included in the image. The number can be determined appropriately.

Further, in the determination device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of feature information corresponding to each number identified by the neural network that performs the convolution process and the pooling process.

Thereby, the determination device 100 according to the embodiment can appropriately determine the number of human faces included in the image by using the information in the neural network that performs the convolution process and the pooling process.

Further, in the determination device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of feature information corresponding to each number identified by the neural network that does not include the fully connected layer.

Thereby, the determination device 100 according to the embodiment can appropriately determine the number of human faces included in the image by using the information in the neural network (FCN) that does not include the fully connected layer.

Further, in the determination device 100 according to the embodiment, the determination unit 134 determines the position of the target included in the input image based on the information regarding the feature amounts of the plurality of regions included in the feature information.

As a result, the determination device 100 according to the embodiment determines the position of the target included in the input image based on the information regarding the feature amounts of the plurality of regions included in the feature information, thereby providing information in the neural network. Is used to appropriately determine the number of objects included in the image. Therefore, the determination device 100 can appropriately determine the position of the target included in the image.

[10. Hardware configuration]
The determination device 100 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the function of the determination device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via the network N and sends it to the CPU 1100, and transmits the data generated by the CPU 1100 to the other device via the network N.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the determination device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the function of the control unit 130 by executing the program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be acquired from another device via the network N.

The embodiments of the present application have been described in detail with reference to the drawings, but these are examples, and various modifications and improvements are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure line of the invention. It is possible to carry out the present invention in other forms described above.

[11. others〕
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in any unit according to various loads and usage conditions. Can be integrated and configured.

Further, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

100 Judgment device 121 Learning information storage unit 122 Image information storage unit 130 Control unit 131 Acquisition unit 132 Learning unit 133 Calculation unit 134 Judgment unit 135 Generation unit 136 Providing unit

Claims (10)

A plurality of feature information based on an input image input to a neural network that identifies the number of objects in an image, and an acquisition unit that acquires a plurality of feature information corresponding to each number identified by the neural network.
A determination unit that determines the number of objects included in the input image based on the plurality of feature information acquired by the acquisition unit.
A calculation unit that calculates a score corresponding to each number based on the feature information corresponding to each number, and a calculation unit.
Based on the area where the feature amount above a predetermined threshold value included in the feature information which is the maximum score among the scores calculated by the calculation unit is located, and a plurality of aspect ratios having different aspect ratios . A generator that generates cropped processed images,
A determination device characterized by comprising. The determination unit
The determination device according to claim 1, wherein the position of the object in the input image is determined based on the information regarding the feature amount in the feature information corresponding to the determined number of objects. The acquisition unit
The determination device according to claim 1 or 2, wherein the plurality of feature information based on an input image input to a neural network for identifying the number of human faces in an image is acquired. The determination unit
The determination device according to claim 3, wherein the number of the objects included in the input image is determined based on the score calculated by the calculation unit. The determination unit
The determination device according to claim 4, wherein the number of the objects included in the input image is determined based on the score corresponding to each of the numbers calculated by the calculation unit. The acquisition unit
The determination device according to any one of claims 1 to 5, wherein a plurality of feature information corresponding to each number identified by the neural network that performs convolution processing and pooling processing is acquired. The acquisition unit
The determination device according to claim 6, wherein a plurality of feature information corresponding to each number identified by the neural network not including the fully connected layer is acquired. The determination unit
The invention according to any one of claims 1 to 7, wherein the position of the target included in the input image is determined based on the information regarding the feature amount of each of the plurality of regions included in the feature information. Judgment device. It is a judgment method executed by a computer.
A plurality of feature information based on an input image input to a neural network that identifies the number of objects in an image, and an acquisition process for acquiring a plurality of feature information corresponding to each number identified by the neural network.
A determination step of determining the number of the objects included in the input image based on the plurality of feature information acquired by the acquisition step, and a determination step.
A calculation process for calculating a score corresponding to each number based on the feature information corresponding to each number, and a calculation step.
Based on the region where the feature amount above a predetermined threshold value included in the feature information which is the maximum score among the scores calculated by the calculation step is located, and a plurality of aspect ratios having different aspect ratios . The generation process to generate cropped processed images and
A determination method characterized by including. An acquisition procedure for acquiring a plurality of feature information based on an input image input to a neural network that identifies the number of objects in an image and corresponding to each number identified by the neural network.
A determination procedure for determining the number of the objects included in the input image based on the plurality of feature information acquired by the acquisition procedure, and a determination procedure.
A calculation procedure for calculating a score corresponding to each number based on the feature information corresponding to each number, and a calculation procedure.
Based on the area where the feature amount above a predetermined threshold value included in the feature information, which is the maximum score among the scores calculated by the calculation procedure, is located, and a plurality of aspect ratios having different aspect ratios . The generation procedure to generate the cropped processed image and
A judgment program characterized by having a computer execute.

Images