JP6826389B2 - Estimator, estimation method, and estimation program - Google Patents

Description

The present invention relates to an estimation device, an estimation method, and an estimation program.

Conventionally, a technique for extracting image features by a neural network has been provided. For example, a convolutional neural network (Convolutional Neural Network) provides a technique for identifying an object contained in an image.

Japanese Unexamined Patent Publication No. 2016-33806

However, in the above-mentioned conventional technique, it is difficult to appropriately estimate in what manner the image includes the object. For example, just by determining that an object is included in an image, it is not always possible to properly estimate the mode in which the object is included in the image.

The present application has been made in view of the above, and an object of the present application is to provide an estimation device, an estimation method, and an estimation program for appropriately estimating an aspect in which an object is included in an image.

The estimation device according to the present application is based on an acquisition unit that acquires an image, a learner that outputs an occupancy rate of a predetermined target in the input image according to the input image, and an image acquired by the acquisition unit. It is characterized by including an estimation unit for estimating the occupancy rate of the predetermined object in the image acquired by the acquisition unit.

According to one aspect of the embodiment, there is an effect that the aspect in which the object is included in the image can be appropriately estimated.

FIG. 1 is a diagram showing an example of a search process according to an embodiment. FIG. 2 is a diagram showing an example of a search process according to the embodiment. FIG. 3 is a diagram showing a configuration example of the search device according to the embodiment. FIG. 4 is a diagram showing an example of the learning information storage unit according to the embodiment. FIG. 5 is a diagram showing an example of an image information storage unit according to an embodiment. FIG. 6 is a diagram showing an example of the list information storage unit according to the embodiment. FIG. 7 is a flowchart showing an example of estimating the occupancy rate of the object in the image according to the embodiment. FIG. 8 is a flowchart showing an example of determining the ranking according to the embodiment. FIG. 9 is a diagram showing an example of the learning process according to the embodiment. FIG. 10 is a diagram showing an example of the learning process according to the embodiment. FIG. 11 is a flowchart showing an example of the learning process according to the embodiment. FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the function of the search device.

Hereinafter, the estimation device, the estimation method, and the mode for carrying out the estimation program (hereinafter referred to as “the embodiment”) according to the present application will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the estimation device, estimation method, and estimation program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

(Embodiment)
[1. Search process]
An example of the search process according to the embodiment will be described with reference to FIGS. 1 and 2. 1 and 2 are diagrams showing an example of a search process according to an embodiment. Specifically, FIG. 1 is a diagram showing an example of estimating the occupancy rate of the target in each of the image groups to be searched, which will be described later. The search device 100 shown in FIG. 1 estimates the occupancy rate of the target in each of the image groups to be searched by using a learning device (model) that outputs (estimates) the occupancy rate of the target in the image.

Further, specifically, FIG. 2 is a diagram showing an example of providing a search result according to the occupancy rate of the target in the query image when the query image is acquired. The search device 100 shown in FIG. 2 estimates the occupancy rate of the target in the query image by using the learner LE that outputs (estimates) the occupancy rate of the target in the image. Further, the search device 100 provides the search result to the user according to the occupancy rate of the target in the query image. The query image referred to here means an image indicating a condition specified in the search, and may include a processing request (inquiry) for extracting an image satisfying a predetermined condition with the query image. Further, the learner LE that outputs (estimates) the occupancy rate of the object in the image is generated by using the input image and the correct answer information indicating the occupancy rate of the object or the like in the input image, and the details will be described later. Further, in the examples shown in FIGS. 1 and 2, the case where the target for estimating the occupancy rate is a cat is shown, but the target is not limited to cats, but other organisms such as dogs and humans, plants, cars, and the like. It may be various objects or the like. Further, the objects referred to here may include various objects as long as they can be identified, and may include various phenomena such as fire and sea waves.

Here, the learning device used by the search device 100 will be briefly described. The learner used by the search device 100 is, for example, a learner in which a plurality of nodes that output calculation results for the input data are connected in multiple layers, and the features of the image abstracted by supervised learning are learned. It is a learning device. For example, the learner is a neural network in which layers having a plurality of nodes are connected in multiple stages, and may be a DNN (Deep Neural Network) realized by a so-called deep learning technique. In addition, the features of the image are not only the specific features that appear in the image such as the presence / absence, color, and composition of characters contained in the image, but also what the imaged object is and what kind of user the image is. It is a concept that also includes the features of an abstracted (meta-ized) image, such as the atmosphere of the image.

For example, the learning device is generated by the following learning method by the technique of deep learning. For example, in the learner, the connection coefficient between each node is initialized, and images having various characteristics are input. Then, the learner is generated by processing such as backpropagation (error backpropagation method) that corrects a parameter (connection coefficient) so that an error between the output in the learner and the input image is reduced. For example, the learner is generated by performing processing such as backpropagation so as to minimize a predetermined loss function such as an error function. By repeating the above-mentioned processing, the learner can output an output capable of better reproducing the input image, that is, an output of the characteristics of the input image.

The learning method of the learning device is not limited to the above-mentioned method, and any known technique can be applied. Further, as the information used when learning the learner, a data set of various images such as an image and information indicating an object included in the image may be used. The information used when learning the learner includes an image containing one target and a set of information indicating that the target is one, and an image containing a plurality of targets (for example, two targets) and a plurality of targets. You may use a set of information indicating that (for example, two), an image that does not include an object, a set of information that indicates that the object is not included (0), and the like. In addition, any method can be applied to the method of inputting an image to the learning device, the format of the data output by the learning device, the content of the feature to be explicitly learned by the learning device, and the like. That is, the search device 100 can use any learning device as long as it can calculate the feature amount indicating the abstracted feature from the image.

In FIG. 1, the search device 100 uses a learner LE based on a so-called convolutional neural network (CNN) that repeats convolution and pooling of a local region of an input image. In the following, the convolutional neural network may be referred to as CNN. For example, the learning device LE by CNN has a function of extracting features from an image and outputting the feature, and also has an output invariance with respect to a positional variation such as a character or an image pickup target included in the image. Therefore, the learner LE can accurately calculate the abstracted features of the image.

For example, in FIG. 1, the search device 100 uses a learning device LE, which is a discriminator (model) that outputs information regarding the occupancy rate of an object (cat) in an image. That is, in FIG. 1, the search device 100 uses the learner LE that has been generated by the predetermined learning process as described above. Note that FIG. 1 shows a case where the search device 100 uses a learning device LE that identifies an object included in an image. However, if the search device 100 can estimate the occupancy rate of a predetermined object, what kind of learning can be done? A vessel may be used. Further, when the learning device LE is generated (learning), a predetermined loss function, correct answer information, and the like are used, but the details will be described later.

As shown in FIG. 2, the search system 1 includes a terminal device 10 and a search device 100. The terminal device 10 and the search device 100 are communicably connected by wire or wirelessly via a predetermined communication network (not shown). Further, the search system 1 shown in FIG. 2 may include a plurality of terminal devices 10 and a plurality of search devices 100.

The terminal device 10 is an information processing device used by the user. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. In the example shown in FIG. 2, a case where the terminal device 10 is a smartphone is shown. In the following, the terminal device 10 may be referred to as a user. That is, in the following, the user can be read as the terminal device 10.

The search device 100 is an estimation device that estimates the occupancy rate of a predetermined object in the acquired image by using the learning device LE that outputs the occupancy rate of the predetermined object in the input image. Further, the search device 100 is an estimation device that acquires a query image from the terminal device 10 and extracts a search result image that satisfies the condition regarding the occupancy rate of the search target from a predetermined image group based on the acquired query image.

First, with reference to FIG. 1, the estimation of the occupancy rate of the target (cat) in each image to be searched by the search device 100 will be described. For example, the images IM11 to IM15 shown in FIG. 1 are stored in the image information storage unit 122 (FIG. 5). In the following, when the images IM11 to IM15 and the like are described without distinction, they may be described as "image IM".

For example, the search device 100 estimates the occupancy rate of the cat in the image IM by inputting the image IM into the learning device LE. Specifically, the learner LE in which the image IM is input outputs information indicating the occupancy rate of the cat in the input image IM, and the search device 100 outputs the occupancy of the cat in the image IM output by the learner LE. Based on the information indicating the rate, the occupancy rate of the cat in the image IM is estimated. The learner LE to which the image IM is input outputs information indicating the occupancy rate of various objects in the input image IM as well as the cat. In FIGS. 1 and 2, for simplification of explanation, only the occupancy rates of the two classes of cat and background are shown, but the learner LE may output information indicating the occupancy rates of the other classes. Good. The class referred to here is a class in machine learning, and may be a type classified by a learning device or the like. For example, the learner LE may output information indicating the occupancy rate for 20 classes. In this case, the learner LE may output information indicating the occupancy rate of classes such as dogs, airplanes, and bicycles, in addition to the two classes corresponding to cats and backgrounds. It should be noted that the learning device LE outputs information indicating the occupancy rate of which object (class) depends on the learning, but the details will be described later.

Further, the search device 100 may estimate the occupancy rate of the cat in the image IM by inputting all the image IMs to be searched into the learning device LE, but it is assumed that the target cats are included. Only the image IM may be input to the learner LE to output information indicating the occupancy rate of the cat in the image IM. For example, the search device 100 inputs only the image IM associated with the tag “cat” among the image IMs in the image information storage unit 122 (FIG. 5) into the learning device LE, and occupies the cat in the image IM. Information indicating the rate may be output. For example, when the tag "cat" is associated with the images IM11 to IM15 shown in FIG. 1, the search device 100 inputs the images IM11 to IM15 and the like into the learning device LE, and the occupancy rate of the cat in the image IM. Information indicating that may be output.

In the example of FIG. 1, the search device 100 inputs the image IM 11 to the learner LE (step S11-1). For example, the search device 100 acquires the image IM 11 from the image information storage unit 122 (FIG. 5) and inputs the image IM 11 to the learning device LE. After that, the search device 100 estimates the occupancy rate of the cat in the image IM 11 based on the output of the learning device LE (step S12-1). For example, as shown in the estimation information OC11, the search device 100 estimates that the occupancy rate of the cat in the image IM11 is 80% (0.8) based on the output of the learning device LE, and the occupancy rate of the background in the image IM11. Is estimated to be 18% (0.18). For example, the search device 100 estimates the occupancy rate of each object so that the total occupancy rate of each object (class) is 100%. For example, the search device 100 estimates that the total occupancy rate of each object (class) other than the cat and the background is 2%. In the present embodiment, the occupancy rate is expressed as a percentage "%" (for example, 80%), but may be a decimal point (for example, 0.8 etc.). For example, the learner LE may output a numerical value in the range of 0 to 1 indicating the occupancy rate of each object (class).

Further, in the example of FIG. 1, the search device 100 inputs the image IM12 to the learner LE (step S11-2). For example, the search device 100 acquires the image IM12 from the image information storage unit 122 (FIG. 5) and inputs the image IM12 to the learner LE. After that, the search device 100 estimates the occupancy rate of the cat in the image IM12 based on the output of the learning device LE (step S12-2). For example, as shown in the estimation information OC12, the search device 100 estimates the occupancy rate of the cat in the image IM12 to be 60% and the occupancy rate of the background in the image IM12 to be 36% based on the output of the learner LE. To do.

Further, in the example of FIG. 1, the search device 100 inputs the image IM 13 to the learner LE (step S11-3). For example, the search device 100 acquires the image IM13 from the image information storage unit 122 (FIG. 5) and inputs the image IM13 to the learning device LE. After that, the search device 100 estimates the occupancy rate of the cat in the image IM 13 based on the output of the learning device LE (step S12-3). For example, as shown in the estimation information OC13, the search device 100 estimates the occupancy rate of the cat in the image IM13 to be 20% and the occupancy rate of the background in the image IM13 to be 75% based on the output of the learner LE. To do.

Further, in the example of FIG. 1, the search device 100 inputs the image IM 14 to the learner LE (step S11-4). For example, the search device 100 acquires the image IM14 from the image information storage unit 122 (FIG. 5) and inputs the image IM14 to the learner LE. After that, the search device 100 estimates the occupancy rate of the cat in the image IM14 based on the output of the learning device LE (step S12-4). For example, as shown in the estimation information OC14, the search device 100 estimates the occupancy rate of the cat in the image IM14 to be 90% and the occupancy rate of the background in the image IM14 to be 9% based on the output of the learner LE. To do.

Further, in the example of FIG. 1, the search device 100 inputs the image IM 15 to the learner LE (step S11-5). For example, the search device 100 acquires the image IM15 from the image information storage unit 122 (FIG. 5) and inputs the image IM15 to the learning device LE. After that, the search device 100 estimates the occupancy rate of the cat in the image IM 15 based on the output of the learning device LE (step S12-5). For example, as shown in the estimation information OC15, the search device 100 estimates the occupancy rate of the cat in the image IM15 to be 70% and the occupancy rate of the background in the image IM15 to be 27% based on the output of the learner LE. To do.

Further, the search device 100 also inputs other image IMs associated with the tag "cat" into the learning device LE, and estimates the occupancy rate of the cat in each image IM. After that, the search device 100 generates list information indicating the occupancy rate of the cat in each of the estimated image IMs (step S13). In the example of FIG. 1, the search device 100 generates list information LT11 indicating the occupancy rate of the cat in the images IM11 to IM15 and the like. In the example of FIG. 1, the case where the search device 100 generates the list information LT11 is taken as an example for the sake of explanation, but the occupancy rate of the cat in each image IM estimated in steps S12-1 to S12-5 and the like is taken as an example. The information indicating the above may be stored in the image information storage unit 122 (FIG. 5) in association with each image.

Next, with reference to FIG. 2, the provision of the search result by the search device 100 when the query image is acquired will be described. First, the search device 100 acquires the query image IM 10 from the terminal device 10 used by the user U1 (step S14). In the example of FIG. 2, the search device 100 acquires the query image IM10 including the cat from the terminal device 10.

After that, the search device 100 inputs the acquired image IM10 to the learner LE (step S15). Then, the search device 100 estimates the occupancy rate of the cat in the image IM 10 based on the output of the learning device LE (step S16). For example, as shown in the estimation information OC10, the search device 100 estimates the occupancy rate of the cat in the image IM10 to be 68% and the occupancy rate of the background in the image IM10 to be 30% based on the output of the learner LE. To do. As described above, in the example of FIG. 2, the search device 100 acquires the image IM10 indicating that the search target is the “cat” and the condition related to the search target is the occupancy rate “68%” from the terminal device 10. To do. That is, by acquiring the query image (image IM10), the search device 100 acquires designated information indicating that the search target is the "cat" and the condition related to the search target is the occupancy rate "68%". ..

In the example of FIG. 2, the case where the search device 100 acquires the designated information by acquiring the query image (image IM10) is shown, but the search device 100 may acquire the designated information as character information. .. For example, when the user U1 inputs the search target "cat" and the occupancy rate "68%" to the terminal device 10, the search device 100 may acquire the input information as designated information. In this case, the search device 100 may provide the terminal device 10 with an input interface such as a screen in which the user U1 inputs the designated information as character information.

Then, the search device 100 extracts a search result image in which the search target is a "cat" and the condition related to the search target satisfies the occupancy rate "68%" (step S17). For example, the search device 100 extracts an image IM within a predetermined range from "68%", which is a condition of the cat occupancy rate, from the image group included in the list information LT11 indicating the cat occupancy rate. In the example of FIG. 2, the search device 100 extracts an image IM in which the occupancy rate of the cat is within ± 25% from “68%” which is a condition. For example, the search device 100 extracts an image IM in which the occupancy rate of the cat is in the range of 43% to 93%. Specifically, the search device 100 extracts the image IM11, the image IM12, the image IM14, the image IM15, and the like, as shown in the extraction information LT12. As described above, in the example of FIG. 2, the search device 100 extracts an image IM other than the image IM that does not satisfy the conditions such as the image IM 13 that the cat occupancy rate is 20% and is out of the range.

After that, the search device 100 determines the ranking of the extracted image IM (step S18). For example, as shown in the ranking information LT13, the search device 100 determines the ranking so that the ranking of the image IM having a small error from "68%", which is the condition of the cat occupancy, is high. Specifically, the search device 100 places the smallest image IM15 having an error of "2 (= 70-68)" as the first place, the image IM19 having the next smallest error after the image IM15 as the second place, and the image IM19. The image IM12 having the next smallest error is ranked third, and the image IM17 having the next smallest error after the image IM12 is ranked fourth.

After that, the search device 100 provides the user U1 with a search result based on the determined ranking (step S19). In the example of FIG. 2, the search device 100 sets the image IM15 as the first place, the image IM19 as the second place, the image IM12 as the third place, and the image IM17 as the fourth place. Provide to U1. For example, the search device 100 transmits a search result based on the determined ranking to the terminal device 10 used by the user U1.

As described above, the search device 100 appropriately estimates the mode in which the target is included in the image, that is, how much the target occupies in the image, by using the learning device that outputs the occupancy rate of the target in the image. be able to. Further, when the query image is acquired, the search device 100 can appropriately extract the image corresponding to the query image by extracting the image according to the occupancy rate of the target in the query image. As shown in FIG. 2, the search device 100 provides the user with a search result that more appropriately reflects the user's intention by extracting an image including the target on condition of the occupancy rate of the target in the query image. Can be done.

In the above example, the search device 100 extracts an image according to the difference in the occupancy rate of the cat, but the image may be extracted based on the degree of similarity. For example, the search device 100 may extract an image based on the distribution of the occupancy rates of a plurality of objects included in each image. For example, the search device 100 may extract an image based on an index value such as KL divergence. For example, the search device 100 determines a KL divergence value between the distribution of the occupancy rates of a plurality of objects included in the image and the distribution of the occupancy rates of a plurality of objects included in the query image among the images of the predetermined image group. An image within the threshold value of may be extracted as a search result image. The search device 100 is not limited to KL divergence, and an image may be extracted by appropriately using various index values as long as they are index values based on the similarity of distribution in each image.

Further, in the above-described example, the case where the search device 100 uses one learning device LE is shown, but the search device 100 may use the learning device properly for each target. For example, the search device 100 may use the learning device for dogs separately from the learning device for cats.

[2. Search device configuration]
Next, the configuration of the search device 100 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the search device 100 according to the embodiment. As shown in FIG. 3, the search device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The search device 100 has an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from the administrator of the search device 100, and a display unit (for example, a liquid crystal display, etc.) for displaying various information. You may.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 110 is connected to the network by wire or wirelessly, and transmits / receives information to / from, for example, the terminal device 10 included in the search system 1.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. As shown in FIG. 3, the storage unit 120 according to the embodiment includes a learning information storage unit 121, an image information storage unit 122, and a list information storage unit 123.

(Learning information storage unit 121)
The learning information storage unit 121 according to the embodiment stores various information related to learning. For example, in FIG. 4, the learning information storage unit 121 stores learning information (model) related to the learning device LE generated by a predetermined learning process. FIG. 4 shows an example of the learning information storage unit 121 according to the embodiment. The learning information storage unit 121 shown in FIG. 4 stores a “weight ( _wij )”.

For example, in the example shown in FIG. 4, it is shown that the “weight (w ₁₁ )” is “0.2” and the “weight (w ₁₂ )” is “−0.3”. Further, in the example shown in FIG. 4, it is shown that the “weight (w ₂₁ )” is “0.5” and the “weight (w ₂₂ )” is “1.3”.

The "weight ( _wij )" may be, for example, a synaptic connection coefficient from the neuron y _i to the neuron x _j in the learner LE. Further, the learning information storage unit 121 is not limited to the above, and may store various information depending on the purpose. Further, when the search device 100 uses different learners for each object, the search device 100 may store information about a plurality of learners. For example, when the search device 100 uses the learning device for dogs separately from the learning device for cats, the search device 100 may also store information about the learning device for dogs.

(Image information storage unit 122)
The image information storage unit 122 according to the embodiment stores various information related to the image. FIG. 5 shows an example of the image information storage unit 122 according to the embodiment. For example, the image information storage unit 122 stores an image to be searched. The image information storage unit 122 shown in FIG. 5 has items such as an “image ID”, an “image”, and a “tag”.

The "image ID" indicates identification information for identifying an image. "Image" indicates image information. Specifically, "image" indicates an image to be searched. In FIG. 5, an image identified by an image ID is illustrated for explanation, but as the “image”, a file path name or the like indicating a storage location of the image may be stored. The "tag" is information attached in association with the image, and indicates an object included in the image.

For example, in the example shown in FIG. 5, the image identified by the image ID “IM11” is the image IM11 including a cat, indicating that the tag “cat” is associated and stored. The tag attached to each image may be attached by an administrator or the like who registers the image, or may be automatically attached by identifying the target by appropriately using various techniques of image recognition. ..

The image information storage unit 122 is not limited to the above, and various information may be stored depending on the purpose. For example, the image information storage unit 122 may store information regarding the date and time when the image was generated. Further, for example, the image information storage unit 122 may store information about an object included in the image. Further, for example, the image information storage unit 122 may store the acquired original image.

(List information storage unit 123)
The list information storage unit 123 according to the embodiment stores various information regarding the occupancy rate of the object in the image. For example, the list information storage unit 123 stores various information regarding the occupancy rate of the target in the image for each target. FIG. 6 is a diagram showing an example of the list information storage unit according to the embodiment. In the example shown in FIG. 6, the list information storage unit 123 stores information (table) for each included object such as the list information LT11 and the list information LT21. For example, the list information LT11 and the list information LT21 have items such as "image ID" and "occupancy rate (%)".

The "image ID" indicates identification information for identifying an image. "Image" indicates image information. The "occupancy rate (%)" of the list information LT11 includes items such as "cat" and "background". In addition, the "occupancy rate (%)" of the list information LT21 includes items such as "dog" and "background".

For example, as shown in the list information LT11 in FIG. 6, the image (IM11) identified by the image ID “IM11” has an occupancy rate of the target “cat” of “80”% and occupies the target “background”. Indicates that the rate is "18"%. Further, for example, as shown in the list information LT11 in FIG. 6, in the image (IM12) identified by the image ID “IM12”, the occupancy rate of the target “cat” is “60”%, and the target “background”. Indicates that the occupancy rate of is "36"%.

The list information storage unit 123 is not limited to the above, and various information may be stored depending on the purpose. For example, the list information storage unit 123 may store information regarding a position in the target image. For example, the list information storage unit 123 may store various information regarding which portion of the image, such as the center, top, bottom, right, left, or the whole, the target is located. For example, the list information storage unit 123 may store an image corresponding to each image ID, a file path name indicating a storage location of the image, and the like. Further, in the example of FIG. 6, the list information storage unit 123 shows a case where a table is separately stored for each target, such as list information LT11 and list information LT21, but various information regarding the occupancy rate of the target and the like is stored. It may be stored in one table. For example, the list information storage unit 123 may store all the target items in one table by including all the target items in the “occupancy rate (%)”.

(Control unit 130)
Returning to the description of FIG. 3, the control unit 130 is a controller, and is stored in a storage device inside the search device 100 by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). It is realized by executing various programs (corresponding to an example of an estimation program) using the RAM as a work area. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 130 includes an acquisition unit 131, a learning unit 132, an estimation unit 133, an extraction unit 134, a determination unit 135, and a provision unit 136, and the information described below. Realize or execute the function or action of processing.

(Acquisition unit 131)
For example, the acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires an image. For example, the acquisition unit 131 acquires an image from the image information storage unit 122. In FIG. 1, the acquisition unit 131 acquires images IM11 to IM15 and the like from the image information storage unit 122. The acquisition unit 131 may acquire images IM11 to IM15 and the like from an external information processing device.

In addition, the acquisition unit 131 acquires information regarding the occupancy rate of the search target based on the designated information in the image search. For example, the acquisition unit 131 acquires information on the occupancy rate of the search target estimated from the query image as designated information. In FIG. 2, the acquisition unit 131 acquires the query image IM 10 from the terminal device 10 used by the user U1. In FIG. 2, the acquisition unit 131 acquires an image IM 10 from the terminal device 10 indicating that the search target is a “cat” and the condition relating to the search target is the occupancy rate of “68%”. For example, the acquisition unit 131 acquires the query image (image IM10), and the search target is "cat" by the estimation of the occupancy rate by the estimation unit 133 described later, and the condition related to the search target is the occupancy rate "68%". Acquires the specified information indicating that. Further, for example, when the user U1 inputs the search target "cat" and the occupancy rate "68%" to the terminal device 10, the acquisition unit 131 may acquire the input information as designated information.

(Learning Department 132)
The learning unit 132 learns various information. In addition, the learning unit 132 generates various information by learning. For example, the learning unit 132 learns a learning device (model). In other words, the learning unit 132 generates a learning device (model) by performing learning. For example, the learning unit 132 learns the learning device LE. For example, the learning unit 132 learns the learning device by combining an image with an occupancy rate of a predetermined object in the image. Further, the learning unit 132 learns the learning device so as to minimize a predetermined evaluation function. The details of the learning process performed by the learning unit 132 will be described later.

(Estimation unit 133)
The estimation unit 133 estimates various information. For example, the estimation unit 133 is acquired by the acquisition unit 131 based on the learning device that outputs the occupancy rate of a predetermined target in the input image according to the input image and the image acquired by the acquisition unit 131. Estimate the occupancy of a given object in an image. For example, the estimation unit 133 estimates the occupancy rate of a predetermined target in the image acquired by the acquisition unit 131 based on the learner which is a neural network learned from the predetermined data (teacher data). For example, the estimation unit 133 occupies a predetermined object in the image acquired by the acquisition unit 131 based on a learner which is a neural network learned by a combination of the image and the occupancy rate of the predetermined object in the image. Estimate the rate. For example, the estimation unit 133 estimates the occupancy rate of a predetermined target in the image acquired by the acquisition unit 131 based on the learner which is a neural network that performs the convolution process and the pooling process.

In FIG. 1, the estimation unit 133 estimates the occupancy rate of the cat in the image IM by inputting the image IM into the learner LE. Specifically, the learner LE to which the image IM is input outputs information indicating the occupancy rate of the cat in the input image IM, and the estimation unit 133 outputs the occupancy of the cat in the image IM output by the learner LE. The occupancy rate of the cat in the image IM is estimated based on the information indicating the rate. For example, the estimation unit 133 estimates the occupancy rate of the cat in the image IM11 based on the output of the learner LE. In FIG. 1, as shown in the estimation information OC11, the estimation unit 133 estimates that the occupancy rate of the cat in the image IM11 is 80% and the occupancy rate of the background in the image IM11 is 18% based on the output of the learner LE. Presumed to be. Further, in FIG. 1, the estimation unit 133 estimates the occupancy rate of the cat in the images IM12 to IM15 and the like based on the output of the learning device LE. For example, the estimation unit 133 estimates the occupancy rate of the cat in each image IM by inputting the image IM associated with the tag "cat" into the learner LE.

Further, in FIG. 2, the estimation unit 133 estimates the occupancy rate of the cat in the image IM10 based on the output of the learning device LE to which the image IM10 is input. For example, as shown in the estimation information OC10, the estimation unit 133 estimates that the occupancy rate of the cat in the image IM10 is 68% and the occupancy rate of the background in the image IM10 is 30% based on the output of the learner LE. To do.

(Extractor 134)
The extraction unit 134 extracts various information. For example, the extraction unit 134 extracts information about an image from the image information storage unit 122 and the list information storage unit 123. For example, the extraction unit 134 extracts an image satisfying the conditions from a predetermined image group stored in the image information storage unit 122 or the list information storage unit 123. For example, the extraction unit 134 extracts a search result image that satisfies the condition regarding the occupancy rate of the search target from a predetermined image group. For example, the extraction unit 134 is based on the degree of similarity between the information on the occupancy rate of the target including the search target in the query image and the information on the occupancy rate of the target including the search target in each image of the predetermined image group. Extract search result images.

In FIG. 2, the extraction unit 134 extracts a search result image in which the search target is a “cat” and the condition related to the search target satisfies the occupancy rate “68%”. For example, the extraction unit 134 extracts an image IM within a predetermined range from "68%", which is a condition of the cat occupancy rate, from the image group included in the list information LT11 indicating the cat occupancy rate. The extraction unit 134 extracts an image IM in which the occupancy rate of the cat is within ± 25% from “68%” which is a condition. For example, the extraction unit 134 extracts an image IM in which the occupancy rate of the cat is in the range of 43% to 93%. Specifically, as shown in the extraction information LT12, the extraction unit 134 extracts the image IM11, the image IM12, the image IM14, the image IM15, and the like. As described above, in the example of FIG. 2, the extraction unit 134 extracts an image IM other than the image IM that does not satisfy the conditions such as the image IM 13 that the cat occupancy rate is 20% and is out of the range.

(Decision part 135)
The determination unit 135 determines various information. For example, the determination unit 135 determines the ranking of a plurality of images based on the occupancy rate of a predetermined target in each of the plurality of images estimated by the estimation unit 133. For example, the determination unit 135 determines the ranking of the search result images according to the information regarding the occupancy rate of the search target in each of the search result images extracted by the extraction unit 134.

In FIG. 2, the determination unit 135 determines the ranking of the extracted image IM. For example, as shown in the ranking information LT13 in FIG. 2, the determination unit 135 determines the ranking so that the ranking of the image IM having a small error from "68%", which is the condition of the cat occupancy, is high. Specifically, the determination unit 135 places the smallest image IM15 having an error of "2 (= 70-68)" as the first place, the image IM19 having the next smallest error after the image IM15 as the second place, and the image IM19. The image IM12 having the next smallest error is ranked third, and the image IM17 having the next smallest error after the image IM12 is ranked fourth. Further, the determination unit 135 determines the ranking based on the similarity between the distribution of the occupancy rates of the plurality of objects included in each extracted image IM and the distribution of the occupancy rates of the plurality of objects included in the query image. May be good. For example, the determination unit 135 determines the higher the KL divergence value between the distribution of the occupancy rates of the plurality of objects included in each extracted image IM and the distribution of the occupancy rates of the plurality of objects included in the query image. You may.

For example, the determination unit 135 may determine the ranking of each of the search result images according to the difference in the occupancy rate of each of the search result images in each ranking of the search result images. For example, the determination unit 135 may determine the ranking of each of the search result images so that the difference in the occupancy rate of each of the consecutive search result images in each ranking of the search result images satisfies a predetermined condition. For example, the determination unit 135 may determine the ranking of the search result images so that the difference in the occupancy rate of the target in the image IM between the consecutive rankings is 5% or more. For example, in the determination unit 135, when the occupancy rate of the cat in the image IM having the first ranking is "70"%, the occupancy rate of the cat in the image IM having the second ranking is larger than "75"%, or The image IM having the second highest ranking may be determined so as to be less than "65"%.

For example, the determination unit 135 may determine the ranking of each of the search result images depending on whether or not each of the search result images includes an object other than the search target. For example, the determination unit 135 determines the search result image so that when the search target is a "cat" and the image includes a target other than a cat (for example, "dog"), the ranking of the image is lowered. Each ranking may be determined.

For example, the determination unit 135 may determine the ranking of each of the search result images depending on whether or not each of the search result images includes character information. For example, the determination unit 135 may determine the ranking of each of the search result images so that the ranking of the images in which the character information is included in each of the search result images is low. In this case, for example, the determination unit 135 may appropriately use various conventional techniques for recognizing characters to determine whether or not character information is included in each of the search result images.

For example, the determination unit 135 may determine the ranking of each of the search result images according to the position of the search target in each of the search result images. For example, when the target position in the query image is on the right, the determination unit 135 determines the ranking of each of the search result images so that the image whose target is located on the right is ranked higher among the search result images. You may. In this case, for example, the determination unit 135 may determine the position of the search target in each of the search result images based on the information indicating the position of the target in each of the search result images acquired by the acquisition unit 131. Further, for example, the determination unit 135 determines the position of the search target in each of the search result images based on the information indicating the position of the target in each of the search result images output from the learning device that holds the position information. May be good. The above is an example, and the determination unit 135 may use any information as long as the position of the search target in each of the search result images can be determined.

(Providing Department 136)
The providing unit 136 provides various information to an external information processing device. For example, the providing unit 136 provides information based on the ranking of a plurality of images (search result images) determined by the determining unit 135. In FIG. 2, the providing unit 136 provides the user U1 with a search result based on the determined ranking. For example, in the providing unit 136, the user U1 uses the search results in the display order based on the ranking in which the image IM15 is the first place, the image IM19 is the second place, the image IM12 is the third place, and the image IM17 is the fourth place. Provided to the terminal device 10. For example, the providing unit 136 transmits the search result based on the determined ranking to the terminal device 10.

[3. Flow of occupancy estimation processing]
Here, the procedure of the occupancy rate estimation process by the search device 100 according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of estimating the occupancy rate of the object in the image according to the embodiment.

As shown in FIG. 7, the search device 100 acquires an image (step S101). In FIG. 1, the search device 100 acquires images IM11 to IM15 and the like. After that, the search device 100 inputs the image acquired in step S101 into the learner (step S102). In FIG. 1, the search device 100 inputs the acquired images IM11 to IM15 and the like into the learning device LE.

After that, the search device 100 estimates the occupancy rate of the target in the input image based on the output of the learner (step S103). In FIG. 1, the search device 100 estimates the occupancy rate of the cat in the input images IM11 to IM15 and the like based on the output of the learning device LE.

After that, the search device 100 generates list information of images including the occupancy rate (step S104). In FIG. 1, the search device 100 generates list information LT11 indicating the occupancy rate of cats in images IM11 to IM15 and the like.

[4. Flow of ranking determination process]
Next, the procedure of the ranking determination process by the search device 100 according to the embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of determining the ranking according to the embodiment.

As shown in FIG. 8, the search device 100 acquires a query image (step S201). In FIG. 2, the search device 100 acquires the image IM10 as a query image. In step S201, the search device 100 may acquire character information instead of the query image. In this case, the search device 100 does not have to perform the process of step S202.

After that, the search device 100 inputs the query image into the learner (step S202). In FIG. 2, the search device 100 inputs the image IM 10 into the learner LE.

After that, the search device 100 estimates the occupancy rate of the target in the query image based on the output of the learner (step S203). In FIG. 2, the search device 100 estimates the occupancy rate of the cat in the input image IM10 based on the output of the learning device LE.

After that, the search device 100 extracts an image from the list information based on the occupancy rate of the search target in each image in the list information and the occupancy rate of the search target in the query image (step S204). In the example of FIG. 2, the search device 100 extracts an image IM in which the occupancy rate of the cat is within ± 25% from “68%” which is a condition. For example, the search device 100 extracts the image IM11, the image IM12, the image IM14, the image IM15, and the like as shown in the extraction information LT12.

After that, the search device 100 determines the ranking of the extracted images (step S205). For example, as shown in the ranking information LT13, the search device 100 determines the ranking so that the ranking of the image IM having a small error from "68%", which is the condition of the cat occupancy, is high.

After that, the search device 100 provides a search result based on the determined ranking (step S206). In the example of FIG. 2, the search device 100 sets the image IM15 as the first place, the image IM19 as the second place, the image IM12 as the third place, and the image IM17 as the fourth place. It is provided to the terminal device 10 used by U1.

[5. Learning process]
Here, the learning process in the learning unit 132 of the search device 100 will be described with reference to FIGS. 9 and 10. 9 and 10 are diagrams showing an example of the learning process according to the embodiment.

First, it will be described with reference to FIG. In the example shown in FIG. 9, the search device 100 teaches a combination of the image IM21 including a cat and the information RO21 indicating the occupancy rate of the cat or the like in the image (hereinafter, may be referred to as “correct answer information RO21”). Acquire as data (step S21). In FIG. 9, for simplification of explanation, only the occupancy rates of the two classes of cat and background are shown in the correct answer information RO21, but the correct answer information RO21 corresponds to the class output by the learner LE. Occupancy may be included. For example, when the learner LE outputs information indicating the occupancy rate of 20 classes, the correct answer information RO21 includes information indicating the occupancy rate of other classes in addition to the occupancy rate of the two classes of cat and background. It may be. When the learner LE outputs information indicating the occupancy rate of classes such as dogs, airplanes, and bicycles in addition to the two classes corresponding to cats and backgrounds, the correct answer information RO21 includes the dog occupancy rate of "0" or Information indicating an airplane occupancy rate "0" or a bicycle occupancy rate "0" may be included.

Then, the image IM21 including the cat is input to the learning device LE (step S22). After that, information indicating the occupancy rate of the target as shown in the output information OC21-1 is output from the learner LE (step S23). In FIG. 9, for the sake of simplicity, only the occupancy rates of the two classes of the cat and the background are shown, but the learner LE may output information indicating the occupancy rates of the other classes. For example, when the learner LE outputs information indicating the occupancy rate for 20 classes, the learner LE outputs the occupancy rate for classes such as dogs, airplanes, and bicycles in addition to the two classes corresponding to cats and backgrounds. Information indicating that may be output.

As described above, for example, the learning unit 132 learns and generates the learning device LE by the technique of deep learning. For example, the learning unit 132 uses a combination of an image and an occupancy rate of a predetermined object in the image as teacher data. For example, the learning unit 132 corrects a parameter (connection coefficient) so that the error between the output in the learner LE and the occupancy rate of a predetermined object included in the teacher data is reduced (error back propagation method). The learner LE is learned by performing processing such as. For example, the learning unit 132 generates the learning device LE by performing processing such as backpropagation so as to minimize a predetermined error (loss) function.

For example, the learning unit 132 uses an error function L as shown in the following equation (1). As shown in the following equation (1), the learning unit 132 uses the cross entropy as an error function, for example, in the case of the N-class classification problem. The error function L may be any function as long as it represents the certainty of the identification result. For example, the error function L may be the entropy obtained from the discrimination probability. Further, for example, the error function L may be any function as long as it indicates the recognition accuracy of the learner LE.

Here, "x" in the above formula (1) and the following formulas (2) to (4) indicates an image. For example, in the example shown in FIG. 9, "x" in the above formula (1) and the following formulas (2) to (4) corresponds to the image IM. Further, 1 to N assigned to the variable "n" correspond to each class identified (classified) by the learner LE. For example, the learner LE corresponding to the above equation (1) indicates that it identifies N classes. For example, "cat", "background", etc. correspond to each class.

Further, "t _n (x)" in the above equation (1) and the following equations (3) and (4) indicates the occupancy rate of the object corresponding to the class n (any of 1 to N) in the image IM21. Shown. For example, "t _n (x)" in the above equation (1) indicates the occupancy rate of the object corresponding to the class n as shown in the correct answer information RO21. In this case, for example, when the target corresponding to class 1 is a "cat", "t ₁ (x)" becomes "0.53 (53%)". Further, for example, "t _n (x)" in the above equation (1) may be defined to take 1 only in the correct answer class and 0 in other cases.

Further, " _pn (x)" in the above equation (1) and the following equations (2) and (3) is a learner for an object corresponding to the class n (any of 1 to N) in the image IM21. The occupancy rate based on the output of LE is shown. For example, “ _pn (x)” in the above equation (1) indicates the occupancy rate of the object corresponding to the class n output by the learner LE as shown in the output information OC21-1. In this case, for example, when the target corresponding to class 1 is "cat", "p ₁ (x)" becomes "0.64 (64%)".

Further, " _pn (x)" in the above equation (1) is defined by the Softmax function as shown in the following equation (2) with a probability of class n with respect to x.

The function "f _n " in the above equation (2) is a class n score output by the CNN (learner LE). “Θ” is a parameter of CNN (learner LE). Further, the function "exp" is an exponential function. In this case, the gradient of the error function L (1) shown in the above equation (1) is calculated by the following equation (3).

As shown in the above equation (3), when _pn (x) = t _n (x) in all the classes from 1 to N, the gradient of the error function L (x) becomes 0 and becomes an extreme value. .. For example, the learning unit 132 performs feedback processing so that the gradient of the error function L (x) becomes 0 (step S24). For example, when the learning unit 132 repeats the above-described processing, the learning device LE can appropriately output the occupancy rate of the target in the input image. Note that FIG. 9 is for visually showing that the process is repeated so as to minimize the error function L and the like in order to bring the output of the learner LE closer to the correct answer information RO21. It may be done automatically.

As described above, “t _n (x)” is defined as the occupancy rate of class n when the entire image is 1, and is represented by, for example, the following equation (4).

"[Delta] _{i, j"} is the Kronecker delta, "j _p" is the label of a pixel p. For example, "j _p" is a label that corresponds to any one of a plurality of classes, such as cats and background. For example, the label for each pixel indicates which object the pixel is for. If the label of a pixel is the label corresponding to the cat, it indicates that the pixel is the pixel related to the target "cat". For example, the denominator of the above equation (4) is the total value of all pixels of the image "x". Further, for example, the molecule of the above formula (4) has the number of pixels labeled with class n in the image “x”. As described above, "t _n (x)" in the above formula (4) has a value of 0 to 1. The occupancy rate of the target corresponding to each class in the image is calculated by the above formula (4). The labeling of each pixel may be performed by, for example, the administrator of the search device 100, the owner of the image, or the like.

Next, it will be described with reference to FIG. FIG. 10 shows a case where the search device 100 learns a learning device LE 31 different from the learning device LE. In the example shown in FIG. 10, the search device 100 includes an image IM 31 including a person or a bottle and an information RO 31 indicating the occupancy rate of the person or the bottle in the image (hereinafter, may be referred to as “correct answer information RO 31”). Is acquired as teacher data (step S31). Then, the image IM31 including the person or the bottle is input to the learning device LE31 (step S32). After that, information indicating the occupancy rate of the target as shown in the output information OC31-1 is output from the learner LE31 (step S33). In the example shown in FIG. 10, similarly to the example shown in FIG. 9, the learning unit 132 uses the above equations (1) to (3) so that the gradient of the error function L (x) becomes 0. Feedback processing is performed (step S34). For example, by repeating the above-mentioned processing by the learning unit 132, the learning device LE31 can appropriately output the occupancy rate of the target in the input image even when there are a plurality of targets. Note that FIG. 10 is for visually demonstrating that the process is repeated so as to minimize the error function L and the like in order to bring the output of the learner LE31 closer to the correct answer information RO31. It may be done automatically.

[6. Learning process flow]
Here, the procedure of the learning process by the search device 100 according to the embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing an example of the learning process according to the embodiment.

As shown in FIG. 11, the search device 100 acquires the image and the correct answer information regarding the occupancy rate of each object in the image (step S301). In FIG. 9, the search device 100 acquires the image IM21 including the cat and the correct answer information RO21 indicating the occupancy rate of the cat or the like in the image. After that, the search device 100 inputs the image acquired in step S301 into the learner (step S302). In FIG. 9, the search device 100 inputs the acquired image IM21 to the learner LE.

After that, the search device 100 learns so that the error between the occupancy rate based on the output of the learner and the occupancy rate of the correct answer information becomes small (step S303). In FIG. 9, the search device 100 learns based on the occupancy rate of each object shown in the output information OC21-1 based on the output of the learning device LE and the occupancy rate of each object shown in the correct answer information RO21.

After that, when the predetermined condition is satisfied (step S304: Yes), the search device 100 ends the process. For example, the search device 100 determines when the error between the occupancy rate based on the output of the learning device and the occupancy rate of the correct answer information is within a predetermined threshold value, or when a predetermined time elapses from the start of learning. If the condition is satisfied, the process may be terminated. Further, when the search device 100 does not satisfy the predetermined condition (step S304: No), the search device 100 repeats the process of step S303. For example, the search device 100 determines when the error between the occupancy rate based on the output of the learning device and the occupancy rate of the correct answer information is larger than a predetermined threshold value, or when a predetermined time has not elapsed since the start of learning. The process of step S303 may be repeated on the assumption that the condition of the above is not satisfied. The above learning process is an example, and the search device 100 may perform learning by various procedures.

[7. effect〕
As described above, the search device 100 according to the embodiment includes an acquisition unit 131 and an estimation unit 133. The acquisition unit 131 acquires an image. The estimation unit 133 is a learner (“learner LE” in the embodiment) that outputs the occupancy rate of a predetermined object (“cat” in the embodiment; the same applies hereinafter) in the input image according to the input image. The occupancy rate of a predetermined target in the image acquired by the acquisition unit 131 is estimated based on (the same applies hereinafter) and the image acquired by the acquisition unit 131.

As a result, the search device 100 according to the embodiment appropriately determines the mode in which the object is included in the image, that is, how much the object occupies in the image, by using the learning device that outputs the occupancy rate of the object in the image. Can be estimated. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, in the search device 100 according to the embodiment, the estimation unit 133 estimates the occupancy rate of a predetermined target in the image acquired by the acquisition unit 131 based on the learner which is a neural network learned from the predetermined data. To do.

As a result, the search device 100 according to the embodiment can appropriately estimate how much the target occupies in the image based on the learner which is a neural network learned from the predetermined data. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, in the search device 100 according to the embodiment, the estimation unit 133 is acquired by the acquisition unit 131 based on the learner which is a neural network learned by the combination of the image and the occupancy rate of a predetermined target in the image. Estimate the occupancy rate of a predetermined object in the image.

As a result, the search device 100 according to the embodiment is based on a learner which is a neural network learned by a combination of an image and an occupancy rate of a predetermined target in the image, so that the ratio of the target in the image is increased. It is possible to properly estimate whether it occupies. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, in the search device 100 according to the embodiment, the estimation unit 133 estimates the occupancy rate of a predetermined target in the image acquired by the acquisition unit 131 based on the learner which is a neural network that performs the convolution process and the pooling process. To do.

As a result, the search device 100 according to the embodiment can appropriately estimate how much the target occupies in the image by being based on the learner which is a neural network that performs the convolution process and the pooling process. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, the search device 100 according to the embodiment has a learning unit 132. The learning unit 132 learns the learning device.

As a result, the search device 100 according to the embodiment can appropriately estimate how much the target occupies in the image based on the learned learner. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, the search device 100 according to the embodiment has a learning unit 132. The learning unit 132 learns the learning device by combining the image and the occupancy rate of a predetermined object in the image.

As a result, the search device 100 according to the embodiment appropriately estimates how much the target occupies in the image based on the learning device learned by the combination of the image and the occupancy rate of the predetermined target in the image. can do. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, the search device 100 according to the embodiment has a learning unit 132. The learning unit 132 learns the learner so as to minimize a predetermined evaluation function.

As a result, the search device 100 according to the embodiment can appropriately estimate how much the target occupies in the image by being based on the learning device learned so as to minimize the predetermined evaluation function. In this way, the search device 100 can appropriately estimate the mode in which the object is included in the image.

Further, the search device 100 according to the embodiment has a determination unit 135. The determination unit 135 determines the ranking of the plurality of images based on the occupancy rate of a predetermined target in each of the plurality of images estimated by the estimation unit 134.

As a result, the search device 100 according to the embodiment appropriately ranks the plurality of images by determining the ranking of the plurality of images based on the occupancy rate of the predetermined target in each of the estimated plurality of images. Can be decided.

Further, the search device 100 according to the embodiment has a providing unit 136. The providing unit 136 provides information based on the ranking of a plurality of images determined by the determining unit 135.

Thereby, the search device 100 according to the embodiment can provide more appropriate information to the user by providing information based on the ranking of a plurality of images determined by using the occupancy rate of the target.

[8. Hardware configuration]
The search device 100 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the function of the search device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD (Hard Disk Drive) 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program that depends on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via the network N and sends it to the CPU 1100, and provides the data generated by the CPU 1100 to the other device via the network N.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the search device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the function of the control unit 130 by executing the program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be acquired from another device via the network N.

The embodiments of the present application have been described in detail with reference to the drawings, but these are examples, and various modifications and improvements are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure line of the invention. It is possible to carry out the present invention in other forms described above.

[9. Others]
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically distributed / physically in any unit according to various loads and usage conditions. It can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

In addition, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

1 Search system 100 Search device (estimator)
121 Learning information storage unit 122 Image information storage unit 123 List information storage unit 130 Control unit 131 Acquisition unit 132 Learning unit 133 Estimating unit 134 Extraction unit 135 Decision unit 136 Providing unit 10 Terminal device N network

Claims (13)

The acquisition department that acquires images and
Based on the learner that outputs the occupancy rate of a predetermined object in the input image according to the input image and the image acquired by the acquisition unit, the predetermined object in the image acquired by the acquisition unit An estimation unit that estimates the occupancy rate and
Using the occupancy rate of the predetermined object in each of the plurality of images estimated by the estimation unit , the ranking of the plurality of images is determined according to the difference in the occupancy rate between the images having consecutive rankings in the ranking. The decision-making part to do
An estimation device characterized by comprising. The acquisition department that acquires images and
Based on the learner that outputs the occupancy rate of a predetermined object in the input image according to the input image and the image acquired by the acquisition unit, the predetermined object in the image acquired by the acquisition unit An estimation unit that estimates the occupancy rate and
Based on the occupancy rate of the predetermined object in each of the plurality of images estimated by the estimation unit, the ranking of the plurality of images is ranked according to whether or not each of the plurality of images contains character information. The decision-making part to decide and
An estimation device characterized by comprising. The estimation unit
The first or second claim is characterized in that the occupancy rate of the predetermined object in the image acquired by the acquisition unit is estimated based on the learner which is a neural network trained by the predetermined data. The estimated device described. The estimation unit
To estimate the occupancy rate of the predetermined object in the image acquired by the acquisition unit based on the learner which is a neural network learned by the combination of the image and the occupancy rate of the predetermined object in the image. The estimation device according to claim 3. The estimation unit
The estimation according to claim 4, wherein the occupancy rate of the predetermined object in the image acquired by the acquisition unit is estimated based on the learner which is the neural network that performs the convolution process and the pooling process. apparatus. Learning unit that learns the learning device,
The estimation device according to any one of claims 1 to 5, further comprising. The learning unit
The estimation device according to claim 6, wherein the learning device is learned by a combination of an image and an occupancy rate of a predetermined object in the image. The learning unit
The estimation device according to claim 6 or 7, wherein the learner is learned so as to minimize a predetermined evaluation function. A providing unit that provides information based on the ranking of the plurality of images determined by the determining unit.
The estimation device according to any one of claims 1 to 8, further comprising. An estimation method performed by a computer
The acquisition process to acquire the image and
Based on the learner that outputs the occupancy rate of a predetermined object in the input image according to the input image and the image acquired by the acquisition step, the predetermined object in the image acquired by the acquisition step The estimation process for estimating the occupancy rate and
Using the occupancy rate of the predetermined object in each of the plurality of images estimated by the estimation step , the ranking of the plurality of images is determined according to the difference in the occupancy rate between the images having consecutive rankings in the ranking. The decision process to do and
An estimation method characterized by including. The acquisition procedure to acquire the image and
Based on the learner that outputs the occupancy rate of the predetermined object in the input image according to the input image and the image acquired by the acquisition procedure, the predetermined object in the image acquired by the acquisition procedure Estimating procedure for estimating occupancy and
Using the occupancy rate of the predetermined object in each of the plurality of images estimated by the estimation procedure , the ranking of the plurality of images is determined according to the difference in the occupancy rate between the images having consecutive ranks in the ranking. The decision procedure to make and
An estimation program characterized by having a computer execute. An estimation method performed by a computer
The acquisition process to acquire the image and
Based on the learner that outputs the occupancy rate of a predetermined object in the input image according to the input image and the image acquired by the acquisition step, the predetermined object in the image acquired by the acquisition step The estimation process for estimating the occupancy rate and
Based on the occupancy rate of the predetermined object in each of the plurality of images estimated by the estimation step, the ranking of the plurality of images is ranked according to whether or not each of the plurality of images contains character information. The decision process to decide and
An estimation method characterized by including. The acquisition procedure to acquire the image and
Based on the learner that outputs the occupancy rate of the predetermined object in the input image according to the input image and the image acquired by the acquisition procedure, the predetermined object in the image acquired by the acquisition procedure Estimating procedure for estimating occupancy and
Based on the occupancy rate of the predetermined object in each of the plurality of images estimated by the estimation procedure, the ranking of the plurality of images is ranked according to whether or not each of the plurality of images contains character information. The decision procedure to decide and
An estimation program characterized by having a computer execute.

Images