JP2023130409A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device, a method, and a program that can appropriately extract an image containing an object.SOLUTION: In an image extraction system, an information processing device 1 comprises: a search unit 10a that searches for images related to an object by using text via a network; an extraction unit 10b that extracts a feature value of each image found; a classification unit 10c that clusters the feature value of each image; an image extraction unit 10d that extracts an image according to the number of images constructing a cluster of feature values; and a detection unit 10f that detects an object contained in the image.SELECTED DRAWING: Figure 15

Description

The present invention relates to an information processing device, an information processing method, and a program.

There is a technology based on image recognition technology that extracts a desired image from a large number of images. For example, Patent Document 1 discloses a photo ordering system that accesses a server via the Internet from an information processing terminal such as a tablet terminal and orders a desired photo. In this photo ordering system, when a user registers a specific person in advance, a photo in which the specific person appears is extracted from a plurality of photos obtained from the server and displayed on the user terminal.

Japanese Patent Application Publication No. 2013-161394

However, in the invention according to Patent Document 1, there is a risk that a photograph of a person whose characteristics are similar to a specific person may also be extracted.

One aspect of the present invention is to provide an information processing device or the like that can appropriately extract an image including a target object.

An information processing device according to one aspect includes a search unit that uses text to search for images related to a target object via a network, an extraction unit that extracts feature amounts of each searched image, and a search unit that searches for images related to an object via a network using text; comprising: a classification unit that clusters feature quantities; an image extraction unit that extracts the images according to the number of images forming a cluster of the feature quantities; and a detection unit that detects objects included in the images; The image extracting unit extracts the image according to the type of the detected object.

In one aspect, it becomes possible to appropriately extract an image including a target object.

FIG. 1 is an explanatory diagram showing an overview of an image extraction system. FIG. 2 is a block diagram showing a configuration example of a server. FIG. 2 is an explanatory diagram showing an example of a record layout of a search history DB. FIG. 2 is an explanatory diagram showing an example of a record layout of an extracted image DB. FIG. 3 is an explanatory diagram illustrating an operation of extracting image feature amounts. FIG. 3 is an explanatory diagram showing clustering processing based on image feature amounts. 12 is a flowchart showing a processing procedure for extracting a desired image by cluster classification based on image feature amounts. FIG. 2 is a block diagram illustrating a configuration example of a server according to a second embodiment. FIG. 2 is an explanatory diagram showing an example of a record layout of a frequency aggregation DB. FIG. 6 is an explanatory diagram illustrating an operation of extracting a desired image according to the appearance frequency of each type of object. 12 is a flowchart showing a processing procedure for extracting a desired image according to the appearance frequency of each type of object. FIG. 3 is a block diagram illustrating a configuration example of a server according to a third embodiment. It is an explanatory view showing an example of a record layout of a thesaurus dictionary. 12 is a flowchart showing a processing procedure when extracting a desired image using synonyms. It is a functional block diagram showing operation of the server of the form mentioned above.

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below based on drawings showing embodiments thereof.

(Embodiment 1)
Embodiment 1 relates to a mode in which a desired image is extracted from a large number of images by cluster classification based on image feature amounts. FIG. 1 is an explanatory diagram showing an overview of an image extraction system. The system of this embodiment includes an information processing device 1 and an information sharing server 2, and each device transmits and receives information via a network N such as the Internet.

The information processing device 1 is an information processing device that processes, stores, transmits and receives various information. The information processing device 1 is, for example, a server device, a personal computer, or the like. In this embodiment, the information processing device 1 is assumed to be a server device, and will be read as server 1 below for brevity.

The information sharing server 2 is a server device that manages an SNS (Social Networking Service) or a server device that functions as a web search engine. In this embodiment, the information sharing server 2 is assumed to be a server device that manages SNS, and will be read as SNS server 2 below for brevity. The SNS server 2 is an information processing device that processes, stores, transmits and receives various information, such as registering and managing users and managing text information or images posted by users.

The server 1 according to the present embodiment collects images that include a certain object as a subject from images included in each posted article uploaded by an unspecified number of people to the SNS server 2 via the network N. For example, assuming application to marketing, the server 1 performs a search using a product name (name of an object) as a search query, and collects images of a certain product.

On the other hand, even when performing an image search using a product name as a search query, there is a risk that images of subjects unrelated to the product may be retrieved, including synonyms or search text that have the same expression as the product name. In this case, it is conceivable to perform image recognition using deep learning, pattern matching, etc., to recognize the target product from the image, and extract the desired image.

However, it is difficult to recognize and extract only specific objects (individuals) from a group of images collected from SNS. For example, when extracting an image of a specific "dog" from a group of unspecified large numbers of images, even if it is easy to extract images that include dog-like subjects, the image of a specific dog (individual) is extracted from among them. It's not easy to do. In this case, for example, a model that has learned only the characteristics of the relevant dog using deep learning or the like, or a pattern matching model (rule) for recognizing the relevant dog must be prepared in advance, which is difficult.

Therefore, in this embodiment, clustering is performed on a group of images searched from SNS, and from the clustering results, images that are estimated to include the target product are estimated and extracted. Specifically, the server 1 extracts the feature amount of each image using a model constructed by machine learning as described later, and divides the extracted feature amount into a plurality of clusters. Note that details of the feature amount extraction process will be described later.

Among the feature amounts divided into each cluster, the server 1 counts a plurality of feature amounts of the same original image in the same cluster as one and totals the number of images. Based on the total number of images in each cluster, the server 1 extracts a group of images classified into any cluster as a group of images containing a target object. For example, the server 1 extracts a group of images classified into a cluster with the largest number of images.

As described above, the server 1 performs an image search using the name of the object (product name) as a search query. Therefore, among the searched images, it is estimated that there are more images that include the target object as a subject than images that include other objects. Therefore, in this embodiment, clustering is performed as described above, and a group of images belonging to the cluster with the largest number of images is extracted. As a result, a group of images that are estimated to include the target object are extracted without recognizing the target object itself.

FIG. 2 is a block diagram showing an example of the configuration of the server 1. As shown in FIG. The server 1 includes a control section 11 , a storage section 12 , a communication section 13 , an input section 14 , a display section 15 , a reading section 16 , and a large-capacity storage section 17 . Each configuration is connected by bus B.

The control unit 11 includes arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and reads and executes the control program 1P stored in the storage unit 12. , performs various information processing, control processing, etc. related to the server 1. Note that although the control unit 11 is described as being a single processor in FIG. 2, it may be a multiprocessor.

The storage unit 12 includes memory elements such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and stores a control program 1P or data necessary for the control unit 11 to execute processing. Further, the storage unit 12 temporarily stores data and the like necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and sends and receives information to and from the SNS server 2 via the network N.

The input unit 14 is an input device such as a mouse, a keyboard, a touch panel, a button, etc., and outputs the received operation information to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11.

The reading unit 16 reads a portable storage medium 1a including a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the mass storage unit 17. Alternatively, the control unit 11 may download the control program 1P from another computer via the network N or the like and store it in the mass storage unit 17. Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

The large-capacity storage unit 17 is a large-capacity storage device including, for example, a hard disk. The large-capacity storage unit 17 includes a search history DB 171, an extracted image DB 172, and an object detection model 173. The search history DB 171 stores information on images related to objects searched from the SNS server 2. The extracted image DB 172 stores information regarding images extracted as containing objects among the searched images. The object detection model 173 is a detector that detects objects in images, and is a trained model generated by machine learning.

Note that in this embodiment, the storage section 12 and the large-capacity storage section 17 may be configured as an integrated storage device. Furthermore, the large-capacity storage section 17 may be configured with a plurality of storage devices. Furthermore, the large-capacity storage section 17 may be an external storage device connected to the server 1.

In this embodiment, the server 1 will be described as one information processing device, but the server 1 may be distributed and processed by a plurality of devices, or may be configured by a virtual machine.

FIG. 3 is an explanatory diagram showing an example of the record layout of the search history DB 171. The search history DB 171 includes a search ID string, a search query string, a search date and time string, and a search image string. The search ID column stores IDs of search history data that are uniquely specified in order to identify the history data of each search. The search query string stores a search query used when searching for an object. A search query is a word used for a keyword search, a hashtag search, etc., for example. The search date and time column stores information on the date and time when images related to the object were searched. The search image string stores images related to the searched object.

FIG. 4 is an explanatory diagram showing an example of the record layout of the extracted image DB 172. The extracted image DB 172 includes a search ID string and an extracted image string. The search ID column stores IDs of historical data that have been searched for images related to the target object. The extracted image sequence stores images to be collected after removing images that are not to be collected.

FIG. 5 is an explanatory diagram illustrating the operation of extracting image feature amounts. The control unit 11 of the server 1 receives a search query representing a target object (for example, a unique name of the target object) through the input unit 14 . Based on the received search query, the control unit 11 causes the communication unit 13 to select a group of images related to the object from images included in each posted article uploaded by an unspecified number of people to the SNS server 2 via the network N. search for.

The control unit 11 stores a group of images related to the searched object in the search history DB 171 of the mass storage unit 17. Specifically, the control unit 11 assigns a search ID and stores the search query, search date and time, and a group of images related to the searched object as one record in the search history DB 171.

For each searched image, the control unit 11 extracts the feature amount of the image region corresponding to each subject (object) included in the image. For example, the control unit 11 is an object detection model 173 that detects an object in an image, and extracts image features using a part of the object detection model 173 constructed by deep learning.

The control unit 11 of the server 1 constructs (generates) the object detection model 173 by performing deep learning to learn the feature amount of the teacher image using predetermined teacher data. For example, the object detection model 173 is a CNN (Convolution Neural Network), which includes an input layer that receives image input, an output layer that outputs the detection results of detecting objects in the image, and features of the image area corresponding to each object. and an intermediate layer that extracts the amount.

The input layer has a plurality of neurons that accept input of pixel values of pixels of each image included in the group of images related to the searched object, and passes the input pixel values to the intermediate layer. The intermediate layer has a plurality of neurons that extract image features, and passes the extracted features of the image region to the output layer. For example, when the object detection model 173 is a CNN, the intermediate layer alternates between a convolution layer that convolves the pixel value of each pixel input from the input layer and a pooling layer that maps the pixel values convolved with the convolution layer. It has a configuration connected to. The intermediate layer compresses the pixel information of the region and finally extracts the image feature amount. The output layer outputs a detection result of detecting an object in the image based on the image feature amount output from the intermediate layer.

For example, the control unit 11 constructs a neural network such as R-CNN (Regins with CNN), which is a type of CNN, semantic segmentation, YOLO (You Look Only Once), SSD (Single Shot MultiBox Detector), etc. as the object detection model 173. It has been done. R-CNN, semantic segmentation, etc. are all neural networks that identify the image region of each object included in an image and identify what each object is for each identified image region. As shown in FIG. 5, when the object detection model 173 receives an image input in the input layer, the intermediate layer identifies the image region corresponding to each object in the image and extracts the feature amount of each region. , the output layer outputs an identification result that identifies the object included in each image region.

However, as already mentioned, it is difficult to construct an object detection model 173 that can detect a specific object (individual), and the object detection model 173 only detects the general name (type) of an object in an image. . Therefore, in this embodiment, instead of directly detecting the target object using the object detection model 173, the output layer is removed from the object detection model 173 and only the input layer and intermediate layer are used. is used as an extraction model for extracting the feature amount of the image region corresponding to each object (see the lower side of FIG. 5). The control unit 11 uses the object detection model 173 (input layer and intermediate layer thereof) to extract feature amounts of image regions corresponding to each object included in the image from each image retrieved from the SNS.

Note that in this embodiment, the control unit 11 extracts the image feature amount using the object detection model 173 constructed by deep learning. Even if features are extracted using local feature extraction methods such as SIFT (Scale Invariant Feature Transform), SURF (Speed-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), and HOG (Histograms of Oriented Gradients), good. That is, the control unit 11 only needs to be able to extract the feature amount of each searched image, and the extraction method is not particularly limited.

FIG. 6 is an explanatory diagram showing clustering processing based on image feature amounts. The control unit 11 of the server 1 performs cluster classification (clustering) according to the extracted feature amount of each image region. Regarding the clustering process, for example, the K-means method (k-means method) or the X-means method may be used. The K-means method is a non-hierarchical clustering algorithm that classifies into a predetermined number of clusters, "k". The X-means method is a modification of the K-means method, and is an algorithm that automatically estimates the optimal number of clusters "k". The control unit 11 performs this for each image area corresponding to each object, and classifies each image area into one of the clusters.

The control unit 11 totals the number of image areas classified into each cluster. Then, the control unit 11 selects one of the plurality of clusters based on the total number of image areas of each cluster. For example, the control unit 11 may select the cluster with the largest number of images. The control unit 11 extracts a group of images having image regions classified into the selected cluster.

FIG. 6 illustrates how images A, B, C, and D are clustered. For example, the control unit 11 of the server 1 performs cluster classification by the above-described clustering process, and classifies them into "cluster 1," "cluster 2," and "cluster 3." The control unit 11 totals the number of image regions classified into each of "cluster 1," "cluster 2," and "cluster 3," and selects a cluster with a large number of total image regions. For example, if cluster 1 is a cluster with a large number of image areas classified into each cluster, the control unit 11 estimates that the feature amount of the image area for the object is included in cluster 1, and selects cluster 1. Select. Thereby, the control unit 11 extracts a group of images (image A, image B, and image C) having image regions classified into the selected cluster 1.

The control unit 11 stores the extracted image group of the target object in the extracted image DB 172 of the mass storage unit 17 in association with the search ID. Further, the control unit 11 generates training data for learning the image of the target object by labeling the image group of the target object extracted above with target object information. Specifically, the control unit 11 assigns a unique name (for example, a product name) of the object to an image region classified into the cluster selected above (“cluster 1” in the above example) in the extracted image. Generate associated training data. Thereby, for example, the control unit 11 can construct a model capable of detecting a specific target object (individual) by relearning the object detection model 173 using the teacher data.

FIG. 7 is a flowchart showing a processing procedure for extracting a desired image by cluster classification based on image feature amounts. The control unit 11 of the server 1 receives a search query representing an object (for example, a unique name of the object) through the input unit 14 (step S101). Based on the received search query, the control unit 11 causes the communication unit 13 to select a group of images related to the object from images included in each posted article uploaded by an unspecified number of people to the SNS server 2 via the network N. Search (step S102). The control unit 11 stores a group of images related to the searched object in the search history DB 171 of the large-capacity storage unit 17 (step S103).

For each image related to the searched object, the control unit 11 extracts the feature amount of the image region corresponding to each object included in the image (step S104). The control unit 11 performs cluster classification according to the extracted feature amount of each image region (step S105). The control unit 11 totals the number of image regions classified into each cluster (step S106). The control unit 11 selects one of the plurality of clusters based on the total number of image regions classified into each cluster (step S107). For example, the control unit 11 may select the cluster with the largest number of image areas. The control unit 11 extracts a group of images having image regions classified into the selected cluster (step S108).

The control unit 11 stores the extracted image group in the extracted image DB 172 of the mass storage unit 17 in association with the search ID (step S109). The control unit 11 generates, for the image group extracted in step S108, teacher data that associates the image region selected in step S107 with object information (step S110).

According to the present embodiment, it is possible to extract an image of a target object by removing images that are not to be collected by using clustering processing on image feature amounts. Therefore, since images unrelated to the target object are excluded, it is possible to improve the accuracy of extraction.

(Embodiment 2)
In the second embodiment, the object detection model 173 detects the type of object (common name) in an image, and extracts a desired image based on the frequency of appearance of each type of object in the entire group of images retrieved from SNS. Concerning form. Note that descriptions of contents that overlap with those of Embodiment 1 will be omitted.

FIG. 8 is a block diagram showing a configuration example of the server 1 according to the second embodiment. Note that the same reference numerals are given to the same contents as those in FIG. 2, and the description thereof will be omitted. The large-capacity storage unit 17 includes a frequency aggregation DB 174. The frequency aggregation DB 174 stores the frequency of appearance of each type of object detected from a plurality of images.

FIG. 9 is an explanatory diagram showing an example of the record layout of the frequency aggregation DB 174. The frequency aggregation DB 174 includes a search ID column, a type column, and a frequency column. The search ID column stores IDs of historical data for which images have been searched. The type column stores the types of objects detected in the image. The frequency column stores the number of times each type of object appears.

FIG. 10 is an explanatory diagram illustrating the operation of extracting a desired image according to the appearance frequency of each type of object. An overview of this embodiment will be explained based on FIG. 10.

In this embodiment, the control unit 11 of the server 1 does not remove the output layer from the object detection model 173,
Function as a model for original object detection. The control unit 11 inputs the image group acquired from the SNS server 2 to the object detection model 173, and acquires from the object detection model 173 a detection result indicating the common name of the object included in each image, for example, the type of the object. As a result, as shown in FIG. 10, the result of identifying the type of object for each image area of each object is output.

The control unit 11 totals the appearance frequency of each type of object in the entire image group acquired from the SNS server 2 based on the detection result of each image. The control unit 11 associates the total appearance frequency of each type of object with a search ID, and stores the search ID, type, and appearance frequency as one record in the frequency total DB 174. The control unit 11 determines the type of object with the highest frequency, and acquires a group of images including the determined type of object as a group of images including the target object.

FIG. 11 is a flowchart showing a processing procedure for extracting a desired image according to the appearance frequency of each type of object. Note that the same reference numerals are given to the same contents as those in FIG. 7, and the description thereof will be omitted. The control unit 11 uses the object detection model 173 to detect an object in each image from a group of images related to the target object retrieved from the SNS server 2 (step S131). The control unit 11 totalizes the appearance frequency of each object type according to the object type included in each detected image (step S132). The control unit 11 stores the aggregated appearance frequency of each type in the frequency aggregation DB 174 in association with the search ID (step S133). The control unit 11 acquires the type of object with the highest frequency, and based on the acquired type, extracts a group of images associated with the type output from the object detection model 173 (step S134). The control unit 11 stores the extracted image group in the extracted image DB 172 of the mass storage unit 17 (step S135).

According to this embodiment, the appearance frequency of each type of object is tallied, and images are extracted according to the tallied frequency. As a result, even if an object different from the target object is mistakenly classified into the largest cluster due to the processing in Embodiment 1, for example, incorrect extraction can be prevented and the accuracy of image extraction can be improved. becomes.

(Embodiment 3)
The third embodiment relates to a form in which a desired image is extracted by determining whether the common name of an object detected by the object detection model 173 and the name of the target object are similar. Note that descriptions of contents that overlap with those of Embodiment 2 will be omitted.

The server 1 extracts synonyms of a search query (for example, the name of an object) when searching for a group of images related to an object, and if the degree of similarity of the extracted synonyms is equal to or higher than a predetermined threshold, the server 1 extracts synonyms from a search query (for example, the name of an object), and if the similarity of the extracted synonyms is equal to or higher than a predetermined threshold, the Extract. Regarding the synonym extraction process, extraction may be performed using a thesaurus dictionary or vectorized data such as Word2Vec. Note that this embodiment will be described using an example using a thesaurus dictionary.

FIG. 12 is a block diagram showing a configuration example of the server 1 according to the third embodiment. Note that the same reference numerals are given to the same content as in FIG. 8, and the explanation thereof will be omitted. The mass storage unit 17 includes a thesaurus dictionary 175. The thesaurus dictionary 175 is a dictionary that compiles words with similar meanings.

FIG. 13 is an explanatory diagram showing an example of the record layout of the thesaurus dictionary 175. Thesaurus dictionary 175 includes a text string and a similarity string. The text string stores text corresponding to the name (common name or proper name) of the object. The similarity column stores the similarity of each text.

FIG. 14 is a flowchart showing the processing procedure when extracting a desired image using synonyms. Note that the same reference numerals are given to the same contents as those in FIG. 11, and the description thereof will be omitted. The control unit 11 acquires, from the thesaurus 175, the degree of similarity between the name of the object type detected in step S131 and the search query (object name) used when searching for a group of images related to the object (step S141). . The control unit 11 extracts images including the object from the image group searched in step S102 according to the obtained similarity (step S142). For example, when the control unit 11 determines that the degree of similarity between the acquired name of each object type and the name of the target object is greater than or equal to a predetermined threshold, the control unit 11 extracts an image that includes the object.

According to this embodiment, by extracting an image based on the degree of similarity between the name of the object type in the detected image and the name of the target object, it is possible to improve the accuracy of image extraction.

(Embodiment 4)
FIG. 15 is a functional block diagram showing the operation of the server 1 of the above-described form. When the control unit 11 executes the control program 1P, the server 1 operates as follows.

The search unit 10a searches through the network N for a group of images related to the object. The extraction unit 10b extracts the feature amount of each searched image. The classification unit 10c classifies each image into one of a plurality of clusters based on the image feature amount. The image extraction unit 10d extracts an image having image regions classified into clusters. The generation unit 10e generates, for the image extracted by the extraction unit 10b, data in which object information is associated with the image area. The detection unit 10f detects an object included in the image.

This Embodiment 4 is as described above, and other aspects are the same as Embodiments 1 to 3, so corresponding parts are given the same reference numerals and detailed explanation thereof will be omitted.

The embodiments disclosed herein are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the claims rather than the above-mentioned meaning, and is intended to include meanings equivalent to the claims and all changes within the scope.

1 Information processing device (server)
11 Control section 12 Storage section 13 Communication section 14 Input section 15 Display section 16 Reading section 17 Mass storage section 171 Search history DB
172 Extracted image DB
173 Object detection model 174 Frequency aggregation DB
175 Thesaurus 1a Portable storage medium 1b Semiconductor memory 1P Control program 2 Information sharing server (SNS server)
10a Search section 10b Extraction section 10c Classification section 10d Image extraction section 10e Generation section 10f Detection section

Claims (7)

a search unit that uses text to search for images related to the object via the network;
an extraction unit that extracts feature amounts of each searched image;
a classification unit that clusters the features of each image;
an image extraction unit that extracts the image according to the number of images forming the cluster of the feature amount;
a detection unit that detects an object included in the image,
The information processing device is characterized in that the image extraction unit extracts the image according to the type of the detected object. The extraction unit extracts the feature amount of each image region in the image,
The classification unit performs clustering for each image region,
The image extraction unit includes:
Selecting one of the plurality of clusters according to the number of the image regions forming the cluster of the feature amount,
The information processing apparatus according to claim 1, wherein the image having the image area constituting the selected cluster is extracted. The information processing device according to claim 1 or 2, wherein the image extraction unit extracts the image according to the appearance frequency of each type of the object in all the images searched by the search unit. The detection unit specifies the type name of the object,
The information processing according to any one of claims 1 to 3, wherein the image extraction unit extracts the image according to the degree of similarity between the name of the object and the name of the type of the object. Device. Search for images related to the object through the network,
Extract the features of each searched image,
Clustering each of the images based on the feature amount,
Extracting the images according to the number of images forming the cluster of the feature amount,
detecting an object included in the image;
An information processing method that causes a computer to execute a process of extracting the image according to the type of the detected object. to the computer,
Search for images related to the object through the network,
Extract the features of each searched image,
Clustering each of the images based on the feature amount,
Extracting the images according to the number of images forming the cluster of the feature amount,
detecting an object included in the image;
A program that executes a process of extracting the image according to the type of the detected object. a search unit that uses text to search for images related to the object via the network;
an extraction unit that extracts feature amounts of each searched image;
a classification unit that clusters the features of each image;
an image extraction unit that extracts the image according to the number of images forming the cluster of the feature amount;
a detection unit that detects an object included in the image,
The detection unit specifies the type name of the object,
The information processing device is characterized in that the image extraction unit extracts the image according to the degree of similarity between the name of the object and the name of the type of the object.

Images