JP2018073429A - Retrieval device, retrieval method, and retrieval program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a retrieval device, a retrieval method and a retrieval program which improve image retrieval accuracy.SOLUTION: A retrieval device comprises: a generation unit for generating a distributed expression space in which a distributed expression of predetermined meta information only is projected, and a distributed expression of first meta information which represents characteristics of first information; a specification unit for specifying one or more pieces of meta information having the distributed expression projected in the distributed expression space similar to the distributed expression of the first information, from among pieces of the predetermined meta information; and a retrieval unit for comparing second meta information representing characteristics of second information with the meta information specified by the specification unit and extracting second information with a higher degree of coincidence between second meta information and the meta information specified by the specification unit than a predetermined threshold, from among pieces of the second information to be a retrieval object to which one or more pieces of the second meta information correspond, so as to retrieve second information similar to the first information on the basis of the comparison result of the extracted second information and the first information.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a search device, a search method, and a search program.

  Conventionally, a technique for searching for an image corresponding to an input search query is known. As an example of such a technique, an image with a tag corresponding to the search query is searched in advance, and an image corresponding to the search query is searched by comparing feature quantities such as color and shape of the inspected image. The technology to do is known.

JP 2011-210000 A

  However, the above method has not always improved the search accuracy of the image corresponding to the search query.

  For example, in the above-described conventional technology, a search using a character string such as a tag and a search using an image feature amount are performed independently. There is a possibility that an image with a tag or an image different from the search intention of the user, but an image having similar feature quantities such as color and shape may be included in the search result.

  The present application has been made in view of the above, and an object thereof is to improve the image search accuracy.

  The search device according to the present application includes a generation unit that generates a distributed expression space in which only a distributed expression of predetermined meta information is projected, a distributed expression of first meta information indicating characteristics of the first information, and the predetermined meta information. A specifying unit for specifying one or more meta-information whose distributed representation projected on the distributed representation space is similar to the distributed representation of the first information, and second meta-information indicating the characteristics of the second information; The meta information specified by the specifying unit is compared, and the second meta information is specified by the specifying unit from the second information to be searched in which one or more second meta information is associated. Second information whose degree of coincidence with the meta information is higher than a predetermined threshold is extracted, and second information similar to the first information is extracted based on a comparison result between the extracted second information and the first information. And a search unit for searching.

  According to one aspect of the embodiment, the image search accuracy is improved.

FIG. 1 is a diagram illustrating an example of an extraction process executed by the search device according to the embodiment. FIG. 2 is a diagram illustrating a configuration example of the search device according to the embodiment. FIG. 3 is a diagram illustrating an example of information registered in the generation model database according to the embodiment. FIG. 4 is a diagram illustrating an example of information registered in the distributed representation database according to the embodiment. FIG. 5 is a diagram illustrating an example of information registered in the image database according to the embodiment. FIG. 6 is a diagram illustrating an example of processing for collecting search target data. FIG. 7 is a diagram showing a variation of processing for collecting search target data. FIG. 8 is a flowchart illustrating an example of a flow of search processing executed by the search device according to the embodiment. FIG. 9 is a diagram illustrating an example of a hardware configuration.

  Hereinafter, modes for carrying out a search device, a search method, and a search program according to the present application (hereinafter referred to as “embodiments”) will be described in detail with reference to the drawings. Note that the search device, the search method, and the search program according to the present application are not limited by this embodiment. In the following embodiments, the same portions are denoted by the same reference numerals, and redundant description is omitted.

[Embodiment]
[1-1. Example of search device)
First, an example of extraction processing executed by the search device will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of an extraction process executed by the search device according to the embodiment. In FIG. 1, the search device 10 can communicate with a terminal device 100 used by a user U01 via a predetermined network N such as the Internet. Note that the search device 10 may be able to communicate with any number of terminal devices 100 used by any user.

  The search device 10 searches for second information similar to the first information input by the user U01 as a search query, and executes search processing for transmitting the searched second information to the terminal device 100 as a search result. For example, the search device 10 is realized by an information processing device such as a server device or a cloud system.

  The terminal device 100 is a terminal device used by the user U01, and is realized by, for example, a mobile mobile terminal device such as a smartphone or a tablet, a PC (Personal Computer) terminal, a server device, or the like. For example, the terminal device 100 has a screen that can display arbitrary information, and displays the search result received from the search device 10, that is, the second information, thereby displaying the first information input by the user U01. Second information similar to is displayed.

  As will be described later, it is assumed that the search device 10 can transmit / receive arbitrary information to / from an arbitrary server via a network N such as the Internet. For example, when the search device 10 accepts postings of various contents such as blogs, microblogs, web pages, messages, still images, moving images, sounds, etc., the retrieval apparatus 10 publishes the accepted contents, thereby publishing various information regarding the contents to the public. It is assumed that communication is possible with a web server that delivers various web contents, such as a server device that provides various SNSs (Social Networking Services) that are generally spread, and a server device that provides services related to electronic commerce and auctions.

[1-2. Example of information)
Here, in the following description, the search device 10 performs a search process for searching for an image similar to an image input as a search query. For example, when receiving an image (hereinafter referred to as “query image”) as the first information from the terminal device 100, the search device 10 describes the image to be searched (hereinafter referred to as “search target image”). In other words, an image similar to the query image is searched from among the images that are the second information, and the search result is transmitted to the terminal device 100.

  More specifically, the search device 10 stores an image to be searched and third information (hereinafter referred to as “meta information”) indicating the characteristics of the image in association with each other in advance. Here, the meta information is information indicating the characteristics of the associated image, for example, text indicating the semantic characteristics of the image, such as information indicating the imaging target included in the associated image. Upon receiving the query image, the search device 10 searches for an image similar to the query image from the search target images in consideration of the meta information. Then, the search device 10 transmits the searched image to the terminal device 100.

[1-3. About search processing)
Here, as a process for performing an image search, a technique for searching for an image having a feature amount similar to that of the query image can be considered. However, in such a method, an image having similar appearance characteristics such as the color and shape of the image is also extracted as a similar image, so an image having a different meaning from the query image, that is, an image having a different semantic is extracted. There is a risk. For example, if “red shoes” are captured in the query image, images with “red socks” and “red shoe ornaments” with similar colors and shapes are extracted as similar images. There is a fear.

  Here, a method is conceivable in which a tag indicating information indicating what is captured in the search target image is provided in advance. However, in such a method, when a query image is input as a search query, the user needs to input text indicating what is captured in the query image as a search query.

  Therefore, the search device 10 executes the following search process. First, the search device 10 generates a distributed expression indicating the characteristics of the first information. Subsequently, the search device 10 specifies third information similar to the distributed expression for which the distributed expression is generated, from the third information indicating the characteristics of the second information to be searched. Then, the search device 10 searches for information similar to the first information from the second information whose characteristic is the specified third information. For example, the search device 10 ranks the second information based on the specified third information, and selects second information similar to the first information among the second information whose rank is higher than a predetermined threshold. Search for.

  More specifically, the search device 10 generates a distributed expression indicating the characteristics of the query image. Further, the search device 10 identifies meta information whose distributed representation is similar to the feature of the query image from the meta information indicating the feature of the search target image. Then, the search device 10 selects an image similar to the query image from the search target images whose characteristics are specified by the identified meta information. For example, the search device 10 extracts a search target image to which the specified meta information is assigned and a search target image whose similarity with the specified meta information exceeds a predetermined threshold, and from the extracted search target images, An image whose appearance feature is similar to the query image is distributed to the terminal device 100 as a search result.

  In other words, the search device 10 identifies meta-information whose semantic features are similar to the semantic features of the query image as meta-information estimated to indicate the semantic features of the query image on the distributed representation space. Then, the search apparatus 10 uses the identified meta information to extract a search target image whose semantic features are similar to the query image from the search target images. Thereafter, the search device 10 searches for the search target image whose color or shape of the imaging target is similar to the query image among the extracted search target images, and transmits the search result to the terminal device 100.

[1-4. Example of search process)
Next, an example of search processing executed by the search device 10 will be described with reference to FIG. Note that the search device 10 stores the search target image and meta information indicating the characteristics of the imaging target whose image is included in the search target image and the semantic information of the search target image in association with each other in advance. .

  For example, the search device 10 receives the query image Q1 from the terminal device 100 used by the user U01 (step S1). In such a case, the search device 10 generates a distributed expression # Q1 indicating the semantic characteristics of the query image, such as the characteristics of the imaging target whose image is included in the query image Q1 and the semantic characteristics of the image (step S2). ). Here, the distributed representation is a multidimensional quantity that is generated from the query image Q1 and indicates the characteristics of the query image Q1.

  For example, the search device 10 uses an image in the query image Q1 using a model that has been learned to extract the characteristics of the imaging target from the image using a technique such as ILSVRC (ImageNet Large Scale Visual Recognition Challenge). A distributed expression indicating the characteristics of the imaging target included, that is, a distributed expression indicating the semantic characteristics of the query image Q1 is generated. Such a model may be realized by a neural network such as a convolutional neural network (CNN). Further, the search device 10 may generate a distributed representation of the query image Q1 using a model that classifies an imaging target whose image is included in the query image Q1 based on the Fisher vector of the query image Q1. As a result of such processing, the search device 10 generates a distributed expression # Q1 indicating the characteristics of the subject such as “a photo of a red boot”, for example.

  Subsequently, the search device 10 identifies meta information whose distributed representation is similar to the distributed representation # Q1 of the query image Q1 from the meta information of the search target image (step S3). For example, the search device 10 generates in advance a distributed representation that is some representation of the meta information. As such a distributed expression, for example, a distributed expression indicating a semantic feature included in the meta information of the search target image can be applied. In addition, the distributed expression may be any expression that expresses any information that can be recognized from the meta information, in addition to the semantic characteristics of the meta information. As a more specific example, the distributed representation is information that can classify meta information based on arbitrary information recognized from the meta information (for example, meta information content, meta information type, etc.). It is. In the following description, as an example of the distributed expression, an example of a process for generating a distributed expression indicating some characteristic that can be identified from the appearance of an image such as an imaging target as well as the semantic characteristics of the meta information will be described.

  Subsequently, the search device 10 specifies a distributed representation of meta information similar to the distributed representation # Q1. For example, the search device 10 specifies the distributed expressions # 1 to # 4 in which the distributed expression # Q1 and the Euclidean distance are within a predetermined range on the distributed expression space including the distributed expression of the meta information. Then, the search device 10 specifies meta information # 1 to # 4 corresponding to the specified distributed expressions # 1 to # 4. As a result of such processing, for example, the search device 10 identifies meta information # 1 to # 4 indicating features similar to the semantic features of the query image Q1 such as “photo of red boots”. Note that the search device 10 may specify a distributed expression whose cosine similarity is within a predetermined range with the distributed expression # Q1.

  Subsequently, the search device 10 uses the aggregated information obtained by aggregating the meta information to perform ranking and cutoff of the images to be searched (step S4). For example, the search device 10 uses the Bag-of-Words technique to generate aggregate information # Q1 in which semantic features of meta information # 1 to # 4 are aggregated. For example, the search device 10 may use, as the aggregate information # Q1, information indicating the category to which the identified meta information # 1 to # 4 belongs most among the categories to which the identified meta information # 1 to # 4 belongs.

  In addition, the search device 10 stores in advance image # P1 to # P4 that are search target images and meta information # P1 to # P4 indicating semantic features of the images # P1 to # P4. In such a case, the search apparatus 10 rearranges the sets of images and meta information # P1 to # P4 in the order of the similarity between the meta information # P1 to # P4 and the aggregate information # Q1, and a predetermined number from the top. Extract a set of That is, the search device 10 extracts an image in which the similarity between the meta information and the aggregate information # Q1 is higher than a predetermined threshold using the top-k algorithm.

  For example, the similarity between the meta information # P1 to # P4 and the aggregate information # Q1 is higher in the order of the meta information # P3, the meta information # P4, the meta information # P1, and the meta information # P2, and the similarity of the meta information When three images are extracted in order from the highest, the search device 10 selects the image # P3 corresponding to the meta information # P3, the image # P4 corresponding to the meta information # P4, and the image # corresponding to the meta information # P1. Extract P1. As a result of such processing, for example, the search device 10 selects an image having features similar to the semantic features of the query image Q1 indicated by the distributed expression # Q1, such as “photo of red footwear” and “photo of boots”. Will be extracted.

  Then, the search device 10 searches for an image having a feature amount as an image, that is, an image whose feature amount indicating the appearance feature of the image is similar to the query image Q1 from images having a high degree of similarity (step S5). ). For example, the search device 10 generates an image feature amount # Q1 of the query image Q1, using a technique such as a Fisher vector, generates an image feature amount # P3 from the image # P3, and generates an image feature amount ## from the image # P4. P4 is generated, and an image feature amount # P1 is generated from the image # P1. Then, the search device 10 calculates the similarity between the image feature amount # P3, # P4, # P1 and the image feature amount # Q1, respectively, and specifies an image feature amount whose calculated similarity exceeds a predetermined threshold. That is, the search device 10 performs similarity determination (similarity determination that is not semantic) at a lower level than when specifying images # P1 to # P4 that have similar semantic characteristics to the query image Q1.

  Thereafter, the search device 1 distributes the image corresponding to the specified image feature amount to the terminal device 100 as a similar image of the query image Q1 (step S6). For example, the search device 10 ranks the images # P3, # P4, and # P1 according to the degree of similarity between the image feature amounts # P3, # P4, and # P1 and the image feature amount # Q1. A web page in which the images # P3, # P4, and # P1 are arranged in this order may be distributed to the terminal device 100 as a similar image search result.

  As described above, the search device 10 searches the distributed representation space for meta information whose distributed representation is similar to the distributed representation indicating the semantic characteristics of the query image Q1, and searches the image corresponding to the searched meta information for the search target image. Extract from That is, the search device 10 generates a distributed representation that indicates the characteristics of the subject of the first information that is an image, and the distributed representation is generated from the third information that is a text indicating the characteristics of the subject of the second information that is an image. The third information similar to the distributed representation generated from the first information is specified. Then, the search device 10 searches for information whose appearance feature is similar to the first information from the second information whose feature is indicated by the specified third information.

  As a result, the search device 10 can exclude images having different semantic characteristics (that is, images that become semantic noise) from the search results, and as a result of reflecting the user's intention in the search results, the image search Accuracy can be improved.

[2. Search process variations)
In the above description, as an example of the search process, an image having a similar query image and image features and semantic features (that is, an image having similar semi-semantic features that can be identified from the appearance of the image) is extracted. A process for searching for an image whose appearance feature is similar to the query image among the extracted images has been described. However, the above description is merely an example, and the embodiment is not limited thereto. Hereinafter, an example of a variation of the search process will be described.

[2-1. Information types)
For example, in addition to images, the search device 10 may use arbitrary content such as moving images, texts, web pages, and sounds as the first information, the second information, and the third information. Further, the search device 10 may use a type of content different from the first information as the second information. For example, when the query image is input, the search device 10 may output arbitrary information such as a moving image or a web page similar to the query image. The third information is, for example, the second information such as an image or a thumbnail image indicating a characteristic part of the image, an image obtained by imaging the same imaging target from a different position, when the first information is an image or a moving image. May be the same type of information that assists or different types of information.

  For example, the search device 10 stores a moving image to be searched and meta information indicating the meaning of the moving image in association with each other. Such meta information includes, for example, not only the semantic characteristics of the subject captured in the moving image but also the appearance characteristics of the moving image, the names of directors and performers, genres such as drama and suspense, and the like. This meta-information indicates various characteristics. On the other hand, when receiving the query image, the search device 10 generates a distributed expression indicating the semantic characteristics of the received search query, and specifies meta information whose distributed expression is similar to the distributed expression of the query image. Then, the search device 10 extracts a moving image in which meta information similar to the identified meta information is associated, and searches the extracted moving images for moving images whose appearance features are similar to the query image.

  In another example, when the search device 10 accepts a movie PV (Promotion Video) as a search query, the search device 10 generates a semantic distributed representation of the PV, and the distributed representation is similar to the PV distributed representation. Identify information. Then, the search device 10 extracts a moving image (for example, movie content) that is highly relevant to the identified meta information, and has a high appearance relevance to the PV that is the search query among the extracted moving images. You may search a moving image.

  In another example, the search device 10 stores an auxiliary image that assists the search target image as meta information. Such an auxiliary image is, for example, an image obtained by photographing the imaging target of the search target image from a different position or an image taken at different timings. For example, when the search target image is a person's face, the auxiliary image may be an image obtained by capturing different facial expressions of the same person. Further, for example, when the search target image is an aerial photograph of a predetermined area, the auxiliary image may be a photograph or a map from the ground of the area. That is, the third information can be any information as long as the domain is different from the second information to be searched, such as an image including information that complements the information of the imaging target included in the search target image. is there.

  When storing such an auxiliary image, the search apparatus 10 generates in advance a distributed expression indicating semantic features such as who's face, profile, gender, age, etc. of the person who is the subject. deep. Then, the search device 10 identifies an auxiliary image whose distributed representation is similar to the distributed representation of the query image among the auxiliary images, and has an appearance feature from among the search target images associated with the identified auxiliary image. An image similar to the query image may be searched.

  In other words, the search device 10 generates a distributed expression indicating the feature of the first information serving as the search query, and the distributed expression is selected from the third information that is the meta information indicating the feature of the second information that is the search target information. If the third information similar to the generated distributed representation is specified, and the information similar to the first information is searched from the second information whose characteristic is the specified third information, any type of information is searched. Information can be adopted as the first information, the second information, and the third information.

  In addition, as described above, the search device 10 may specify information having a different type from the first information as the third information. For example, the search device 10 may adopt the image as the first information and the third information, and text indicating the semantic feature of the search target image as the meta information, that is, the third information.

  Even if the type of the first information is different from the type of the third information, a distributed expression indicating the semantic characteristics of the first information and a distributed expression indicating the semantic characteristics of the third information are obtained. By comparing, it is possible to identify the third information whose semantic features are similar to the first information. Further, the distributed representation of the first information and the distributed representation of the third information may not be information indicating the completely same feature as long as they indicate some common features among the plurality of features. .

[2-2. (About distributed expression)
Here, as long as the first information received as a search query can be classified using the distributed expression, the search device 10 may generate a distributed expression indicating the characteristics of the first information by an arbitrary method. Similarly, the search device 10 may generate a distributed representation of meta information in advance by any method as long as it can classify meta information using the distributed representation. Thus, if the first information and the meta information can be classified using the distributed representation, the first information after the classification and the meta information after the classification are associated with each other. It is possible to specify meta-information whose semantic features are similar to one information.

  That is, the search device 10 generates a distributed expression indicating the classification to which the feature of the first information belongs. Then, the search device 10 identifies a distributed representation that indicates the classification to which the feature of the third information belongs, that is, a distributed representation that is similar to the generated distributed representation from among the distributed representations of the meta information. The corresponding third information may be specified.

  The search device 10 is trained so as to generate similar distributed representations from the first information and the third information having similar semantic characteristics, and from the first information and the third information belonging to the same field. The distributed representation may be generated using the model. For example, when the query image is a shoe image, the search device 10 generates a distributed representation similar to a distributed representation of text whose meaning belongs to the field of footwear (for example, “boots”, “sneakers”, etc.) from the query image. A distributed representation of the query image may be generated using the model learned as described above.

  In other words, the search device 10 is arbitrary if it uses a model learned from a certain first information to generate a distributed expression similar to the distributed expression of the third information indicating the meaning of the first information. A distributed representation may be generated using the model. Such a model can be generated using, for example, a convolutional neural network or a Fisher vector.

  Further, the search device 10 may specify the third information whose distributed representation is similar to the distributed representation of the first information by any method. For example, when each element of the distributed expression is expressed by 1 or 0, the search apparatus 10 identifies the distributed expression in which the Hamming distance is within the predetermined range and the distributed expression of the first information, and corresponds to the specified distributed expression The third information to be specified may be specified. Further, the search device 10 may identify the third information corresponding to the distributed representation in which the Euclidean distance or the cosine similarity from the distributed representation generated from the first information is within a predetermined range among the third information. Good.

  For example, when generating a distributed representation that represents a semantic feature from a query image, meta information, or the like, depending on the distributed representation generation method, the norm of the distributed representation may indicate a semantic feature, or the semantic feature may not be displayed. Exists. Here, when the norm indicates a feature of meaning, if the similarity is determined based on the cosine similarity, it may not be possible to specify appropriate third information. Therefore, when the distributed expression is generated from the first information by a method in which a semantic feature appears in the norm, the search device 10 determines that the distributed expression generated from the first information and the Euclidean distance in the distributed expression space are a predetermined threshold value. What is necessary is just to specify the 3rd information corresponding to the following distributed expression. Further, when the distributed expression is generated from the first information by a method in which a semantic feature does not appear in the norm, the search device 10 has a predetermined threshold value based on the distributed expression generated from the first information and the cosine similarity in the distributed expression space. What is necessary is just to specify the 3rd information corresponding to the dispersion | distribution expression used as the above.

  Note that the search device 10 is an arbitrary expression that can be identified from the first information and the third information as long as it is a distributed representation of the first information that becomes the search query and the third information that indicates the characteristics of the second information that is the search target. Distributed representations having any number of features may be generated and compared. For example, when the first information is an image, the search device 10 may generate a distributed expression indicating an arbitrary combination of arbitrary information such as an image appearance, an imaging target, and an imaging date of the image. In addition, the search device 10 may generate a distributed representation showing the same feature as the first information from the third information, or may generate a distributed representation showing a feature different from the first information from the third information. .

[2-3. (About multiple search queries)
Here, the search device 10 may accept a plurality of search queries. Specifically, the search device 10 generates one distributed expression indicating the characteristics of the plurality of first information, specifies third information similar to the distributed expression from which the distributed expression is generated, and specifies the specified third expression Information similar to a plurality of pieces of first information received as a search query may be searched from the second information indicating the feature.

  For example, when receiving a plurality of query images, the search device 10 generates a distributed expression indicating the characteristics of each query image from each of the query images, and aggregates the generated distributed expressions, thereby Generate one distributed representation showing the features. Here, the search device 10 may simply link the distributed expressions generated from the query images, or may calculate a tensor product of each distributed expression. Then, the search device 10 identifies the third information similar to the one distributed expression generated by the distributed expression, and the degree of similarity with each query image is selected from the second information characterized by the identified third information. You may search the 2nd information which satisfy | fills a predetermined condition.

[2-4. About search)
Further, in the above description, the search device 10 identifies the third information whose distributed representation is similar to the distributed representation of the first information, and the first information is the first among the second information whose characteristics are identified. Information similar to information was searched. However, the embodiment is not limited to this. The search device 10 may execute an arbitrary search process as long as it searches for the third information whose first feature, semantic feature, and appearance feature are similar.

  For example, the search device 10 specifies a plurality of third information whose distributed representation is similar to the distributed representation generated from the first information, and information obtained by aggregating the plurality of specified third information is characterized by the second information Information similar to the first information may be retrieved from the list. For example, the search device 10 identifies the category to which the most third information belongs among the categories to which the plurality of identified third information belongs, and the third information belonging to the identified category is characterized by the second information Information similar to the first information may be searched from the inside. Further, the search device 10 identifies the third information having the largest number of associated second information among the plurality of third information, and the first information among the second information associated with the identified third information. Information similar to information may be searched.

[2-5. (Processing considering user attributes)
Here, considering image search for products in electronic commerce, it is predicted that more appropriate search processing can be realized by changing the range of similar images according to the attributes of the user U01. For example, even if an image of a product of a different type from the product captured in the query image is included, if the user U01 has a distracting personality, the type of the product different from that of the product captured in the query image As a result of generating the interest of the user U01 for the product, it is predicted that the sales opportunity can be improved. On the other hand, when the user U01 does not have a moving personality, it is predicted that sales opportunities can be improved by using only images of products of the same type as the products captured in the query image as search results.

  Further, depending on the attribute of the user U01, the semantic feature of the query image may change. For example, depending on demographic attributes such as the gender and age of the user U01 and psychographic attributes such as the preference of the user U01, the semantic features of the query image may change.

  Also, the purpose of the search process is to search for products that are subject to electronic commerce, to search for images that reinforce the description of products that are subject to electronic commerce, and to search for images posted on the net Then, it is considered that it is better to generate a distributed representation or search for similar images based on different features among the semantic features of the image. Also, depending on the purpose of such search processing, it may be better to change the range of similar images.

  Therefore, the search apparatus 10 may dynamically change the similarity range between the distributed representations or the similarity range between the first information and the third information according to the attribute of the user U01 and the usage mode of the image. The model used when generating the distributed representation from the first information may be dynamically changed.

  For example, the search device 10 uses the model corresponding to at least one of the attribute of the user U01 that is the destination of the second information serving as the search result and the purpose of use of the second information. A distributed representation may be generated from For example, the search device 10 distributes query images according to the gender and age of the user U01 from among a plurality of models that reflect different features among the semantic features of the input image in the distributed representation. A model to be used when generating an expression may be selected, and a distributed expression may be generated using the selected model. In addition, the search device 10 selects a model to be used when generating a distributed representation of the query image from a plurality of models according to the purpose of use of the image that is the search result, and distributes using the selected model. Expression generation may also be performed.

  In addition, for example, the search device 10 sets the similarity range between the distributed representation generated from the first information and the third information in accordance with at least one of the attribute of the user U01 or the purpose of using the second information. It may be changed. For example, when the attribute of the user U01 is moving, the search device 10 expands the range of the classification information of the third information determined to be similar to the classification information of the first information, and the attribute of the user U01 is changed. If not, the range of the classification information of the third information determined to be similar to the classification information of the first information may be reduced. That is, the search device 10 may change the range of meta information that is determined to have similar semantic characteristics to the query image according to the attribute of the user U01 and the purpose of use of the image that is the search result.

  For example, the search device 10 may change the similarity range between the first information and the second information according to at least one of the attribute of the user U01 and the purpose of using the second information. For example, the search device 10 expands the range of the second information that is determined to be similar to the first information when the attribute of the user U01 is moving, and the first device when the attribute of the user U01 is not moving. The range of the second information determined to be similar to the information may be reduced.

[2-6. (Semantic features)
Here, the search device 10 generates a distributed expression indicating the semantic characteristics of the first information from the first information such as the query image. Here, the semantic feature of the first information may be, for example, the topic of the text when the first information is text, or the subject when the first information is an image or a moving image. It may be a feature such as an object, a landscape, or a phenomenon. Further, for example, the semantic feature of the first information is a feature of information accompanying the first information such as a director, a performer, a screenwriter, and a genre when the first information is a moving image such as a movie. May be. Further, when the first information is music content, the semantic characteristics of the first information may be, for example, characteristics of a song title, a composer, a player, and the like.

  That is, the semantic feature of the first information is a concept including the content of information that the first information is intended to convey to the user U01, the content of incidental information recalled from the first information, and the like. is there. More specifically, the semantic features of the first information include features other than the information format used for transmitting information to the user U01, such as a text typeface, an image appearance, and a sound waveform. It is a concept that includes.

  Similarly, the third information indicating the semantic characteristics of the second information is the content of the information that the second information is intended to convey to the user U01 and the incidental information recalled from the first information. It is a concept that includes content and the like. In addition, the distributed expression indicating the semantic characteristics of the third information means the content of information that the third information tries to convey to the user U01, the content of incidental information recalled from the third information, etc. It is a concept that includes

[3. (Configuration of search device)
Hereinafter, an example of a functional configuration of the search device 10 that realizes the above-described search processing will be described. In the following description, when the query device 10 receives the query image as the first information, the search device 10 identifies the tag information whose distributed representation is similar to the distributed representation of the query image as the second information, and identifies the third information as the third information. It is assumed that processing for searching for an image similar to the query image from images corresponding to the tag information is executed.

  FIG. 2 is a diagram illustrating a configuration example of the search device according to the embodiment. As illustrated in FIG. 2, the search device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

  The communication unit 20 is realized by, for example, a NIC (Network Interface Card). The communication unit 20 is connected to the network N by wire or wirelessly and transmits / receives information to / from the terminal device 100.

  The storage unit 30 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. In addition, the storage unit 30 stores a generation model database 31, a distributed representation database 32, and an image database 33 (hereinafter may be collectively referred to as “databases 31 to 33”).

  In the generation model database 31, models used when generating a distributed expression from a query image are registered. For example, FIG. 3 is a diagram illustrating an example of information registered in the generation model database according to the embodiment. As shown in FIG. 3, information having items such as “model ID (Identifier)” and “model data” is registered in the generated model database 31. Here, the “model ID” is an identifier for identifying a model. The “model data” is model data, for example, data indicating a connection relationship between nodes in the CNN and a propagation coefficient.

  In addition to the information shown in FIG. 3, arbitrary information may be registered in the generated model database 31 as long as the information is related to the model. For example, in the generated model database 31, each model is a model that is selected when the attribute of the user U01 is selected, and an image that is a search result is selected for what purpose of use. Information such as whether it is a model to be registered may be registered.

  The model registered in the generation model database 31 is learned to generate a distributed expression indicating the characteristics of the image from the image using a technique such as a convolutional neural network or a Fisher vector, for example. It is a model. Each model registered in the generated model database 31 is a model that has been optimized according to the attribute of the user U01, the purpose of use of the image searched as a search result, and the like.

  For example, in the example illustrated in FIG. 3, the model ID “model ID # 1” and the model data “model data # 1” are registered in the generated model database 31 in association with each other. Such information indicates, for example, that the model data indicated by the model ID “model ID # 1” is model data “model data # 1”. In the example shown in FIG. 3, conceptual values such as “model ID # 1” and “model data # 1” are described, but actually, a character string serving as an identifier, model data, and the like are registered. The Rukoto.

  Returning to FIG. 2, the description will be continued. In the distributed expression database 32, distributed expressions of meta information are registered. That is, a distributed expression space for searching for a distributed expression similar to the distributed expression of the query image that is the first information is registered in the distributed expression database 32. For example, FIG. 4 is a diagram illustrating an example of information registered in the distributed expression database according to the embodiment. As shown in FIG. 4, information having items such as “meta information” and “distributed expression” is registered in the distributed expression database 32. Here, “meta information” may be added not only to meta information actually given to an image to be searched (that is, third information) but also to an image to be searched. Also includes meta information. The “distributed expression” is a distributed expression indicating the semantic characteristics of the associated meta-information. Semantic characteristics (that is, semi-semantic characteristics) that can be specified from the appearance of the image to be searched are represented. This is a distributed representation.

  For example, in the example shown in FIG. 4, meta information “meta information # T01” and a distributed expression “distributed expression # T01” are registered in the distributed expression database 32 in association with each other. Such information indicates that the distributed expression indicating the semantic characteristics of the meta information “meta information # T01” is the distributed expression “distributed expression # T01”. In the example shown in FIG. 4, conceptual values such as “meta information # T01” and “distributed expression # T01” are described. However, in actuality, a word indicating a semantic feature of an image to be searched or A text such as a predetermined numerical value is registered as meta information, and a multidimensional quantity indicating a semantic feature of the associated meta information is registered as a distributed expression.

  Returning to FIG. 2, the description will be continued. In the image database 33, images to be searched are registered. For example, FIG. 5 is a diagram illustrating an example of information registered in the image database according to the embodiment. As shown in FIG. 5, information having items such as “image ID”, “image data”, and “meta information” is registered in the image database 33. Here, the “image ID” is an identifier for identifying an image. “Image data” is data of an image to be searched. The “meta information” is meta information such as a word or a numerical value indicating a semantic feature of the associated image.

  For example, in the example illustrated in FIG. 5, the image database 33 includes an image ID “image ID # 01”, image data “image data # 01”, and meta information “meta information # P1 (meta information # T01, meta information # T01). T03...) ”Are registered in association with each other. Such information indicates that the image data of the image indicated by the image ID “image ID # 01” is the image data “image data # 01”, and indicates the semantic characteristics of the image indicated by the image ID “image ID # 01”. This indicates that the meta information is meta information “meta information # P1”. Here, “meta information # P1” is information including “meta information # T01” and “meta information # T03”, or information in which “meta information # T01” and “meta information # T03” are aggregated. And

  In the example shown in FIG. 5, conceptual values such as “image ID # 01” and “image data # 03” are described. However, in actuality, numerical values, character strings, and arbitrary formats for identifying images are used. The image data and the like of the image will be registered.

  Returning to FIG. 2, the description will be continued. The control unit 40 is a controller. For example, various programs stored in a storage device inside the search device 10 are stored in a RAM or the like by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). This is realized by being executed as a work area. The control unit 40 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

  As illustrated in FIG. 2, the control unit 40 includes a reception unit 41, a generation unit 42, a specification unit 43, a search unit 44, an output unit 45, and a collection unit 46. For example, the reception unit 41 receives a query image from the terminal device 100. In such a case, the search device 10 executes a search process to execute a similar image search similar to the query image.

  The generation unit 42 generates a distributed expression indicating the characteristics of the first information. Specifically, the generation unit 42 generates a distributed expression indicating a semantic feature of the first information such as a query image, that is, a distributed expression indicating a classification to which the feature of the first information belongs. For example, the generation unit 42 generates a distributed expression indicating the characteristics of the subject included in the query image. The generation unit 42 is a distributed expression that can be identified from the appearance of the image, and shows a distributed expression indicating a semantic characteristic of the image (for example, an imaging target), that is, a semi-semantic characteristic. If a distributed expression is generated, a distributed expression indicating arbitrary information may be generated.

  For example, the generation unit 42 uses a model used to generate a distributed representation of a query image from models registered in the generation model database 31 according to the characteristics of the user U01 and the purpose of use of the image that is the search result. Select. More specifically, the generation unit 42 selects a model that has been optimized from the models registered in the generation model database 31 according to the characteristics of the user U01 and the purpose of use of the search result image. Then, using the selected model, a distributed expression indicating the semantic characteristics of the query image is generated. For example, when receiving a plurality of query images, the generation unit 42 generates a distributed expression for each query image.

  The specifying unit 43 specifies third information whose distributed representation is similar to the distributed representation generated from the first information from the third information indicating the characteristics of the second information to be searched. More specifically, the specifying unit 43 specifies meta information, such as text, that is different in type from the query image that is the first information, as the third information.

  For example, the specifying unit 43 receives a distributed representation of the query image generated by the generating unit 42. In such a case, the specifying unit 43 specifies a distributed representation that is similar to the distributed representation generated by the generating unit 42 among the distributed representations registered in the distributed representation database 32, and associates the meta representation associated with the specified distributed representation. Information is specified from the distributed representation database 32. That is, the specifying unit 43 selects one or a plurality of distributed expressions similar to the distributed expression generated by the generating unit 42 from the distributed expressions indicating the semantic characteristics of the meta information such as the field to which the characteristics of the meta information belong. And the meta information corresponding to the specified distributed representation is specified.

  For example, the specifying unit 43 calculates the Euclidean distance and cosine similarity between the distributed representation generated by the generating unit 42 and each distributed representation registered in the distributed representation database 32, and the calculated distance and similarity are within a predetermined range. Identify the inner distributed representation. That is, the specifying unit 43 specifies a distributed expression similar to the distributed expression of the query image. And the specific | specification part 43 specifies the meta information matched with the specified dispersion | distribution expression.

  Such meta information is text indicating the semantic characteristics of the image that is the search result, such as the characteristics of the subject of the image that is the search result. The specifying unit 43 can specify meta information of an image having a semantic characteristic similar to that of the query image among the images that are the search results by executing the above-described processing. More specifically, the specifying unit 43 may specify meta information such as text indicating the semantic characteristics of the image that is the search result, and the distributed representation is similar to the distributed representation of the query image. it can.

  Note that the specifying unit 43 may change the similarity range to the distributed representation according to at least one of the attribute of the user U01 that is the destination of the image that is the search result and the purpose of use of the image. . For example, when the user U01 is moving, the specifying unit 43 expands the range of the distributed expression similar to the distributed expression of the query image (for example, the Euclidean distance) to the first threshold value that is larger than the reference threshold value. . In addition, when the user U01 is not moving, the specifying unit 43 reduces the range of the distributed expression similar to the distributed expression of the query image to a second threshold value that is smaller than the reference threshold value. As a result, the specifying unit 43 moves the range of meta information determined to have similar semantic characteristics to the query image in the meta information according to at least one of the attribute of the user U01 and the purpose of use of the image. Can be changed.

  The search unit 44 searches for information similar to the first information from the second information whose characteristic is the specified third information. More specifically, the search unit 44 ranks the images that are the search results based on the meta information specified by the specifying unit 43, and among the images whose rank is higher than a predetermined threshold, The image database 33 is searched for images having similar appearance characteristics.

  For example, when a plurality of pieces of meta information specified by the specifying unit 43 are received, the search unit 44 executes an aggregation process of the received meta information. For example, the search unit 44 generates aggregate information in which distributed representations of each meta information are linked. In another example, the search unit 44 specifies the category to which the meta information specified by the specifying unit 43 belongs most among the categories to which the meta information specified by the specifying unit 43 belongs. And the search part 44 is good also considering the distributed expression which shows the identified category as aggregate information. Then, the search unit 44 calculates the similarity between the meta information of each image registered in the image database 33 and the aggregate information.

  In addition, the search unit 44 specifies a predetermined number of images in descending order of the calculated similarity from the images registered in the image database 33. Note that the search unit 44 may specify only images whose similarity between the meta information and the aggregated information is equal to or greater than a predetermined threshold. Then, the search unit 44 searches the identified images for images that have similar appearance characteristics to the query image. For example, the search unit 44 compares the image feature amount of the specified image with the image feature amount of the query image, and specifies a predetermined number of images in descending order of similarity of the image feature amounts. Note that the search unit 44 may specify only images whose image feature quantity similarity is equal to or greater than a predetermined threshold. Then, the search unit 44 notifies the output unit 45 of the identified image.

  Note that the search unit 44 may change the similarity range with the query image according to at least one of the attribute of the user U01 and the purpose of use of the image. For example, the search unit 44 may increase the number of images specified from the images registered in the image database 33, and change the similarity between the meta information and the aggregated information or the similarity of the image feature amount. May be.

  The output unit 45 provides the user U01 with an image that is the result of the search process. For example, the output unit 45 generates, as a search result, a web page in which a plurality of images searched by the search unit 44 are arranged in descending order of similarity of image feature amounts. Then, the output unit 45 transmits the generated web page to the terminal device 100.

  The collection unit 46 collects a set of images to be searched and meta information. For example, the collection unit 46 collects, as search target data, a set of an image and text indicating a semantic characteristic of the image, that is, meta information, from an SNS server, a web server, or the like. Then, the collection unit 46 registers a set of the collected image and meta information in the image database 33.

  Therefore, the collection unit 46 acquires a set of an image and a text arranged in a web page related to electronic commerce or an auction. For example, FIG. 6 is a diagram illustrating an example of processing for collecting search target data. For example, the web page WP01 shown in FIG. 6 is a web page for selling a predetermined transaction target in electronic commerce or auction. Such a web page WP01 includes “image # 1” which is an image of a predetermined transaction target and “description # 1” which is text explaining the predetermined transaction target. Further, the web page WP01 includes a text “category # 1” indicating a category to which a predetermined transaction target belongs, and a search query “search query # 1” used when searching for the predetermined transaction target.

  In the web page WP01 relating to such electronic commerce and auctions, text such as “description # 1”, “category # 1”, “search query # 1”, etc. is information indicating characteristics of a predetermined transaction target. it is conceivable that. Further, “image # 1” arranged on such a web page WP01 is considered to be an image having a transaction target as an object, that is, an image showing the transaction target. Then, in the web page WP01, a set of “image # 1” and “description # 1”, “category # 1” or “search query # 1” can be regarded as a set of an image and tag information. it can.

  Accordingly, the collection unit 46 extracts “image # 1”, “description # 1”, “category # 1”, and “search query # 1” arranged in the web page WP01 as illustrated in FIG. To do. Further, the collection unit 46 generates meta information indicating the meanings of “description # 1”, “category # 1”, and “search query # 1” using a sentence analysis technique such as morphological analysis. The collection unit 46 may use “description # 1”, “category # 1”, and “search query # 1” as meta information as they are. Then, the collection unit 46 generates search data D01 in which meta information generated from “description # 1”, “category # 1”, and “search query # 1” is associated with “image # 1”. Then, the generated search data D01 is registered in the image database 33.

  In addition, the collection unit 46 uses a predetermined model to generate a distributed expression indicating the semantic characteristics of the meta information registered in the image database 33. Then, the collection unit 46 registers the set of the generated meta information and the distributed representation of the meta information in the distributed representation database 32. Note that the collection unit 46 may generate the distributed representation from the meta information using any model as long as the meta information can be classified using the distributed representation. The collection unit 46 collects arbitrary information such as text and images from the Internet as meta information, and registers the collected meta information and the distributed representation of the meta information in association with each other in the distributed representation database 32. Also good.

  Further, the collection unit 46 may collect not only text but also an image that assists the main image in the web page as meta information. That is, the collection unit 46 may use the same type of information as the first information as the third information. For example, FIG. 7 is a diagram illustrating a variation of processing for collecting search target data.

  For example, the web page WP02 illustrated in FIG. 7 includes “image # 2-1” and “image # 2-2” in addition to “image # 2” which is a predetermined transaction target image. Such “Image # 2-1” and “Image # 2-2” are images indicating the same transaction object as “Image # 2”, and are different from “Image # 2”, that is, “Image # 2”. It is considered that the information assists the semantic characteristics of “# 2”. Such “image # 2-1” and “image # 2-2” may be used as information indicating the semantic characteristics of “image # 2”, that is, tag information.

  Therefore, as illustrated in FIG. 7, the collection unit 46 collects “image # 2-1” and “image # 2-2” together with “image # 2” arranged on the web page WP02. Then, the collection unit 46 registers the collected “image # 2-1” and “image # 2-2” in the image database 33 as meta information of “image # 2”. In addition, the collection unit 46 uses a technique such as a convolutional neural network or a Fisher vector to generate and generate a distributed expression indicating the semantic features of “image # 2-1” and “image # 2-2”. The distributed representation may be registered in the distributed representation database 32.

[4. Example of search process)
Next, an example of the flow of search processing executed by the search device 10 will be described using FIG. FIG. 8 is a flowchart illustrating an example of a flow of search processing executed by the search device according to the embodiment. For example, the search device 10 receives a query image (step S101). In such a case, the search device 10 generates a distributed expression indicating the semantic characteristics of the query image (step S102). Subsequently, the search device 10 identifies meta information whose distributed representation is similar to the distributed representation of the query image (step S103), and ranks the images to be retrieved based on the identified meta information (step S104). ). Then, the search device 10 searches for an image similar to the query image from images whose rank is higher than a predetermined threshold (step S105), outputs the searched image (step S106), and ends the process.

[5. (Modification)
In the above, an example of search processing by the search device 10 has been described. However, the embodiment is not limited to this. Hereinafter, variations of the search processing executed by the search device 10 will be described.

[5-1. Realization method)
For example, the search device 10 performs processing for generating a distributed representation of the first information, processing for specifying third information whose distributed representation is similar to the distributed representation of the first information, and second information in which the specified third information is characteristic. The neural network may be learned so as to execute processing for searching for information similar to the first information from the information, and the search processing described above may be realized using the learned neural network. Further, the search device 10 may realize the search process described above by using a combination of neural networks that execute a part of each process included in the search process.

[5-2. About distributed expression space)
Here, when there are a large number of distributed representations of the third information belonging to the predetermined field, a robust result cannot be obtained when a distributed representation of the third information similar to the distributed representation of the first information is searched. There is a case. Therefore, the search device 10 may apply categorization processing using distributed representation to search processing.

  For example, the search device 10 generates a distributed representation of the third information in advance, and performs category classification of the third information based on the generated distributed representation. Then, when receiving the first information, the search device 10 generates a distributed representation of the received first information, and specifies the category of the third information to which the first information belongs, using the generated distributed representation. Thereafter, the search device 10 may search for information similar to the first information in the second information corresponding to the identified category (for example, an image in which the third information belonging to the identified category is associated). .

[5-3. About recommendations)
In the above description, the search device 10 distributes the second information similar to the first information input as the search query by the user U01 as the search result. However, the embodiment is not limited to this. For example, the search device 10 may distribute second information similar to the first information based on the input, action, or the like of the user U01 as recommended information for the user U01.

  For example, the search device 10 specifies an image of a product favored by the user U01 according to a browsing history of web content by the user U01, a purchase history in electronic commerce, or the like. In such a case, the search device 10 may search for images of similar products using the specified image as a query image, and distribute the image as a search result to the user U01 as recommendation information.

  In another example, the search device 10 identifies a PV of a movie in which the user U01 is interested based on the browsing result of the web content by the user U01. In such a case, the search device 10 searches for a moving image similar to the specified PV, such as another movie shot by the director of the specified PV movie, and uses the searched moving image as recommendation information. You may distribute to person U01.

[5-4. Device configuration〕
The search device 10 may be realized by a front-end server that transmits / receives information to / from the terminal device 100 and a back-end server that executes the above-described extraction process. In such a case, the back-end server is realized by an information processing apparatus having a generation unit 42, a specification unit 43, a search unit 44, and a collection unit 46. The front-end server is realized by an information processing apparatus having a reception unit 41 and an output unit 45. Further, the back-end server may be realized by a plurality of information processing apparatuses operating in cooperation.

  Further, the databases 31 to 33 stored in the storage unit 30 may be stored not in the search device 10 but in, for example, an external storage server.

[5-5. Others]
In addition, among the processes described in the above embodiment, all or part of the processes described as being automatically performed can be performed manually, or the processes described as being performed manually can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  In addition, the above-described embodiments can be appropriately combined within a range in which processing contents are not contradictory.

[6. program〕
Further, the search device 10 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 9, for example. FIG. 9 is a diagram illustrating an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090. Have

  The arithmetic device 1030 operates based on a program stored in the primary storage device 1040 and the secondary storage device 1050, a program read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various arithmetic operations. The secondary storage device 1050 is a storage device in which data used for various calculations by the calculation device 1030 and various databases are registered, and is realized by a ROM (Read Only Memory), HDD, flash memory, or the like.

  The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), This is realized by a standard connector such as HDMI (registered trademark) (High Definition Multimedia Interface). The input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, and a scanner, and is realized by, for example, a USB.

  The input device 1020 includes, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), and a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), and a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. The input device 1020 may be an external storage medium such as a USB memory.

  The network IF 1080 receives data from other devices via the network N and sends the data to the arithmetic device 1030, and transmits data generated by the arithmetic device 1030 to other devices via the network N.

  The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

  For example, when the computer 1000 functions as the search device 10, the arithmetic device 1030 of the computer 1000 implements the function of the control unit 40 by executing a program loaded on the primary storage device 1040.

[7. effect〕
As described above, the search device 10 generates a distributed expression indicating the characteristics of the first information. Further, the search device 10 identifies third information whose distributed representation is similar to the distributed representation of the first information from the third information indicating the characteristics of the second information to be searched. Then, the search device 10 searches for information similar to the first information from the second information whose characteristic is the specified third information.

  In this way, the search device 10 searches for information similar to the first information from the second information corresponding to the third information whose distributed representation is similar to the first information, among the second information to be searched. To do. As a result of such processing, for example, the search device 10 searches for information similar to the first information from the second information similar in semantic characteristics to the first information. As a result of realizing the reflected search, the search accuracy can be improved.

  Further, the search device 10 specifies information having a type different from that of the first information as the third information. For example, the search device 10 generates a distributed expression indicating the classification to which the feature of the first information belongs. Then, the search device 10 identifies a distributed representation similar to the distributed representation of the first information from among the distributed representations indicating the classification to which the feature of the third information belongs, and identifies the third information corresponding to the identified distributed representation To do. For this reason, the search device 10 can improve the search accuracy even when information different in type from the first information, such as meta information, is associated with the second information to be searched.

  In addition, the search device 10 generates a distributed expression indicating the semantic characteristics of the first information, and the distributed expression is a distributed expression of the first information from the third information indicating the semantic characteristics of the second information. Specify similar second information. For this reason, the search device 10 can search information similar to the first information from the second information similar in semantic characteristics to the first information.

  In addition, the search device 10 generates a distributed representation that indicates the characteristics of the first information that is an image, and the distributed representation is the third information that is the text that indicates the characteristics of the second information that is the image. The third information similar to the feature is specified. For this reason, the search device 10 can improve the accuracy of the image search.

  Further, the search device 10 uses the model corresponding to at least one of the attribute of the user to whom the searched second information is provided or the purpose of use of the second information to perform distributed representation from the first information. Generate. In addition, the search device 10 sets a similar range to the distributed representation of the first information according to at least one of the attribute of the user who provides the searched second information or the purpose of use of the second information. change. In addition, the search device 10 changes the similarity range with the first information according to at least one of the attribute of the user who provides the searched second information and the purpose of using the second information. For this reason, the search device 10 can further improve the search accuracy as a result of reflecting the user's intention by the search process.

  Further, the search device 10 ranks the second information based on the specified third information, and the second information similar to the first information among the second information whose rank is higher than a predetermined threshold value. Search for. For this reason, for example, the search device 10 can preferentially include the second information whose semantic features are more similar to the first information in the search result.

  In addition, the search device 10 generates a distributed expression indicating the characteristics of the subject of the first information that is an image. In addition, the search device 10 identifies third information whose distributed representation is similar to the distributed representation of the first information from the third information that is the text indicating the characteristics of the subject of the second information that is the image. Then, the search device 10 searches for information whose appearance feature is similar to the first information from the second information whose feature is indicated by the third information specified by the specifying unit. For this reason, the search device 10 can improve the accuracy of the image search.

  In addition, the search device 10 identifies a plurality of third information whose distributed representation is similar to the distributed representation of the first information, and information obtained by aggregating the plurality of identified third information is selected from among the second information whose features are Search for information similar to the first information. For example, the search device 10 identifies the category to which the most third information belongs among the categories to which the plurality of identified third information belongs, and the third information belonging to the identified category is characterized by the second information Search for information similar to the first information. For this reason, the search device 10 can search for the second information whose features are generally similar to the first information.

  Further, the search device 10 identifies the third information corresponding to the distributed expression in which the Euclidean distance or the cosine similarity from the distributed expression of the first information is within a predetermined range among the third information. For this reason, the search device 10 can search for the second information in which the third information whose feature is similar to the first information indicates the feature.

  In addition, the search device 10 generates a distributed expression indicating the characteristics of the first information using a convolutional neural network or a Fisher vector. For this reason, the search device 10 can accurately compare the distributed representation of the first information and the distributed representation of the third information.

  In addition, the search device 10 generates one distributed expression indicating the characteristics of the plurality of first information, and searches for information similar to the plurality of first information. For this reason, the search device 10 can search for the second information whose features are generally similar to the first information.

  As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are merely examples, and various modifications, including the aspects described in the disclosure section of the invention, based on the knowledge of those skilled in the art, It is possible to implement the present invention in other forms with improvements.

  Moreover, the above-mentioned “section (module, unit)” can be read as “means”, “circuit”, and the like. For example, the specifying unit can be read as specifying means or a specific circuit.

DESCRIPTION OF SYMBOLS 10 Search apparatus 20 Communication part 30 Memory | storage part 31 Generation | occurrence | production model database 32 Distributed expression database 33 Image database 40 Control part 41 Reception part 42 Generation part 43 Specification part 44 Search part 45 Output part 46 Collection part 100 Terminal apparatus

Claims (16)

A generating unit that generates a distributed expression space in which only the distributed expression of predetermined meta information is projected, and a distributed expression of first meta information indicating characteristics of the first information;
A specifying unit that specifies one or more pieces of meta information whose distributed representation projected on the distributed representation space is similar to the distributed representation of the first information from among the predetermined meta information;
The second meta information indicating the characteristics of the second information is compared with the meta information specified by the specifying unit, and from among the second information to be searched for one or more of the second meta information, Based on a comparison result between the extracted second information and the first information, the second information having a matching degree between the second meta information and the meta information specified by the specifying unit is higher than a predetermined threshold. And a search unit for searching for second information similar to the first information. The generation unit generates a distributed representation of first meta information indicating a classification to which the feature of the first information belongs,
The specifying unit specifies, as the predetermined meta information, meta information whose distributed representation is similar to the distributed representation generated by the generating unit from meta information indicating a classification to which the feature of the second information belongs. The search device according to claim 1. The generating unit generates a distributed representation of first meta information indicating a semantic feature of the first information;
The specifying unit specifies, as the predetermined meta information, meta information whose distributed representation is similar to the distributed representation generated by the generating unit from meta information indicating a semantic feature of the second information. The search device according to claim 1, wherein: The generation unit generates a distributed representation of first meta information indicating characteristics of the first information that is an image,
The specifying unit specifies, as the predetermined meta information, meta information whose distributed representation is similar to the distributed representation generated by the generating unit from meta information indicating characteristics of the second information that is an image. The search device according to any one of claims 1 to 3. The generation unit uses the model corresponding to at least one of the attribute of the user who provides the second information searched by the search unit or the purpose of use of the second information. The distributed representation of 1st meta information is produced | generated. The search device as described in any one of Claims 1-4 characterized by the above-mentioned. The identification unit is a distributed expression generated by the generation unit according to at least one of an attribute of a user who provides the second information searched by the search unit and a purpose of using the second information. The search device according to claim 1, wherein the similarity range is changed. The search unit sets a similar range to the first information according to at least one of an attribute of a user who provides the second information searched by the search unit and a purpose of use of the second information. It changes. The search device as described in any one of Claims 1-6 characterized by the above-mentioned. The search unit ranks the second information based on the degree of coincidence between the meta information specified by the specifying unit and the second meta information indicating the feature of each second information, and the rank is a predetermined threshold value. The search device according to any one of claims 1 to 7, wherein second information similar to the first information is searched from among higher second information. The generation unit generates a distributed representation of first meta information indicating characteristics of a subject of the first information that is an image,
The specifying unit is a meta-analog having a distributed representation similar to the distributed representation generated by the generating unit from the meta information that is a text indicating the characteristics of the subject of the second information that is an image as the predetermined meta-information. Information is specified. The search device according to claim 1, wherein information is specified. The search unit extracts second information associated with second meta information whose degree of coincidence with aggregated information obtained by aggregating a plurality of meta information specified by the specifying unit is higher than a predetermined threshold. The search device according to any one of claims 1 to 9. The search unit specifies a category to which most meta information belongs among categories to which the plurality of meta information specified by the specifying unit belongs, and is associated with second meta information belonging to the specified category. The search device according to claim 10, wherein information similar to the first information is searched from information. The identifying unit is characterized in that, among predetermined meta information, the distributed representation identifies meta information whose Euclidean distance or cosine similarity from the distributed representation generated by the generating unit is within a predetermined range. The search device according to any one of claims 1 to 11. The search device according to any one of claims 1 to 12, wherein the generation unit generates a distributed expression indicating a feature of the first information using a convolutional neural network or a Fisher vector. The generation unit generates one distributed expression indicating the characteristics of a plurality of pieces of first information,
The search device according to any one of claims 1 to 13, wherein the search unit searches for information similar to the plurality of first information. A search method executed by a search device,
Generating a distributed representation space in which only the distributed representation of the predetermined meta information is projected, and a distributed representation of the first meta information indicating the characteristics of the first information;
A specifying step of identifying one or more pieces of meta information whose distributed representation projected on the distributed representation space is similar to the distributed representation of the first information from the predetermined meta information;
The second meta information indicating the characteristics of the second information is compared with the meta information specified in the specifying step, and the second meta information is a search target associated with one or more of the second meta information. Based on a comparison result between the extracted second information and the first information, the second information having a degree of coincidence between the second meta information and the meta information specified by the specifying step is higher than a predetermined threshold. And a search step for searching for second information similar to the first information. On the computer,
A generation procedure for generating a distributed representation space in which only a distributed representation of predetermined meta information is projected, and a distributed representation of first meta information indicating characteristics of the first information;
A specific procedure for identifying one or more pieces of meta-information whose distributed representation projected on the distributed representation space is similar to the distributed representation of the first information from the predetermined meta-information;
The second meta information indicating the characteristics of the second information is compared with the meta information specified by the specifying procedure, and from among the second information to be searched for one or more of the second meta information, Based on a comparison result between the extracted second information and the first information, the second information whose degree of coincidence between the second meta information and the meta information specified by the specifying procedure is higher than a predetermined threshold is extracted. A search program for executing a search procedure for searching for second information similar to the first information.

Images