JP6204957B2 - Information processing apparatus, information processing method, and information processing program - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

  In recent years, a search technique for searching for target data from a set of data is known. As a technique related to such a search, a technique for searching for an object depicted in an image using image content is known. For example, in such a technique, a search result using a landmark name in a search query image with position information input from a user terminal as a search query is received.

JP 2010-224745 A

  However, with the above-described conventional technology, it is not always possible to obtain a search result desired by the user. For example, in the above prior art, a landmark in a search query image is estimated based on feature information in a search query image with position information input from a user terminal, and the name of the estimated landmark is searched as a search query. To do. For this reason, the search query may not be what the user desires. For this reason, the above-described conventional technology cannot always obtain a search result desired by the user.

  The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of enabling a search desired by a user.

  An information processing apparatus according to the present application includes: an accepting unit that accepts an image; an extracting unit that extracts search query candidates based on the image accepted by the accepting unit; and a search query candidate extracted by the extracting unit. And an output unit for outputting.

  According to one aspect of the embodiment, there is an effect that a search desired by the user can be made possible.

FIG. 1 is an explanatory diagram illustrating an example of a presentation process performed by the information processing apparatus according to the embodiment. FIG. 2 is a diagram illustrating a configuration example of the presentation system according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of the information processing apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of a search query information storage unit according to the embodiment. FIG. 5 is a sequence showing a presentation processing procedure by the presentation system. FIG. 6 is a diagram illustrating an example of a display screen when there are a plurality of subjects. FIG. 7 is a diagram illustrating an example of a display screen when the search query is an image. FIG. 8 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing apparatus.

  Hereinafter, a mode for carrying out an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings. Note that the information processing apparatus, the information processing method, and the information processing program according to the present application are not limited by this embodiment. Moreover, in the following embodiment, the same code | symbol is attached | subjected to the same site | part and the overlapping description is abbreviate | omitted.

[1. Embodiment)
[1-1. Presentation processing according to embodiment]
First, an example of presentation processing according to the embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram illustrating an example of a presentation process performed by the information processing apparatus 100 according to the embodiment. In the example of FIG. 1, a presentation process for presenting search query candidates based on an image input from the terminal device 10 owned by the user is performed.

  First, the user having the terminal device 10 captures the subject Ob1. Specifically, the user points the camera toward the subject Ob1 for which a search query candidate is desired. Then, as shown in FIG. 1A, the user focuses the subject Ob1 by placing the subject Ob1 inside an AF (Auto Focus) frame Fr displayed on the display screen Wi of the terminal device 10. Then, the user presses the shooting button Bt1 while focusing on the subject Ob1. Thereby, the terminal device 10 generates an image P1 in which the subject Ob1 is depicted.

  Subsequently, as illustrated in FIG. 1B, the user selects an image TP by designating a range of a portion including the face of the subject Ob <b> 1 in the image P <b> 1 displayed on the display screen Wi of the terminal device 10. . Then, the user presses the search button Bt2 with the image TP selected. Thereby, the terminal apparatus 10 transmits the image TP to the information processing apparatus 100.

  Then, the information processing apparatus 100 receives the image TP from the terminal device 10. Subsequently, the information processing apparatus 100 extracts search query candidates based on the received image TP. Specifically, the information processing apparatus 100 selects a search query candidate based on the similarity between the feature quantity of the image TP and the feature quantity of the reference image associated with the search query and stored in the storage unit. Extract. For example, the information processing apparatus 100 extracts search query candidates based on the similarity between the facial feature quantity of the subject Ob1 depicted in the image TP and the facial feature quantity of the person depicted in the reference image. To do. As an example, the information processing apparatus 100 extracts keywords such as names corresponding to reference images whose similarity to the image TP is equal to or greater than a predetermined threshold as search query candidates.

  Subsequently, the information processing apparatus 100 outputs the extracted search query candidates. Specifically, the information processing apparatus 100 outputs a list in which search query candidates are displayed in an order based on the similarity. For example, the information processing apparatus 100 outputs a list displaying search query candidates from the top in descending order of similarity to the terminal apparatus 10. Thereby, the terminal device 10 displays, on the display screen Wi, the list Li in which the search query candidates SQ1 to SQ5 are listed in descending order of similarity, as illustrated in FIG. As a result, the information processing apparatus 100 can present a search query candidate desired by the user to the terminal device 10.

  Thereafter, for example, when the search query candidate SQ1 is selected by the user, the terminal device 10 sends a search result request for the search for “A man” corresponding to the selected search query candidate SQ1. 50. Subsequently, the terminal apparatus 10 receives a search result SR obtained by performing a search for “Male A” as a response to the search result request. Then, as illustrated in FIG. 1D, the terminal device 10 displays the received search result SR for “A man” on the display screen Wi.

  As described above, the information processing apparatus 100 according to the embodiment receives an image. In addition, the information processing apparatus 100 extracts search query candidates based on the received image. In addition, the information processing apparatus 100 outputs the extracted search query candidates.

  Thereby, since the information processing apparatus 100 can present search query candidates based on the image, it is possible to enable a search desired by the user. For example, since the information processing apparatus 100 can extract a keyword that the user desires to search, the user's intention can be verbalized. As an example, the information processing apparatus 100 can present the name of the subject drawn in the image to the user.

  Moreover, since the information processing apparatus 100 can present the user with a list on which search query candidates are displayed, the user can easily grasp the search query. For example, the information processing apparatus 100 can present various types of information related to the subject such as the name of the subject, and thus can provide useful information for a user who wants to know the subject deeply.

[1-2. Configuration of Presentation System According to Embodiment]
Next, the configuration of the presentation system 1 according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating a configuration example of the presentation system 1 according to the embodiment. As shown in FIG. 2, the presentation system 1 includes a terminal device 10, an information processing device 100, and a search server 50. The terminal device 10, the information processing device 100, and the search server 50 are connected to the network N so that they can communicate with each other by wire or wirelessly.

  The terminal device 10 is an information processing device such as a smartphone, a mobile phone, a tablet terminal, a PDA (Personal Digital Assistant), or a notebook PC (Personal Computer). The terminal device 10 is owned by a user who uses various functions (for example, a search query presenting function). Specifically, the terminal device 10 transmits an image selected by the user to the information processing device 100.

  The information processing apparatus 100 is a server apparatus that outputs search query candidates. Specifically, the information processing apparatus 100 extracts search query candidates based on the image received from the terminal apparatus 10, and outputs the extracted search query candidates to the terminal apparatus 10.

  The search server 50 is a server device that outputs a search result obtained by performing a search for a search query. Specifically, the search server 50 receives a request for a search result of the search query from the terminal device 10. Then, the search server 50 transmits the search query search result to the terminal device 10 as a response to the received request.

  2 illustrates an example in which the presentation system 1 includes one terminal device 10, one information processing device 100, and one search server 50. However, the presentation system 1 includes A plurality of terminal devices 10, a plurality of information processing devices 100, and a plurality of search servers 50 may be included.

[1-3. Configuration of Information Processing Device According to Embodiment]
Next, the configuration of the information processing apparatus 100 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a configuration example of the information processing apparatus 100 according to the embodiment. As illustrated in FIG. 3, the information processing apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing apparatus 100 includes an input unit (for example, a keyboard and a mouse) that receives various operations from an administrator who uses the information processing apparatus 100, and a display unit (for example, a liquid crystal display) that displays various types of information. ).

(About the communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). The communication unit 110 is connected to a network by wire or wireless, and transmits / receives information to / from the terminal device 10 via the network. For example, the communication unit 110 transmits and receives images and search query candidates to and from the terminal device 10 via the network.

(About the storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 includes a search query information storage unit 121.

(Regarding the search query information storage unit 121)
The search query information storage unit 121 stores information related to the search query. Specifically, the search query information storage unit 121 stores a keyword and an image corresponding to the search query for each search query. Here, FIG. 4 shows an example of the search query information storage unit 121 according to the embodiment. As shown in FIG. 4, the search query information storage unit 121 includes items such as “search query ID”, “keyword”, and “image”.

  “Search query ID” indicates identification information for identifying a search query. For example, “search query ID” stores an identifier such as a unique alphanumeric character assigned to each search query. “Keyword” indicates a keyword corresponding to the search query. For example, the “keyword” stores a character string representing a search query. “Image” indicates an image corresponding to the search query. For example, a reference image in which an image corresponding to a search query is drawn is stored in “image”. As an example, “image” stores a plurality of reference images taken from various angles corresponding to the search query. The “image” is not limited to a plurality of reference images shown in the figure, and an arbitrary number of reference images is stored. FIG. 4 illustrates an example in which conceptual information such as “RP11” is stored in “image”, but actually, a still image, a file path name indicating a storage location of the image, and the like are stored.

  That is, FIG. 4 illustrates an example in which the keyword corresponding to the search query identified by the search query ID “Q1” is “A man”. Further, an example is shown in which the images corresponding to the search query identified by the search query ID “Q1” are “RP11”, “RP12”, “RP13”, and the like.

(About the control unit 130)
Returning to FIG. 3, the control unit 130 uses various programs (an example of an information processing program) stored in a storage device inside the information processing apparatus 100 by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). This is realized by executing the RAM as a work area. The control unit 130 is realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

  As illustrated in FIG. 3, the control unit 130 includes a reception unit 131, an extraction unit 132, and an output unit 133, and realizes or executes functions and operations of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration illustrated in FIG. 3, and may be other configurations as long as the presentation process described later is performed. Further, the connection relationship between the processing units included in the control unit 130 is not limited to the connection relationship illustrated in FIG. 3, and may be another connection relationship.

(About the reception unit 131)
The accepting unit 131 accepts an image. Specifically, the accepting unit 131 accepts from the terminal device 10 an image in which an object for which the user desires a search query is depicted. More specifically, the receiving unit 131 receives an image by receiving an image selected by the user from the terminal device 10. For example, the accepting unit 131 accepts, as an image, a captured image obtained by capturing an object for which the user desires a search query. As an example, the accepting unit 131 accepts an image displayed on the monitor as a captured image when the user presses the capture button.

(About the extraction unit 132)
The extraction unit 132 extracts search query candidates based on the image received by the reception unit 131. Specifically, the extraction unit 132 extracts search query candidates based on the similarity between the feature amount of the image and the feature amount of the reference image associated with the search query and stored in the storage unit. . For example, the extraction unit 132 first refers to the search query information storage unit 121 and calculates the similarity by collating the image received by the reception unit 131 with each reference image. Then, the extraction unit 132 preferentially extracts keywords corresponding to reference images having a high degree of similarity as search query candidates. As an example, the extraction unit 132 extracts a keyword corresponding to a reference image whose similarity with the image is equal to or greater than a predetermined threshold as a search query candidate. As one aspect, in the case where a person's face is depicted in the image received by the accepting unit 131, the extracting unit 132 includes a storage unit in which a reference image depicting the person's face is stored for each person. The search query candidate is extracted based on the similarity between the feature amount of the face of the subject depicted in the image and the feature amount of the face of the person depicted in the reference image.

(About the output unit 133)
The output unit 133 outputs search query candidates extracted by the extraction unit 132. Specifically, the output unit 133 outputs a list on which search query candidates are displayed to the terminal device 10. For example, the output unit 133 displays search query candidates in an order based on the similarity between the received image feature quantity and the reference image feature quantity associated with the search query and stored in the storage unit. Output a list. As an example, the output unit 133 outputs, to the terminal device 10, a list in which keywords corresponding to the reference image extracted by the extraction unit 132 are displayed from the top in descending order of similarity. Thereby, the terminal device 10 displays the list output by the output unit 133 on the display screen.

[1-4. Presentation processing procedure according to embodiment]
Next, a processing procedure by the presentation system 1 according to the embodiment will be described with reference to FIG. FIG. 5 is a sequence showing a presentation processing procedure by the presentation system 1 according to the embodiment.

  As shown in FIG. 5, the terminal device 10 selects an image (step S101). For example, the terminal device 10 selects an image obtained by photographing a subject for which the user desires a search query. Then, the information processing apparatus 100 receives an image (step S102). For example, the information processing apparatus 100 receives an image by receiving an image selected by the user from the terminal device 10.

  Thereafter, the information processing apparatus 100 extracts search query candidates based on the received image (step S103). Specifically, the information processing apparatus 100 determines the search query based on the similarity between the received image feature amount and the reference image feature amount associated with the search query and stored in the storage unit. Extract candidates. As an example, the information processing apparatus 100 extracts, as candidates, a search query corresponding to a reference image whose similarity with an image is equal to or greater than a predetermined threshold.

  Subsequently, the information processing apparatus 100 outputs the extracted search query candidates (step S104). Specifically, the information processing apparatus 100 outputs a list in which search query candidates are displayed in an order based on the similarity. For example, the information processing apparatus 100 outputs a list displaying search query candidates from the top in descending order of similarity to the terminal apparatus 10.

  As a result, the terminal device 10 receives search query candidates from the information processing device 100. Then, the terminal device 10 displays the search query candidates output from the information processing apparatus 100 (step S105). For example, the terminal device 10 displays a list in which search query candidates are displayed from the top in descending order of similarity.

  Thereafter, when a search query candidate is selected by the user (step S106), the terminal device 10 transmits a search result request for the selected search query candidate to the search server 50 (step S107). Thereby, the search server 50 receives a search result request for the search query from the terminal device 10.

  Subsequently, the search server 50 searches for the search query indicated by the received request. And the search server 50 transmits the search result about a search query to the terminal device 10 as a response to the request from the terminal device 10 (step S108). Thereby, the terminal device 10 receives the search result for the search query from the search server 50. Then, the terminal device 10 displays the received search result (step S109).

[1-5. Effects of the embodiment
As described above, the information processing apparatus 100 according to the embodiment includes the reception unit 131, the extraction unit 132, and the output unit 133. The accepting unit 131 accepts an image. The extraction unit 132 extracts search query candidates based on the image received by the reception unit 131. The output unit 133 outputs search query candidates extracted by the extraction unit 132.

  Thereby, since the information processing apparatus 100 can present search query candidates based on the image, it is possible to enable a search desired by the user. For example, since the information processing apparatus 100 can extract a keyword that the user desires to search, the user's intention can be verbalized. As an example, the information processing apparatus 100 can present the name of the subject drawn in the image to the user.

  Moreover, since the information processing apparatus 100 can present the user with a list on which search query candidates are displayed, the user can easily grasp the search query. For example, the information processing apparatus 100 can present various types of information related to the subject such as the name of the subject, and thus can provide useful information to a user who wants to know the subject deeply.

  In the information processing apparatus 100 according to the embodiment, the extraction unit 132 is based on the similarity between the feature amount of the image and the feature amount of the reference image stored in the storage unit in association with the search query. Extract search query candidates. As a result, the information processing apparatus 100 can extract a search query corresponding to a reference image similar to the received image, thereby enabling a search desired by the user with high accuracy.

  In the information processing apparatus 100 according to the embodiment, the output unit 133 outputs a list in which search query candidates are displayed in an order based on the similarity. As a result, the information processing apparatus 100 can present the search query candidates that are highly likely to be a search desired by the user to the user in an easy-to-view manner, thereby improving user convenience.

  In the information processing apparatus 100 according to the embodiment, the reception unit 131 receives a captured image in which a search target is captured as an image. As a result, the information processing apparatus 100 can easily accept an image simply by the user capturing an image of the target, and thus can quickly and easily present a search query for a target that the user desires to search to the user.

[2. (Modification)
The information processing apparatus 100 according to the above-described embodiment may be implemented in various different forms other than the above-described embodiment. Therefore, hereinafter, another embodiment of the information processing apparatus 100 will be described.

[2-1. Search query based on action history)
In the embodiment described above, an example in which the information processing apparatus 100 outputs a list in which search query candidates are displayed in the order based on the similarity is described. Here, the information processing apparatus 100 may output search query candidates based on various types of information.

  Specifically, the output unit 133 of the information processing apparatus 100 outputs search query candidates based on the user's behavior history. For example, the information processing apparatus 100 outputs search query candidates based on the user search history or the history of messages about the user as the user action history. As an example, the information processing apparatus 100 outputs more preferentially the search query related to the keyword searched by the user most recently. In another example, the information processing apparatus 100 preferentially outputs a search query related to a keyword included in a message transmitted / received most recently by the user. In another example, the information processing apparatus 100 outputs search query candidates based on a web service usage history or an application usage history as a user activity history. For example, the information processing apparatus 100 gives priority to a search query related to various web services and applications such as online shopping, auction, video distribution service, music distribution service, map information providing service, reservation service, etc. that the user has used most recently. Output. As one aspect, the information processing apparatus 100 outputs a list that is displayed from the top as a search query having a higher priority based on a user's behavior history. Note that the information processing apparatus 100 may output search query candidates based on user attributes indicating the user's hobbies and preferences.

  Thus, the information processing apparatus 100 according to the modification outputs search query candidates based on the user's behavior history. Thereby, since the information processing apparatus 100 can output a search query related to the user's behavior, it is possible to present a search query that enables a search desired by the user.

  Further, the information processing apparatus 100 according to the modification outputs search query candidates based on the user search history or the history of messages about the user as the user action history. As a result, the information processing apparatus 100 can output a search query related to a keyword used by a user in a search or a message, and can therefore present a search query that enables a search desired by the user with high accuracy. .

[2-2. Input image)
In the above-described embodiment, an example in which the information processing apparatus 100 receives a captured image in which a search target is captured as an image has been described. Here, the information processing apparatus 100 may accept various images without being limited to the captured images captured by the user.

  Specifically, the reception unit 131 of the information processing apparatus 100 receives an image being displayed on the screen of the terminal device 10 owned by the user as an image. More specifically, the information processing apparatus 100 receives from the terminal device 10 an image displayed on the screen at a timing before the user presses the shooting button. In another example, the information processing apparatus 100 accepts not only a captured image or an image currently displayed on the screen at the timing before shooting but also an image downloaded via the Internet, a captured image, and the like from the terminal device 10. Also good.

  As described above, the information processing apparatus 100 according to the modification receives an image being displayed on the screen of the terminal device owned by the user as an image. Thereby, since the information processing apparatus 100 can accept the image being displayed on the screen, it is possible to reduce time and effort for the user to shoot the target for which the search query is requested.

[2-3. (Search query candidates are extracted according to the area of the subject)
In the embodiment described above, an example in which the information processing apparatus 100 extracts search query candidates based on an image received by the receiving unit 131 has been described. Here, the information processing apparatus 100 may extract search query candidates based on various conditions.

  Specifically, the extraction unit 132 of the information processing apparatus 100 extracts a search query candidate for a subject that occupies an area of a predetermined ratio or more in the image received by the reception unit 131. For example, the information processing apparatus 100 extracts search query candidates by using a subject that occupies an area of 30% or more of the image received by the receiving unit 131 as a target for which the user desires a search query. As one aspect, the information processing apparatus 100 extracts search query candidates for each subject when there are a plurality of subjects occupying an area of a predetermined ratio or more. As another aspect, when the subject is a person, the information processing apparatus 100 extracts search query candidates for all persons depicted in the image.

  As described above, the information processing apparatus 100 according to the modified example extracts a search query candidate for a subject that occupies an area of a predetermined ratio or more in the image. Accordingly, the information processing apparatus 100 can extract a search query candidate for a subject that satisfies a certain level in the image, and thus extracts a search query candidate that enables a search desired by the user with high accuracy. be able to.

[2-4. Extract search query candidates based on location information)
In the above embodiment, the information processing apparatus 100 is based on the similarity between the feature amount of the image received by the receiving unit 131 and the feature amount of the reference image stored in the storage unit in association with the search query. An example of extracting search query candidates has been described. Here, the information processing apparatus 100 may extract search query candidates based on various types of information without being limited to the similarity.

  Specifically, the extraction unit 132 of the information processing apparatus 100 extracts search query candidates based on position information indicating the position where the image received by the reception unit 131 was captured. For example, the information processing apparatus 100 preferentially extracts search queries that are closely related to the location where the image was taken. As an example, when a building is depicted in the image, the information processing apparatus 100 extracts information on the building estimated from the place where the image was captured as a search query candidate.

  As described above, the information processing apparatus 100 according to the modification extracts search query candidates based on the position information indicating the position where the image is taken. As a result, the information processing apparatus 100 can extract search query candidates by utilizing position information, and therefore can extract search query candidates that enable a search desired by the user with high accuracy.

[2-5. Extract search query candidates based on time information)
In the above modification, the information processing apparatus 100 has been described by taking an example in which search query candidates are extracted based on position information indicating the position where the image received by the receiving unit 131 is captured. Here, the information processing apparatus 100 may extract search query candidates based on various types of information without being limited to the position information.

  Specifically, the extraction unit 132 of the information processing apparatus 100 extracts search query candidates based on time information indicating the time when the image received by the reception unit 131 was captured. For example, the information processing apparatus 100 preferentially extracts search queries that are closely related to the time at which an image was captured. As an example, the information processing apparatus 100 may be displayed in such a downtown area when an image depicting the downtown area is taken at a lunch time from 11:00 to 14:00 or a dinner time from 17:00 to 22:00. Information on existing restaurants is extracted as a search query candidate.

  In another example, the information processing apparatus 100 extracts search query candidates according to the season. As an example, when an image in which a mountain is depicted in summer, such as July to August, is captured, the information processing apparatus 100 extracts information related to mountain climbing as a search query candidate. On the other hand, when an image depicting a mountain is taken in winter, such as December to February, the information processing apparatus 100 extracts information related to winter sports in the mountain as a search query candidate.

  As described above, the information processing apparatus 100 according to the modification extracts search query candidates based on the time information indicating the time when the image was captured. As a result, the information processing apparatus 100 can extract search query candidates using location information, and therefore can extract search query candidates that enable a search desired by the user with high accuracy.

[2-6. list〕
In the above embodiment, the information processing apparatus 100 is based on the similarity between the feature amount of the image received by the receiving unit 131 and the feature amount of the reference image stored in the storage unit in association with the search query. An example of outputting a list in which search query candidates are displayed in order has been described. Here, the information processing apparatus 100 may output a list in which search query candidates in various data formats are displayed.

  Specifically, the output unit 133 of the information processing apparatus 100 outputs a list in which information on text data, image data, or audio data is arranged as a search query candidate. For example, the information processing apparatus 100 outputs a list in which keywords, images, and sounds are arranged as search query candidates to the terminal device 10.

  As described above, the information processing apparatus 100 according to the modification outputs a list in which information on text data, image data, or audio data is arranged as a search query candidate. As a result, the information processing apparatus 100 can output search query candidates based on the image received by the receiving unit 131 regardless of the data format, and thus presents a list of a variety of data to the user. can do.

[2-7. (Search query candidates are extracted based on features related to the subject)
In the above embodiment, the information processing apparatus 100 is based on the similarity between the feature amount of the face of the subject depicted in the image accepted by the accepting unit 131 and the feature amount of the face of the subject depicted in the reference image. In this example, search query candidates are extracted. Here, the information processing apparatus 100 may extract search query candidates based on various types of information, not limited to the face of the subject.

  Specifically, the extraction unit 133 of the information processing apparatus 100 extracts search query candidates estimated from the features related to the subject depicted in the image received by the reception unit 131. For example, the information processing apparatus 100 extracts search query candidates based on the face of the subject depicted in the image and the clothes worn by the subject. As an example, the information processing apparatus 100 searches for candidate names of subjects estimated from the subject's face and information about a appearing work such as a movie in which the subject estimated from clothes worn by the subject appears. Extract as

  As described above, the information processing apparatus 100 according to the modification extracts search query candidates estimated from the features related to the subject depicted in the image received by the receiving unit 131. Accordingly, the information processing apparatus 100 can extract search query candidates based on various characteristics of the subject, and thus can extract search query candidates desired by the user with higher accuracy.

[2-8. Display search query)
In the above embodiment, the information processing apparatus 100 is based on the similarity between the feature amount of the image received by the receiving unit 131 and the feature amount of the reference image stored in the storage unit in association with the search query. An example of outputting a list in which search query candidates are displayed in order has been described. Here, the information processing apparatus 100 may output search query candidates in various modes.

  Specifically, the output unit 133 of the information processing apparatus 100 outputs an image in which search query candidates are displayed on the image received by the receiving unit 131. For example, the information processing apparatus 100 outputs an image in which the search query candidate is displayed at a position based on the subject corresponding to the search query candidate on the image received by the reception unit 131. This point will be described with reference to FIG.

  FIG. 6 is a diagram illustrating an example of a display screen when there are a plurality of subjects. First, as shown in FIG. 6, it is assumed that the user has selected an image P2 in which the subjects Ob21 to Ob24 are drawn as desired search query images. In this case, the terminal apparatus 10 transmits the image P2 to the information processing apparatus 100. Thereby, the information processing apparatus 100 receives the image P2 from the terminal device 10.

  Subsequently, the information processing apparatus 100 extracts search query candidates for each of the subjects Ob21 to Ob24 depicted in the image P2. For example, the information processing apparatus 100 selects search query candidates SQ21 to SQ24 corresponding to the reference image having the highest degree of similarity between the facial feature amount of the subjects Ob21 to Ob24 and the facial feature amount depicted in the reference image. Extract each one.

  Thereafter, the information processing apparatus 100 outputs an image P3 in which the search query candidates SQ21 to SQ24 are displayed on the image P2 received by the receiving unit 131. Specifically, the information processing apparatus 100 places the search query candidates SQ21 to SQ24 at positions indicating the subjects Ob21 to Ob24 corresponding to the search query candidates SQ21 to SQ24 on the image P2 received by the reception unit 131. Each displayed image P3 is output. For example, the information processing apparatus 100 outputs an image in which search query candidates are displayed at front, back, left, and right positions close to the subject. As an example, as illustrated in FIG. 6, the information processing apparatus 100 outputs an image P3 in which a search query candidate SQ21 is displayed at a position above the subject Ob21. In another example, as illustrated in FIG. 6, the information processing apparatus 100 outputs an image P3 in which a search query candidate SQ23 is displayed at a position below the subject Ob23.

  As described above, the information processing apparatus 100 according to the modified example outputs an image in which search query candidates are displayed on the image received by the receiving unit 131. As a result, the information processing apparatus 100 can display search query candidates on the image received from the user, so that the user can easily grasp the search query candidates corresponding to the subject.

  Further, the information processing apparatus 100 according to the modification outputs an image in which the search query candidate is displayed at a position based on the subject corresponding to the search query candidate on the image received by the reception unit 131. As a result, the information processing apparatus 100 can display search query candidates at positions corresponding to the subject in the image received from the user, so that the user can easily grasp the search query candidates corresponding to the subject. be able to.

[2-9. (Display incidental information)
In the above modification, the information processing apparatus 100 has been described with an example in which the search query candidate is output on the image received by the receiving unit 131. Here, the information processing apparatus 100 may display the auxiliary information related to the similarity together with the search query candidates. For example, as illustrated in FIG. 1, the information processing apparatus 100 outputs, to the terminal device 10, a list in which the similarity corresponding to the search query is displayed together with the search query candidates.

  Further, the information processing apparatus 100 may output incidental information according to the similarity together with search query candidates. Specifically, the information processing apparatus 100 outputs a character string expressing the level of similarity together with search query candidates. For example, as illustrated in FIG. 6, the information processing apparatus 100 displays a character string “probably” when the similarity is relatively high, and “probably” when the similarity is relatively low. Output an image that displays the columns. In another example, as illustrated in FIG. 6, when the similarity is relatively high, the information processing apparatus 100 displays “!” At the end of the search query candidate, and the similarity is relatively low. Output an image displaying “?” At the end of the candidate search query.

  In another example, the information processing apparatus 100 outputs an image in which the density or shape of the search query candidate is changed according to the similarity. For example, the information processing apparatus 100 displays the search query candidates relatively dark when the similarity is relatively high, and displays the search query candidates relatively light when the similarity is relatively low. The image to be output is output. In another example, the information processing apparatus 100 displays search query candidates in a relatively large size when the similarity is relatively high, and displays the search query candidates when the similarity is relatively low. Output an image to be displayed in a relatively small size.

  As described above, the information processing apparatus 100 according to the modification includes the similarity between the feature amount of the image received by the reception unit 131 and the feature amount of the reference image stored in the storage unit in association with the search query. Attached information about is displayed along with search query candidates. As a result, the information processing apparatus 100 can display the index according to the similarity together with the search query candidate, so that the user can grasp the reliability of the search query candidate.

[2-10. Search query display format)
In the above-described modification, the information processing apparatus 100 has been described with an example of outputting an image in which a name corresponding to a subject is displayed as a search query candidate. Here, the information processing apparatus 100 may output an image displaying search query candidates in various formats, not limited to keywords such as names. This point will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of a display screen when the search query is an image. As illustrated in FIG. 7, first, the information processing apparatus 100 receives an image P <b> 2 in which the subjects Ob <b> 21 to Ob <b> 24 are drawn from the terminal device 10.

  Subsequently, the information processing apparatus 100 extracts the reference image PSQ21 having the highest similarity between the facial feature amount of the subject Ob21 and the facial feature amount depicted in the reference image as a search query candidate. Similarly, the information processing apparatus 100 uses the reference images PSQ22 to PSQ24 having the highest degree of similarity between the facial feature values of the subjects Ob22 to Ob24 and the facial feature values depicted in the reference image as search query candidates, respectively. Extract.

  Thereafter, the information processing apparatus 100 outputs an image P4 in which the reference images PSQ21 to PSQ24 are displayed on the image P2 received by the receiving unit 131. Specifically, the information processing apparatus 100 displays the reference images PSQ21 to PSQ24 at positions indicating the subjects Ob21 to Ob24 corresponding to the reference images PSQ21 to PSQ24 on the image P2 received by the reception unit 131, respectively. P4 is output.

  As described above, the information processing apparatus 100 according to the modified example outputs an image displaying search query candidates in various formats. Thereby, since the information processing apparatus 100 can display search query candidates regardless of the data format, the user can be made aware of search query candidates in a variety of manners. For example, the information processing apparatus 100 can display the reference image having the highest similarity as a search query candidate on the image and output it, so that the user can easily grasp the search query candidate.

[2-11. (Applicable)
In the above-described embodiment, an example in which the information processing apparatus 100 extracts a subject name as a search query candidate has been described. Here, the information processing apparatus 100 may output not only the name of the subject but also various types of information regarding the subject.

  For example, when the information processing apparatus 100 receives an image depicting food such as vegetables and fish, the information processing apparatus 100 extracts information on how to eat the food, a cooking method, an available place, and the like as search query candidates.

  In another example, when the information processing apparatus 100 accepts an image in which the sky is depicted, the information processing apparatus 100 extracts information about the weather as a search query candidate. As an example, the information processing apparatus 100 extracts, as search query candidates, the weather, temperature, humidity, and the like of a place where an image depicting the sky is captured.

  In another example, when the information processing apparatus 100 receives an image depicting a restaurant, the information processing apparatus 100 extracts a menu, a Web site, a coupon, and the like provided by the restaurant as search query candidates.

  In another example, when the information processing apparatus 100 receives an image depicting a restaurant street, the information processing device 100 extracts information about restaurants that exist in the restaurant street as search query candidates. As an example, the information processing apparatus 100 extracts a list of reviews and evaluation rankings of restaurants in the restaurant area as search query candidates.

  In another example, when the information processing apparatus 100 receives an image in which a product is drawn, the information processing device 100 extracts the price, sales location, similar product, word-of-mouth, sales maker, and the like of the product as search query candidates.

  As described above, the information processing apparatus 100 according to the modification outputs various types of information regarding the subject. As a result, the information processing apparatus 100 can output a variety of information about the subject, thereby enabling a search desired by the user.

[2-12. Effect of modification)
As described above, in the information processing apparatus 100 according to the modification, the output unit 133 outputs search query candidates based on the user's behavior history. Thereby, since the information processing apparatus 100 can output a search query related to the user's behavior, it is possible to present a search query that enables a search desired by the user.

  In the information processing apparatus 100 according to the modification, the output unit 133 outputs search query candidates based on the user search history or the history of messages about the user as the user action history. As a result, the information processing apparatus 100 can output a search query related to a keyword used by a user in a search or a message, and can therefore present a search query that enables a search desired by the user with high accuracy. .

  In the information processing apparatus 100 according to the modification, the reception unit 131 receives an image being displayed on the screen of the terminal device owned by the user as an image. Thereby, since the information processing apparatus 100 can accept the image being displayed on the screen, it is possible to reduce time and effort for the user to shoot the target for which the search query is requested.

  Further, in the information processing apparatus 100 according to the modification, the extraction unit 132 extracts a search query candidate for a subject that occupies an area of a predetermined ratio or more in the image. Accordingly, the information processing apparatus 100 can extract a search query candidate for a subject that satisfies a certain level in the image, and thus extracts a search query candidate that enables a search desired by the user with high accuracy. be able to.

  Further, in the information processing apparatus 100 according to the modification, the extraction unit 132 extracts search query candidates based on position information indicating a position where an image is taken. As a result, the information processing apparatus 100 can extract search query candidates by utilizing position information, and therefore can extract search query candidates that enable a search desired by the user with high accuracy.

  Further, in the information processing apparatus 100 according to the modification, the extraction unit 132 extracts search query candidates based on time information indicating the time at which an image was captured. As a result, the information processing apparatus 100 can extract search query candidates using location information, and therefore can extract search query candidates that enable a search desired by the user with high accuracy.

  In the information processing apparatus 100 according to the modification, the output unit 133 outputs a list in which information on text data, image data, or audio data is arranged as a search query candidate. As a result, the information processing apparatus 100 can output search query candidates based on the image received by the receiving unit 131 regardless of the data format, and thus presents a list of a variety of data to the user. can do.

  Further, in the information processing apparatus 100 according to the modification, the extraction unit 132 extracts search query candidates estimated from the characteristics related to the subject depicted in the image received by the reception unit 131. As a result, the information processing apparatus 100 can extract search query candidates based on various characteristics of the subject, so that the search query candidates that enable a search desired by the user can be extracted with higher accuracy. Can do.

  Further, in the information processing apparatus 100 according to the modification, the output unit 133 outputs an image in which search query candidates are displayed on an image received by the receiving unit 131. As a result, the information processing apparatus 100 can display search query candidates on the image received from the user, so that the user can easily grasp the search query candidates corresponding to the subject.

  In the information processing apparatus 100 according to the modification, the output unit 133 displays an image in which the search query candidate is displayed at a position based on the subject corresponding to the search query candidate on the image received by the reception unit 131. Output. As a result, the information processing apparatus 100 can display search query candidates at positions corresponding to the subject in the image received from the user, so that the user can easily grasp the search query candidates corresponding to the subject. be able to.

[3. Others]
In addition, among the processes described in the above embodiment, all or part of the processes described as being automatically performed can be performed manually, or the processes described as being performed manually can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  For example, the processing performed by the reception unit 131, the extraction unit 132, and the output unit 133 included in the information processing apparatus 100 described in the above embodiment may be performed by the terminal device 10. As an example, such processing is executed by an application installed in the terminal device 10. In this case, the terminal device 10 outputs an image receiving unit, an extracting unit that extracts search query candidates based on the image received by the receiving unit, and a search query candidate extracted by the extracting unit. And an output unit.

  Specifically, the terminal device 10 first accepts an image by performing a selection operation by the user. Subsequently, the terminal device 10 extracts search query candidates based on the received image. For example, the terminal device 10 extracts search query candidates based on the similarity between the received image feature amount and the reference image feature amount associated with the search query and stored in the storage unit. . As an example, the terminal device 10 extracts, as candidates, a search query corresponding to a reference image whose similarity with an image is equal to or greater than a predetermined threshold. Thereafter, the terminal device 10 outputs the extracted search query candidates. For example, the terminal device 10 outputs a list in which search query candidates are displayed in an order based on the similarity. As an example, the terminal device 10 outputs a list in which search query candidates are displayed from the top in descending order of similarity on a display screen included in the terminal device 10. Thereby, the terminal device 10 can extract and display the search query candidates based on the image without performing communication with the server. In other words, it is possible to realize processing for outputting search query candidates based on images in the terminal device 10.

  In another example, the processing performed by the reception unit 131 and the output unit 133 described in the above embodiment may be performed by the terminal device 10, and the processing performed by the extraction unit 132 may be performed by the information processing device 100. In this case, the information processing apparatus 100 includes the extraction unit 132 without including the reception unit 131 and the output unit 133. The terminal device 10 includes a reception unit that receives an image and an output unit that outputs search query candidates extracted by the extraction unit 132 of the information processing device 100.

  Specifically, the terminal device 10 first accepts an image by performing a selection operation by the user. Subsequently, the terminal apparatus 10 transmits information regarding the received image to the information processing apparatus 100. For example, the terminal device 10 transmits the image itself such as the feature amount of the image to the information processing apparatus 100 as information about the image. As a result, the information processing apparatus 100 receives information about the image from the terminal apparatus 10. Then, the information processing apparatus 100 extracts search query candidates based on the received information about the image. For example, the information processing apparatus 100 extracts a search query candidate based on the similarity between the received image feature amount and the reference image feature amount associated with the search query and stored in the storage unit. . Thereafter, the information processing apparatus 100 transmits information related to the extracted search query candidates to the terminal apparatus 10. As a result, the terminal device 10 receives information related to the search query candidate from the information processing device 100. Then, the terminal device 10 outputs the search query candidates extracted by the information processing device 100 to the display screen included in the terminal device 10. Thereby, since the information processing apparatus 100 can extract search query candidates on the server side, the processing load on the terminal apparatus 10 side can be reduced.

  In another example, the search query information storage unit 121 illustrated in FIG. 3 may be stored in a storage server or the like without being stored in the information processing apparatus 100. In this case, the information processing apparatus 100 acquires information related to the search query by accessing the storage server.

  Further, the information processing apparatus 100 may be an information processing apparatus that performs only an extraction process without performing an output process. In this case, the information processing apparatus 100 does not have the output unit 133. Then, the output device having the output unit 133 outputs search query candidates extracted by the information processing device 100.

  Further, the information processing apparatus 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in FIG. 8, for example. Hereinafter, the information processing apparatus 100 will be described as an example. FIG. 8 is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of the information processing apparatus 100. The computer 1000 includes a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

  The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

  The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. The communication interface 1500 receives data from other devices via the network N and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

  The CPU 1100 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. In addition, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

  The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disk) or PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. It is.

  For example, when the computer 1000 functions as the information processing apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the function of the control unit 130 by executing a program loaded on the RAM 1200. The HDD 1400 stores data in the storage unit 120. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them. However, as another example, these programs may be acquired from other devices via the network N.

  As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are merely examples, and various modifications based on the knowledge of those skilled in the art, including the aspects described in the Summary of Invention section. It is possible to implement the present invention in other forms with improvements.

  In addition, the information processing apparatus 100 described above may be realized by a plurality of server computers, and depending on the function, an external platform or the like may be realized by calling an API (Application Programming Interface) or network computing. Can be changed flexibly.

  Further, “section (module, unit)” described in the claims can be read as “means”, “circuit”, and the like. For example, the reception unit can be read as reception means or a reception circuit.

DESCRIPTION OF SYMBOLS 1 Presentation system 10 Terminal device 50 Search server 100 Information processing apparatus 121 Search query information storage part 131 Reception part 132 Extraction part 133 Output part

Claims (14)

A reception unit for receiving images;
An extraction unit that extracts search query candidates based on the image received by the reception unit;
An output unit that outputs a query from search query candidates extracted by the extraction unit based on a web service usage history or an application usage history before the time when the image is received by the reception unit. An information processing apparatus characterized by that. The extraction unit includes:
A feature value of the image, to claim 1, characterized in that to extract the candidate of a search query based on the similarity between the feature quantity of the reference image stored in the storage unit associated with the search query The information processing apparatus described. The output unit is
Outputting a list in which candidates for the search query are displayed in an order based on the similarity between the feature amount of the image and the feature amount of the reference image associated with the search query and stored in the storage unit the information processing apparatus according to claim 1 or 2, characterized in. The reception unit
The image as information processing apparatus according to any one of claims 1-3, characterized in that to accept the captured image is searched was taken. The reception unit
The image as information processing apparatus according to any one of claims 1-4, characterized in that to accept the image being displayed on the screen of the terminal device having a user. The extraction unit includes:
The information processing apparatus according to any one of claims 1-5, characterized in that to extract the candidate of a search query of the subject occupies a predetermined proportion or more of the area of the image. The extraction unit includes:
Based on the position information indicating the position where the image was captured, the information processing apparatus according to any one of claims 1-6, characterized in that to extract the candidate of a search query. The extraction unit includes:
Based on the time information indicating the time at which the image was captured, the information processing apparatus according to any one of claims 1-7, characterized in that to extract the candidate of a search query. The output unit is
The information processing apparatus according to any one of claims 1 to 8 , wherein a list in which information on text data, image data, or audio data is arranged is output as the search query candidate. The extraction unit includes:
The information processing apparatus according to any one of claims 1-9, characterized in that to extract the candidate of a search query which is estimated from the feature associated with the subject being depicted in the image. The output unit is
The information processing apparatus according to any one of claims 1 to 10 , wherein an image in which the search query candidates are displayed on the image is output. The output unit is
The information according to any one of claims 1 to 11 , wherein an image displaying the search query candidate at a position based on a subject corresponding to the search query candidate is output on the image. Processing equipment. An information processing method executed by an information processing apparatus,
A reception process for receiving images;
An extraction step of extracting search query candidates based on the image received by the reception step;
An output step of outputting a query from search query candidates extracted by the extraction step based on a web service usage history or an application usage history before the time when the image is received by the receiving step. An information processing method characterized by that. Acceptance procedure to accept images,
An extraction procedure for extracting search query candidates based on the image accepted by the acceptance procedure;
An output procedure for outputting a query from the search query candidates extracted by the extraction procedure based on a web service usage history or an application usage history before the time when the image is received by the reception procedure. An information processing program to be executed.

Images