JP2020119177A - Wearable search system, search method, and search program - Google Patents

Abstract

To provide a wearable search system, a search method and a search program which do not require user operation for a search.SOLUTION: A wearable search system 100 comprises: camera-equipped eyeglasses 200 to detect the focus of a user U and photograph an object S present on the focus of the user U; a portable terminal 300 carried by the user U; and a server 400 connected to the camera-equipped eyeglasses 200 and the portable terminal 300 over a communications network N. The camera-equipped eyeglasses 200 transmit image information and location information of the object S (focus information and location information of the user U) to the server 400. The server 400 searches a data table on the basis of the image information and location information of the object S transmitted from the camera-equipped eyeglasses 200, thereby identifying the object S and extracting object information, and transmits the extracted object information to the portable terminal 300. The portable terminal 300 displays the object information on a display unit 301.SELECTED DRAWING: Figure 1

Description

The present invention relates to a wearable search system, a search method, and a search program.

In Patent Document 1, a wearable motion sensor that detects whether or not a user has made a predetermined gesture, a glasses-type transmissive HMD (Head Mounted Display), and a wearable camera that is built in the transmissive HMD. A search system is described. In the wearable search system described in Patent Document 1, when the user makes a predetermined gesture for searching, the wearable motion sensor detects the gesture and the camera takes a picture of the camera in the visual field direction of the user. Then, the search target is extracted from the image captured by the camera, the extracted search target image is transmitted to the image recognition server, and the search is executed in the image recognition server. In addition, the search result is displayed on the transparent HMD.

JP, 2016-115125, A

However, the wearable search system described in Patent Document 1 requires a search operation by the user, such as device operation, designation of an object, and gesture. That is, unless the user has a sense of purpose of “searching”, the search cannot be performed. Therefore, there is a problem that the user cannot unknowingly provide the information about the object “looking”.

An object of the present invention is to provide a wearable search system, a search method, and a search program that do not require a user's operation for search.

A wearable search system according to a first aspect of the present invention detects a user's focus, and wears camera-equipped glasses for photographing an object existing on the focus of the user, and a mobile terminal carried by the user, The camera-equipped glasses and a server connected to the mobile terminal via a communication network, the camera-equipped glasses together with image information of the photographed object, focus information regarding the focus of the user, and the position of the user. Data in which information is transmitted to the server as position information of the object, and the server associates the image information of the object, the position information of the object, and the object information about the object with each other. The table is searched based on the image information and the position information of the object transmitted from the glasses with the camera to specify the object and extract the object information, and the extracted object The information is transmitted to the mobile terminal, and the mobile terminal displays the object information on the display unit of the mobile terminal.

A search method according to a second aspect of the present invention is a pair of glasses with a camera worn by a user, a mobile terminal carried by the user, a server connected to the glasses with a camera and the mobile terminal by a communication network, A wearable search system including a search method, wherein the camera-equipped glasses detect a focus of the user, photograph an object existing on the focus of the user, and photograph the object. Together with the image information of the user, focus information regarding the focus of the user and position information of the user are transmitted to the server as position information of the target object, and the server sends the image information of the target object and the target object. By searching a data table in which position information and target object information regarding the target object are associated with each other based on the image information and the position information of the target object transmitted from the glasses with camera, the target The object is specified, the object information is extracted, the extracted object information is transmitted to the mobile terminal, and the mobile terminal displays the object information on the display unit of the mobile terminal.

A search program according to a third aspect of the present invention is a pair of glasses with a camera worn by a user, a mobile terminal carried by the user, a server with the glasses with a camera and the mobile terminal connected by a communication network, A search program executed in a wearable search system comprising: the camera-attached glasses that detect the focus of the user, photograph an object existing on the focus of the user, and photograph the object. Together with the image information of, the focus information regarding the focus of the user and the position information of the user are transmitted to the server as the position information of the target object, and the server performs the image information of the target object, Retrieving a data table in which the position information of the target object and the target object information regarding the target object are associated with each other based on the image information and the position information of the target object transmitted from the glasses with a camera. To specify the target object, extract the target object information, and execute a process of transmitting the extracted target object information to the mobile terminal, and display the target object information on the mobile terminal on the mobile terminal. Causes the process displayed on the department to be executed.

A wearable search system, a search method, and a search program that do not require a user's operation for search can be provided.

It is a figure which shows an example of the wearable search system which concerns on Embodiment 1 of this invention. 5 is a flowchart showing an example of a search method implemented in the wearable search system according to the first embodiment of the present invention. 5 is a flowchart showing an example of a search method implemented in the wearable search system according to the first embodiment of the present invention.

Embodiment 1
Hereinafter, Embodiment 1 of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing an example of wearable search system 100 according to Embodiment 1 of the present invention. As shown in FIG. 1, the wearable search system 100 includes camera-equipped glasses 200 worn by a user U, a mobile terminal 300 carried by the user U, and a server 400. Further, the server 400 is connected to the glasses with camera 200 and the mobile terminal 300 by the communication network N.

The camera-equipped glasses 200 are glasses equipped with a camera and include a control unit (not shown). The control unit includes a CPU (Central Processing Unit) (not shown), a storage unit (not shown), and the like. Then, the CPU executes the program stored in the storage unit to implement all the processes in the control unit.
In addition, the program stored in each storage unit of the control unit includes a code that is executed by the CPU to realize the processing in each of the control units. The storage unit is configured to include, for example, an arbitrary storage device capable of storing this program and various kinds of information used for processing in the control unit. The storage device is, for example, a memory or the like.

The camera-equipped glasses 200 detect the focus of the eye of the user U. The detection of the focus of the eye of the user U by the camera-equipped glasses 200 can be realized by using a known technique. Specifically, the camera-equipped glasses 200 may detect the focus by irradiating the eye of the user U with a marker by infrared light to measure the eye refractive power. Here, detecting the focus of the eye of the user U means detecting a distance from the camera-equipped glasses 200 to the focus of the eye of the user U (hereinafter, also referred to as a focal length).

Moreover, the camera-equipped glasses 200 capture an image of the object S existing on the focus of the user U. Specifically, the camera-equipped eyeglasses capture an image by aligning the focus position of the camera mounted on the camera-equipped eyeglass with the focus position of the eye of the user U.

Further, the camera-equipped glasses 200 acquire the position information of the user U. Specifically, the camera-equipped glasses 200 receive a signal from, for example, a GPS (Global Positioning System) satellite, and acquire position information of the user U. Similarly, when the mobile terminal 300 can acquire the position information of the user U, the camera-equipped glasses 200 may receive the position information of the user U from the mobile terminal 300 via the communication network N.

The camera-equipped glasses 200 also send the image information of the photographed object S to the server. In addition, the camera-equipped glasses 200 transmit, to the server 400, as the position information of the target S, the focus information regarding the detected focal length of the eye of the user U and the position information of the user U together with the image information of the target S.

The server 400 is a part responsible for a search process in the wearable search system 100, and includes a control unit (not shown). The control unit includes a CPU (Central Processing Unit) (not shown), a storage unit (not shown), and the like. Then, the CPU executes the program stored in the storage unit to implement all the processes in the control unit.
In addition, the program stored in each storage unit of the control unit includes a code that is executed by the CPU to realize the processing in each of the control units. The storage unit is configured to include, for example, an arbitrary storage device capable of storing this program and various kinds of information used for processing in the control unit. The storage device is, for example, a memory or the like.

The server 400 relates to the image information of the target S, the position information of the target S, and the target S based on the image information of the target S and the position information of the target S transmitted from the glasses 200 with camera. The data table in which the target object information and the related information related to the target object S are associated with each other is searched. Specifically, from the data table, the image information and the position information of the target S that match the image information and the position information of the target S transmitted from the camera-equipped glasses 200 are extracted. As a result, the server 400 identifies the target S and extracts target information and related information of the target S. Here, the object information of the object S is, for example, the name of the building when the object S is a building or the like. Further, the related information of the target object S is, for example, when the target object S is a building or the like, the name and type of the store occupying the building. Here, the type of store includes a pharmacy, a furniture store, a general store, a clothing store, a decorative article store, a specialized tool store, and the like. The data table may be stored in the storage unit of the server 400 or may be stored in an external storage device.
In addition, when the server 400 compares the image information of the target S transmitted from the glasses with camera 200 with the image information of the target S stored in the data table, the server 400 uses Various image processing such as extraction of feature points is performed on the transmitted image information of the object S and the image information of the object S stored in the data table. As a result, the server 400 detects the image information of the target S that matches the image information of the target S transmitted from the glasses with camera 200 from the data table.
Further, the server 400 transmits the extracted object information of the object S and the related information to the mobile terminal 300 of the user U.

The mobile terminal 300 is a part that presents the search result of the server 400 to the user U, and includes a control unit (not shown). The control unit includes a CPU (Central Processing Unit) (not shown), a storage unit (not shown), and the like. Then, the CPU executes the program stored in the storage unit to implement all the processes in the control unit.
In addition, the program stored in each storage unit of the control unit includes a code that is executed by the CPU to realize the processing in each of the control units. The storage unit is configured to include, for example, an arbitrary storage device capable of storing this program and various kinds of information used for processing in the control unit. The storage device is, for example, a memory or the like.

The mobile terminal 300 includes a display unit 301, and displays the target object information and the related information of the target object S transmitted from the server 400 on the display unit 301.
In addition, the mobile terminal 300 selects related information useful for the user U from the related information transmitted from the server 400 based on the search history of the mobile terminal 300, and displays the selected related information together with the object information on the display unit. It may be displayed on 301. Specifically, the mobile terminal 300 selects, from the related information transmitted from the server 400, related information that is highly related to the search result of which the number of searches is greater than the predetermined number of times in the search history of the mobile terminal 300. For example, when the number of times furniture is searched in the search history of the mobile terminal 300 is more than a predetermined number of times, the mobile terminal 300, if the related information of the object S includes the information of the furniture store, displays the information of the furniture store. It is displayed on the display unit 301.

The mobile terminal 300 also transmits to the server 400 a stop command for stopping the transmission of the object information and the related information from the server 400. Specifically, when the distribution stop is set by the user U operating the mobile terminal 300, the mobile terminal 300 transmits a distribution stop command to the server 400.

Next, a search method implemented in the wearable search system 100 will be described with reference to FIGS. 2 and 3.
First, the camera-equipped glasses 200 detect the focus of the eye of the user U and acquire focus information regarding the focal length (step S101).

Next, the camera-equipped glasses 200 capture an image of the object S existing on the focus of the user U and acquire image information of the object S (step S102).

Next, the camera-equipped glasses 200 receive signals from GPS satellites and acquire the position information of the user U (step S103). Note that the camera-equipped glasses 200 may acquire the position information of the user U from the mobile terminal 300.

Next, the camera-equipped glasses 200 transmit the detected focus information of the user U and the position information of the user U to the server 400 as the position information of the target S together with the image information of the target S (step S104). If the information cannot be obtained in steps S101 to S103, or if the information cannot be transmitted to the server 400 in step S104, the camera-equipped glasses 200 informs the mobile terminal 300 to that effect. You may send it. In this case, the mobile terminal 300 may display “Not searchable” on the display unit 301.

Next, the server 400 receives the image information of the target S and the position information of the target S (focus information of the user U and position information of the user U) transmitted from the glasses 200 with camera (step S105).

Next, the server 400 performs various image processing such as extraction of feature points on the image information of the object S transmitted from the glasses with camera 200 and the image information of the object S stored in the data table. Is performed (step S106).

Next, the server 400 matches the image information and the position information of the target S transmitted from the camera-equipped glasses 200 from the data table, based on the information obtained in step S106 and the position information of the target S. The image information and the position information of the object S are detected. As a result, the server 400 identifies the target S (step S107). If the target S cannot be specified in step S107, the server 400 may send a message to that effect to the mobile terminal 300. In this case, the mobile terminal 300 may display “Not searchable” on the display unit 301.

Next, the server 400 extracts the object information and the related information of the object S identified in step S107 from the data table (step S108).

Next, the server 400 transmits the target object information and the related information of the target object S extracted in step S108 to the mobile terminal 300 (step S109).

Next, the mobile terminal 300 receives the target object information and the related information of the target object S transmitted from the server 400 (step S110).

Next, the mobile terminal 300 selects related information useful for the user U from the related information transmitted from the server 400 based on the search history of the mobile terminal 300 (step S111) and targets the selected related information. It is displayed on the display unit 301 together with the object information (step S112).

Next, the mobile terminal 300 determines whether the distribution stop is set by the user U operating the mobile terminal 300 (step S113).
In step S113, when the distribution stop is not set by the user U operating the mobile terminal 300 (step S113; No), this process is repeated.
In step S113, when the distribution stop is set by the user U operating the mobile terminal 300 (step S113; Yes), the mobile terminal 300 transmits a distribution stop command to the server 400 (step S114). ..

Next, the server 400 receives the distribution stop command from the mobile terminal 300 (step S115), and stops the transmission of the object information and the related information to the mobile terminal 300 (step S116).

According to the wearable search system 100, the search method, and the search program according to the first embodiment described above, the camera-equipped glasses 200 automatically acquire the image information of the target object S located on the focus of the user U. , The image information of the target S and the position information of the target S (position information and focus information of the user U) are transmitted, and the server 400 transmits the image information and the position information of the target S transmitted from the glasses with camera 200. Based on the above, the target object information and the related information of the target object S are searched from the data table, and the target object information and the related information are displayed on the display unit 301 of the mobile terminal 300. That is, even if the user U does not have the purpose of “searching”, the object S existing on the focus of the user U can be searched. Accordingly, it is possible to provide the wearable search system 100, the search method, and the search program that do not require the operation of the user U for the search.

Further, since the server 400 retrieves not only the target information of the target S but also the related information, not only the target S is specified but also the related information of the specified target S is presented to the user U. be able to.

Further, since the portable terminal 300 selects the relevant information useful for the user U from the relevant information transmitted from the server 400, the relevant information useful for the user U can be presented.

Further, when the mobile terminal 300 is operated by the user U, the mobile terminal 300 transmits a distribution stop command to the server 400 to stop the transmission of the target object information and related information from the server 400. When the related information does not need to be distributed, the distribution can be stopped.

The present invention is not limited to the above-mentioned embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, in Embodiment 1 above, the camera-equipped glasses 200 may be transmissive HMDs. In this case, the object information and the related information of the object S may be transmitted from the server 400 to the camera-equipped glasses 200, and the object information and the related information may be displayed on the camera-equipped glasses 200.

Further, although the present invention has been described as a hardware configuration in the above-described embodiments, the present invention is not limited to this. The present invention can also be realized by causing a CPU (Central Processing Unit) to execute a computer program, the processing procedure described in the flowcharts of FIGS. 2 and 3.
In addition, the program described above can be stored using various types of non-transitory computer readable mediums and supplied to the computer. Non-transitory computer readable media include various types of tangible storage media. Examples of the non-transitory computer-readable medium are magnetic recording media (for example, flexible disk, magnetic tape, hard disk drive), magneto-optical recording media (for example, magneto-optical disk), CD-ROM (Read Only Memory) CD-R, CD. -R/W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)) are included. In addition, the program may be supplied to the computer by various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

The whole or part of the exemplary embodiments disclosed above can be described as, but not limited to, the following supplementary notes.
(Appendix 1)
Detecting the focus of the user, and camera-equipped glasses for photographing an object existing on the focus of the user,
A mobile terminal carried by the user,
A server connected to the camera-equipped glasses and the mobile terminal by a communication network,
Equipped with
The camera-equipped glasses, together with the image information of the imaged object, transmits focus information regarding the focus of the user and position information of the user to the server as position information of the object,
The server transmits a data table in which the image information of the target object, the position information of the target object, and the target object information regarding the target object are associated with each other. By searching on the basis of the image information and the position information, the target object is identified and the target object information is extracted, and the extracted target object information is transmitted to the mobile terminal,
The wearable search system, wherein the mobile terminal displays the object information on a display unit of the mobile terminal.
(Appendix 2)
In the data table, the image information of the target object, the position information of the target object, the target object information regarding the target object, and related information related to the target object are associated with each other,
The server identifies the target object by searching the data table based on the image information and the position information of the target object transmitted from the glasses with camera, and at the same time, the target object information and the relation Extracting information, transmitting the extracted object information and the related information to the mobile terminal,
The wearable search system according to appendix 1, wherein the mobile terminal displays the object information and the related information on a display unit of the mobile terminal.
(Appendix 3)
The mobile terminal selects the related information useful for the user from the related information transmitted from the server based on a search history of the mobile terminal, and selects the selected related information together with the object information. The wearable search system according to appendix 2, which is displayed on the display unit.
(Appendix 4)
4. The wearable search system according to appendix 2 or 3, wherein the mobile terminal transmits a stop command for stopping transmission of the object information and the related information from the server to the server.
(Appendix 5)
A search method implemented in a wearable search system, comprising: glasses with a camera worn by a user, a mobile terminal carried by the user, and a server connected to the glasses with the camera and the mobile terminal via a communication network. hand,
The camera-equipped glasses detect the focus of the user, photograph an object existing on the focus of the user, and together with image information of the photographed object, focus information regarding the focus of the user and the Transmitting the user's position information to the server as the position information of the object,
The server transmits a data table in which the image information of the target object, the position information of the target object, and the target object information related to the target object are associated with each other. By searching on the basis of the image information and the position information, the target object is identified and the target object information is extracted, and the extracted target object information is transmitted to the mobile terminal,
The search method, wherein the mobile terminal displays the object information on a display unit of the mobile terminal.
(Appendix 6)
In the data table, the image information of the target object, the position information of the target object, the target object information regarding the target object, and related information related to the target object are associated with each other,
The server identifies the target object by searching the data table based on the image information and the position information of the target object transmitted from the glasses with camera, and at the same time, the target object information and the relation. Extracting information, transmitting the extracted object information and the related information to the mobile terminal,
6. The search method according to appendix 5, wherein the mobile terminal displays the object information and the related information on a display unit of the mobile terminal.
(Appendix 7)
The mobile terminal selects the related information useful for the user from the related information transmitted from the server based on the search history of the mobile terminal, and selects the selected related information together with the object information. The search method according to appendix 6, which is displayed on the display unit.
(Appendix 8)
8. The search method according to appendix 6 or 7, wherein the mobile terminal transmits a stop command for stopping transmission of the object information and the related information from the server to the server.
(Appendix 9)
It is a search program executed in a wearable search system comprising: glasses with a camera worn by a user, a mobile terminal carried by the user, and a server connected to the glasses with the camera and the mobile terminal via a communication network. hand,
The camera-attached glasses detect the focus of the user, photograph an object existing on the focus of the user, and together with image information of the photographed object, focus information regarding the focus of the user and the Causing a process of transmitting the position information of the user to the server as the position information of the object,
A data table in which the image information of the target object, position information of the target object, and target object information related to the target object are associated with the server, and the target object transmitted from the camera-equipped glasses is displayed. By performing a search based on the image information and the position information, the target object is specified and the target object information is extracted, and a process of transmitting the extracted target object information to the mobile terminal is executed,
A search program that causes the mobile terminal to execute a process of displaying the object information on a display unit of the mobile terminal.
(Appendix 10)
In the data table, the image information of the target object, the position information of the target object, the target object information regarding the target object, and related information related to the target object are associated with each other,
The server is searched for the data table based on the image information and the position information of the object transmitted from the glasses with a camera to specify the object and the object information and the relation. Extracting information, and executing a process of transmitting the extracted object information and the related information to the mobile terminal,
10. The search program according to appendix 9, which causes the mobile terminal to execute a process of displaying the object information and the related information on a display unit of the mobile terminal.
(Appendix 11)
In the mobile terminal, based on the search history in the mobile terminal, from the related information transmitted from the server, the related information useful for the user is selected, and the selected related information together with the object information The search program according to appendix 10, which causes the display unit to execute the processing to be displayed.
(Appendix 12)
12. The search program according to supplementary note 10 or 11, which causes the mobile terminal to execute a process of transmitting a stop instruction to the server to stop transmission of the object information and the related information from the server.

100 Wearable Search System 200 Glasses with Camera 300 Mobile Terminal 301 Display 400 Server S Target U User

Claims (10)

Detecting the focus of the user, and camera-equipped glasses for photographing an object existing on the focus of the user,
A mobile terminal carried by the user,
A server connected to the camera-equipped glasses and the mobile terminal by a communication network,
Equipped with
The camera-equipped glasses, together with the image information of the imaged object, transmits focus information regarding the focus of the user and position information of the user to the server as position information of the object,
The server transmits a data table in which the image information of the target object, the position information of the target object, and the target object information regarding the target object are associated with each other. By searching on the basis of the image information and the position information, the target object is identified and the target object information is extracted, and the extracted target object information is transmitted to the mobile terminal,
The wearable search system, wherein the mobile terminal displays the object information on a display unit of the mobile terminal. In the data table, the image information of the target object, the position information of the target object, the target object information regarding the target object, and related information related to the target object are associated with each other,
The server identifies the target object by searching the data table based on the image information and the position information of the target object transmitted from the glasses with camera, and at the same time, the target object information and the relation Extracting information, transmitting the extracted object information and the related information to the mobile terminal,
The wearable search system according to claim 1, wherein the mobile terminal displays the object information and the related information on a display unit of the mobile terminal. The mobile terminal selects the related information useful for the user from the related information transmitted from the server based on a search history of the mobile terminal, and selects the selected related information together with the object information. The wearable search system according to claim 2, which is displayed on the display unit. The wearable search system according to claim 2, wherein the mobile terminal transmits a stop command for stopping transmission of the object information and the related information from the server to the server. A search method implemented in a wearable search system, comprising: glasses with a camera worn by a user, a mobile terminal carried by the user, and a server connected to the glasses with the camera and the mobile terminal via a communication network. hand,
The camera-equipped glasses detect the focus of the user, photograph an object existing on the focus of the user, and together with image information of the photographed object, focus information regarding the focus of the user and the Transmitting the user's position information to the server as the position information of the object,
The server transmits a data table in which the image information of the target object, the position information of the target object, and the target object information related to the target object are associated with each other. By searching on the basis of the image information and the position information, the target object is identified and the target object information is extracted, and the extracted target object information is transmitted to the mobile terminal,
The search method, wherein the mobile terminal displays the object information on a display unit of the mobile terminal. In the data table, the image information of the target object, the position information of the target object, the target object information regarding the target object, and related information related to the target object are associated with each other,
The server identifies the target object by searching the data table based on the image information and the position information of the target object transmitted from the glasses with camera, and at the same time, the target object information and the relation. Extracting information, transmitting the extracted object information and the related information to the mobile terminal,
The search method according to claim 5, wherein the mobile terminal displays the object information and the related information on a display unit of the mobile terminal. The mobile terminal selects the related information useful for the user from the related information transmitted from the server based on the search history of the mobile terminal, and selects the selected related information together with the object information. The search method according to claim 6, which is displayed on the display unit. The search method according to claim 6 or 7, wherein the mobile terminal transmits a stop instruction for stopping transmission of the object information and the related information from the server to the server. It is a search program executed in a wearable search system comprising: glasses with a camera worn by a user, a mobile terminal carried by the user, and a server connected to the glasses with the camera and the mobile terminal via a communication network. hand,
The camera-attached glasses detect the focus of the user, photograph an object existing on the focus of the user, and together with image information of the photographed object, focus information regarding the focus of the user and the Causing a process of transmitting the position information of the user to the server as the position information of the object,
A data table in which the image information of the target object, position information of the target object, and target object information related to the target object are associated with the server, and the target object transmitted from the camera-equipped glasses is displayed. By performing a search based on the image information and the position information, the target object is specified and the target object information is extracted, and a process of transmitting the extracted target object information to the mobile terminal is executed,
A search program that causes the mobile terminal to execute a process of displaying the object information on a display unit of the mobile terminal. In the data table, the image information of the target object, the position information of the target object, the target object information regarding the target object, and related information related to the target object are associated with each other,
The server is searched for the data table based on the image information and the position information of the object transmitted from the glasses with a camera to specify the object and the object information and the relation. Extracting information, and executing a process of transmitting the extracted object information and the related information to the mobile terminal,
The search program according to claim 9, which causes the mobile terminal to execute a process of displaying the object information and the related information on a display unit of the mobile terminal.

Images