JP2016115125A - Wearable search system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a wearable search system configured to search for an image with less power consumption, in indoor and outdoor use, only by intuitive action, without requiring voice or complicated operation.SOLUTION: A wearable search system includes a wearable computer 2 having a transparent display and a wearable action sensor 1. The wearable action sensor 1 identifies a search request action of a user, and transmits a start request to the wearable computer. The wearable computer 2 has a camera function for displaying a camera image on the transparent display, biases the camera function in response to the start request from the wearable action sensor 1, recognizes a hand area on the camera image, extracts a search image from a search image range having a predetermined positional relationship with the hand area, transmits a search request including the search image to an image recognition server 3 to execute searching, and overlay-displays a search result transmitted from the image recognition server 3 on the transparent display.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a wearable search system that extracts a search target from a camera image, executes an image search, and displays a search result on a transparent display of a wearable computer. The present invention relates to a wearable search system for starting an image search involving a message.

  With the spread of smartphones, users can easily obtain information anytime and anywhere. However, although it is not difficult to operate the smartphone, it is necessary to perform operations such as turning on the screen display or starting a camera or an application each time an image search is executed.

  For example, when you want to investigate the details of a new product at an electronics store, take out the smartphone from the pocket and turn on the screen display, then launch the browser and enter the product name, or start the camera and QR code (registered trademark) An operation procedure such as shooting is required.

  On the other hand, in recent years, scouter-type transmissive HMD that can be worn all the time has attracted attention, and various contents can be displayed through a small display in the field of view without blocking the user's line of sight, and information can be searched more timely. Became possible.

  Patent Document 1 discloses Google Glass (registered trademark) of Google Inc. as such a transmission type HMD. In "Google Glass", it is recommended to use a dialogue interface that can handle voice input and output and touch operation. For example, the screen of the glass is turned on with a shout of “OK, glass”, and the selection / confirmation action on the screen is performed by touching the touch pad on the temple of the glasses while reading out the voice command.

  Non-Patent Document 1 discloses an optical transmission type in which an image is projected onto a light guide plate arranged at a short distance, such as a lens of glasses, and the outside can be seen through while displaying the image on the surface of the glasses. HMD's “MOVERIO” is disclosed. By using such a device, it is possible to secure a field of view and additionally display related information on a search target that exists in the field of view on the HMD.

  Patent Document 2 discloses a method of adding and presenting related information to an object on a field of view specified by image recognition using a transmissive HMD camera. However, “MOVERIO” does not provide an audio interface, and operation with a remote control on the touch panel connected to the screen switch ON or glasses with a cable is essential.

  Patent Document 3 discloses a method of presenting information corresponding to a user's position and traveling direction to a user wearing a transmissive HMD. In this method, after identifying the user position by GPS, the relevant information for each object in the first determination area that does not include the direct direction of the traveling direction, and each of the second determination area that includes the direct direction of the traveling direction Can be provided. In addition, since this method is a push type, no operation other than the switch ON of the screen is required.

United States Patent Application No.0130044042 Japanese Patent No. 05215211 JP 2012-78224

See-through HMD Epson MOVERIO (Seiko Epson Corporation) http://www.epson.jp/products/moverio/

  According to Google Glass user test results, there is a sense of resistance to verifying information aloud to devices in public. Also, when using touch operations on voice when shooting with a camera or selecting photos, it is not intuitive to touch an appropriate place on an invisible vine, move left or right, or trace from top to bottom. There is no indication that it is complicated.

  According to “MOVERIO” of Non-Patent Document 1, it is possible to realize information presentation in which related information is additionally displayed on an object in view. However, the recognition target cannot be specified (designated) when a plurality of objects are present in the user's field of view. In addition, since “MOVERIO” requires touch operations with the remote control at hand, operations such as turning on the screen display of the glass, starting applications, and selecting up, down, left, and right are the same as with Google Glass. Similarly, it is not intuitive and complicated.

  Furthermore, in general use of these glasses devices, the battery is theoretically held for about one day, but if the camera is always used for recognition, there are problems of overheating and drastic increase in battery consumption, so frequent on / off operation is required. It becomes.

  In Patent Document 3, it is possible to estimate the user's line of sight by using the position information obtained by GPS and the traveling direction of the user, and automatically push and provide information on the object corresponding thereto. However, the problem that the user pushes the related information even if the user does not actually have interest in the object is not solved when the user passes by the object or unconsciously hits it. Also, since this method is based on outdoor position measurement using GPS, it cannot be used for indoor use scenes such as in-store product surveys.

  An object of the present invention is to provide a wearable search system that solves the above technical problem and enables image search with low power consumption both indoors and outdoors, with only intuitive operation without requiring utterance or complicated operation. It is to provide.

  In order to achieve the above object, the present invention is characterized in that a wearable search system including a wearable computer having a transmissive display and a wearable motion sensor has the following configuration.

  (1) The wearable motion sensor includes means for identifying a user's search request operation, and means for transmitting an activation request to the wearable computer in response to the search request operation. The wearable computer displays the camera image as a transmissive display. A camera image to be displayed, means for energizing the camera function in response to an activation request, means for recognizing a hand area on the camera image, and a search image from a search image range having a predetermined positional relationship with the hand area , A means for searching for information related to the search image, and a means for displaying the search result on the transmissive display.

  (2) The means for displaying the search result is displayed as an overlay on the search image range.

According to the present invention, the following effects are achieved.
(1) Since the search request operation of the user is detected and triggered as an image search start, it is possible to search for an image only by an intuitive operation regardless of whether indoors or outdoors.

  (2) The image search application of the wearable computer is not started until the user's search request operation is detected, and the camera function unit with large power consumption is not pushed, so the power consumption of the wearable computer is suppressed, The usable time can be extended.

  (3) Since the search result is displayed in an overlay at the display position of the search target, it becomes easy to associate the search target with the search result.

It is the figure which showed the structure of the wearable search system to which this invention is applied. It is the block diagram which showed the structure of the wearable motion sensor. It is the figure which showed the example (1st gesture) of the search operation | movement which this invention assumes. It is the figure which showed the example (2nd gesture) of the search operation | movement which this invention assumes. It is the figure which showed the structure of the principal part of T-HMD2. It is a figure for demonstrating the extraction method of a search image. It is the figure which showed the example of a display of a search result (the 1). It is the figure which showed the example of a display of a search result (the 2). It is the flowchart which showed operation | movement of one Embodiment of this invention. It is the block diagram which showed the structure of other embodiment of T-HMD.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing the configuration of a wearable search system to which the present invention is applied. A glasses type transparent HMD (hereinafter referred to as T-HMD) 2 and a search for a user wearing this T-HMD 2 It is comprised by the wearable motion sensor 1 which detects operation | movement.

  The T-HMD 2 activates the image search application in response to the activation request from the wearable motion sensor 1 and starts camera shooting in the viewing direction by the built-in camera. The image search application extracts a search target from the camera image, transmits a search request including the search target image to the image recognition server 3, and acquires a search result. The search results are displayed on the T-HMD2 transmissive display.

  As the wearable motion sensor 1, a smart watch or a wristband that incorporates a three-dimensional acceleration sensor and is worn on the user's wrist can be used. As the T-HMD2, glasses-type information terminals (smart glasses) represented by Google Glass (registered trademark) and “MOVERIO” can be used. The user can see the information displayed in an overlay on the T-HMD display so as to float in front of the user while seeing the view direction in a see-through manner.

  FIG. 2 is a diagram showing a configuration of a main part of the wearable motion sensor 1, and here, description will be made taking application to a smart watch as an example. It should be noted that the configuration unnecessary for the description of the present invention is not shown.

  The 3D acceleration sensor 101 detects acceleration in the x, y, and z directions at a predetermined sampling period. The user interface (I / F) 102 provides a touch sensor that accepts a user operation and a physical operator. The clock function unit 103 includes a clock function and a display function for causing the smart watch to function as a wristwatch. The power control unit 104 controls power supply to each unit by the battery 105.

  The wireless communication module 106 includes an interface for short-range wireless communication represented by a BLE (Bluetooth (registered trademark) Low Energy) 106a and a Wi-Fi 106b. The processor 100 implements various functions described in detail below based on various control programs stored in advance in the ROM 107 and various applications stored in the RAM 108.

  In the processor 100, the feature amount calculation unit 100a statistically processes the acceleration signal output from the 3D acceleration sensor 101 to generate a feature amount (feature vector). In this embodiment, each acceleration in the x, y, and z directions is detected every predetermined sampling period, and, for example, (1) an average value is used as a statistical quantity of accelerations detected within a certain period (for example, 1 sec). , (2) standard deviation, (3) absolute average value, (4) skewness, (5) maximum value, and (6) minimum value are calculated, and this is used as a feature value of user action.

  The search operation identification unit 100b is a function of the operation determination application 108a, and compares the feature amount with a determination index (for example, a separation hyperplane in the SVM) of a search request operation that has been learned in advance. Whether the operation is a search request operation is identified.

  When the user operation is identified as a search request operation, the activation request unit 100 c transmits an activation request to the T-HMD 2 by short-range wireless communication by the wireless communication module 106.

  The learning unit 100d requests the user to perform a search request operation and other operations, positive examples of acceleration feature amounts detected during the search request operation, and negative examples of acceleration feature amounts detected during other operations. Is applied to machine learning such as SVM, and for example, a separation hyperplane is constructed as a learning result 108c of a determination index for identifying a user's operation as a search request operation or other.

  As negative learning data, both (1) the user does not specify the search target and (2) the user specifies the search target but not detected are applied. It is desirable to do.

  3 and 4 are diagrams showing an example of a search operation assumed by the present invention. The first gesture of FIG. 3 is a search by placing both hands simulating L-shaped with the index finger and the thumb diagonally on the front and back. This is a method of specifying the target rectangular range. The second gesture shown in FIG. 4 is a method of designating a rectangular range to be searched by placing both hands simulating an L shape with an index finger and a thumb adjacent to each other.

  FIG. 5 is a diagram showing the configuration of the main part of the T-HMD 2. Here, application to smart glasses will be described as an example. It should be noted that the configuration unnecessary for the description of the present invention is not shown.

  The transmissive display 201 displays a see-through video in the user's field of view. The camera function unit 202 captures the direction of the user's field of view and outputs the camera image to the transmissive display 201. The user interface 203 provides a touch sensor function 203a, a microphone function 203b, and a speaker function 203c. The power control unit 204 controls power supply to each unit by the battery 205.

  The wireless communication module 206 includes an interface for near field communication such as BLE 206a and Wi-Fi 206b and an interface 206c for portable communication such as 3G, 4G and LTE. The processor 200 implements various functions described in detail below based on various control programs stored in the ROM 207 and various applications stored in the RAM 208.

  In the processor 200, the activation request detection unit 200a activates the image search application 208a when receiving the activation request from the wearable motion sensor 1. As a result, the camera function unit 202 starts camera shooting, and power consumption due to camera shooting occurs for the first time.

  The hand area detection unit 200b detects a hand area from the camera image. In this embodiment, an area that matches the skin color is detected from the frame image of the camera and is regarded as a hand area. Since skin color detection is easily affected by environmental changes such as lighting, in order to improve accuracy, skin color detection from the camera frame is matched in the HSV color space instead of RGB, and the captured skin color image is detected. Then, an area having a close range of H (Hue) and S (Saturation) is detected as a skin color.

  This is because the HSV color space is more compatible with human color recognition than RGB, and the combination of hue and saturation is less affected by changes in illumination. If the hand area is tapped while the application is running, the flesh color portion may be sampled again, and flesh color detection optimized for flesh color data in the current environment may be performed.

  When the search image extraction unit 200c tracks the hand region on the camera image and can identify the first or second gesture described with reference to FIGS. 3 and 4, the search image extraction unit 200c is in a rectangular range having a predetermined positional relationship with the hand region. An image area is extracted as a search image.

  FIG. 6 is a diagram for explaining a search image extraction method. Here, the second gesture will be described as an example. First, the polygon formed from the detection position of each finger edge (fingertip point) extracted from the frame image is estimated (broken line in the figure). Based on the detection positions P1 and P2 of the base of the finger (finger crotch) within the range of the polygon, a rectangular area A having the base of both hands as the apexes of the base and the length of the index finger is recognized. Then cut out the image.

  Returning to FIG. 5, the search execution unit 200 d creates a search request message including the search target image, transmits the search request message to the image recognition server 3, and receives a search result responded by the image recognition server 3. In the image recognition server 3, related information is stored in advance in association with a registered image to be searched, and related information is extracted by an image similarity search, and this is returned as a search result. The related information is composed of a name and detailed information about the object, reference destination information such as a URL of a Web site, and the like.

  As shown in FIGS. 7 and 8, the search result display unit 200e blends the search result with the background at the position of the rectangular range of the transmissive display 201 so that the association with the search target can be easily understood. Apply an effect so that it can be seen through.

  Such an effect can be obtained by mixing the colors of the corresponding background point (x, y) and the search result point (cx, cy) according to a certain standard. For example, in the case of a translucent example, if the RGB component of (x, y) is (255, 0, 0) (= red) and (cx, cy) is (0, 0, 255) (= blue) For example, the average is taken to (127, 0, 127) (= dark purple).

  Next, the operation of the embodiment of the present invention will be described with reference to the flowchart of FIG. In the wearable motion sensor 1, in step S1, the user's motion is repeatedly detected by the 3D acceleration sensor 101 as accelerations Gx, Gy, and Gz in the x, y, and z directions at predetermined intervals. In step S2, a plurality of accelerations Gx, Gy, Gz for a plurality of times are collected by the feature amount calculation unit 100a for each fixed period, and a feature amount is calculated.

  In step S3, the search operation identification unit 100b identifies whether or not the user's operation is a search request operation based on a comparison result between the separated hyperplane constructed by machine learning such as SVM and the feature amount. Is done. In the present embodiment, as shown in FIGS. 3 and 4, when the user operates both hands so that the search target (here, the smartphone) is surrounded by the thumb and index finger of both hands, this is identified as the search request operation. In step S4, the activation request message is transmitted from the wireless communication module 106 to the T-HMD2.

  In step S21, when the activation request is detected and detected by the activation request detection unit 200a in step S21, the preinstalled image search application 208a is activated in step S22. In step S23, the camera function unit 202 is activated by the image search application and a camera image is acquired.

  In step S24, the hand region is identified from the camera image by the hand region detection unit 200b, and whether or not the hand region is the first or second gesture for designating a search target is determined as appropriate, such as pattern recognition or image matching. This method is implemented.

  If it is the first or second gesture, the process proceeds to step S25, and an image within a rectangular range having a predetermined positional relationship with the hand region is extracted as a search image by the search image extraction unit 200c. In step S26, the search execution unit 200d generates a search request message to which the search image is attached, and transmits the search request message from the interface 206c to the image recognition server 3 via the Internet. In step S27, the search result returned by the image recognition server 3 is received. In step S28, as described with reference to FIGS. 7 and 8, the search result is displayed on the display 201 as an overlay.

  In step S29, it is determined whether or not the search request gesture has ended. If the search request gesture is maintained, the display of the search result is continued. If the search request gesture has been completed, the process proceeds to step S30 and the display of the search result is deleted.

  In step S31, a time-out determination is performed. If it is before the time-out, the process returns to step S23 and the above processes are repeated. On the other hand, if the search request gesture is not detected for a predetermined time or longer and a time-out determination is made, the process proceeds to step S32, the image search application 208a is closed, and the camera shooting is also ended.

  According to the present embodiment, until the user finds a search target and performs a search request operation, and this is detected, the image search application is not activated in T-HMD2, and the camera function unit that consumes a large amount of power is urged. Therefore, the power consumption of the T-HMD 2 can be reduced without impairing the operability related to the user's information search, and the usable time can be extended.

  Further, according to the present embodiment, since the user can start the search simply by expressing a predetermined gesture by hand, the search request and the search target designation can be performed only by an intuitive operation without utterance or complicated operation. Can be done.

  In the above embodiment, the image recognition server 3 is described as being provided on the network. However, the present invention is not limited to this, and the image recognition server 3 is illustrated as an example in FIG. A functional unit 209 corresponding to the above may be provided locally in the T-HMD 2.

  In the above-described embodiment, the user's movement is detected by a 3D (3-axis) acceleration sensor. However, the present invention is not limited to this, and a 3-axis gyroscope or 3-axis Alternatively, a 6-axis or 9-axis sensor obtained by appropriately combining these sensors may be employed.

  DESCRIPTION OF SYMBOLS 1 ... Wearable motion sensor, 2 ... Transparent HMD (T-HMD), 3 ... Image recognition server, 100, 200 ... Processor, 100a ... Feature-value calculation part, 100b ... Search operation | movement identification part, 100c ... Start-up request part, 100d ... Learning unit, 101 ... 3D acceleration sensor, 102 ... User interface, 103 ... Clock function unit, 104, 204 ... Power supply control unit, 105, 205 ... Battery, 106, 206 ... Wireless communication module, 107, 207 ... ROM, 108 208 ... RAM, 201 ... transparent display, 202 ... camera function unit, 203 ... user interface

Claims (7)

Including a wearable computer with a transmissive display and a wearable motion sensor;
The wearable motion sensor is
A search action identifying means for identifying a user's search request action;
Activation request means for transmitting an activation request to the wearable computer in response to the search request operation,
The wearable computer is
A camera function for displaying camera images on a transmissive display;
Activation request detecting means for energizing the camera function in response to the activation request;
Hand region recognition means for recognizing a hand region on the camera image;
Search image extraction means for extracting a search image from a search image range in a predetermined positional relationship with the hand region;
Search execution means for searching for information related to the search image;
A wearable search system comprising search result display means for displaying the search results on a transparent display.   The wearable search system according to claim 1, wherein the search result display means displays the search result in an overlay on the search image range. The wearable motion sensor is
A 3D acceleration sensor, a 3D gyroscope, a 3D magnetic sensor, or a combination thereof that detects a user's movement as a movement in three directions at a predetermined cycle;
Comprising a feature amount calculation means for calculating a feature amount of the user action based on a plurality of accelerations detected within a predetermined period,
The wearable search system according to claim 1, wherein the search operation identification unit identifies a user operation based on a comparison result between an identification index learned in advance and the feature amount. The wearable motion sensor is
A learning means for learning a user operation as a positive example of a search request operation, a negative example other than the search request operation, and setting boundary conditions for identifying each operation of the positive example and the negative example,
The wearable search system according to claim 3, wherein the search operation identification unit uses the boundary condition as an identification index.   The wearable search system according to any one of claims 1 to 4, wherein the search execution unit acquires a search result by transmitting a search request including the search image to an image recognition server on a network. The wearable computer includes an image recognition server function unit storing related information of search target candidates,
The wearable search system according to any one of claims 1 to 4, wherein the search execution unit provides a search request including the search image to an image recognition server function unit to acquire a search result. The wearable computer and the wearable motion sensor include a short-range wireless communication interface;
The wearable search system according to claim 1, wherein the wearable motion sensor transmits the activation request to the wearable computer by short-range wireless communication.