JP6784056B2 - Head-mounted display, display system, head-mounted display control method, and computer program - Google Patents

Description

The present invention relates to a head-mounted display device, a display system, a control method for the head-mounted display device, and a computer program.

Patent Document 1 describes a system in which a personal purchasing device is provided to an individual shopper at a shopping facility to support the individual purchasing activity of the shopper, and information is provided to the shopper using a consumer interface. Is described.

Japanese Patent Publication No. 2008-539493 Japanese Unexamined Patent Publication No. 2011-217098 Japanese Unexamined Patent Publication No. 2014-215748

However, with the technique described in Patent Document 1, although support can be received in the shopping facility, support cannot be received in a place away from the shopping facility. In addition, when they were away from the shopping facility, they could not purchase the products as if they were actually shopping in the shopping facility. In this way, the reality is that sufficient ingenuity has not been made to receive purchasing support in places away from shopping facilities. It should be noted that such a problem is not limited to shopping support, but is a common problem when it is desired to make some request from outside the facility based on the situation inside the facility.

The present invention has been made to solve at least a part of the above-mentioned problems, and can be realized as the following forms.

(1) According to the first aspect of the present invention, a head-mounted display device that supports shopping is provided. This head-mounted display device includes a line-of-sight direction detection unit that detects the line-of-sight direction of the user, a camera that captures the direction that the user sees, a voice interface that transmits information to the user, and wireless communication. It comprises a section, through the wireless communication unit, have rows send and receive information to and from the external information processing apparatus, and a processing unit for transmitting information to the user through the voice interface. The processing unit receives a movement destination instruction from the information processing device via the wireless communication unit, transmits the movement destination information to the user via the voice interface, and is imaged by the camera. At least a part of the image is transmitted as an image for transmission to the information processing apparatus via the wireless communication unit, and a predetermined range including the line-of-sight direction detected by the line-of-sight direction detection unit is specified. presented via the wireless communication unit performs communication with the short-range wireless communication terminal existing in a predetermined range which is the specified, receives data from the short-range wireless communication terminal, based on the received data Information is transmitted to the information processing device to receive an instruction of the moving destination, transmit the information of the moving destination, transmit the captured image for transmission to the information processing device, and specify the predetermined range. , The reception of data from the short-range wireless communication terminal and the transmission of the presented information to the information processing device are repeatedly executed . According to this form of the head-mounted display device, presentation information based on data received from a short-range wireless communication terminal existing in a predetermined range including the line-of-sight direction of the user can be sent to the information processing device. Therefore, the operator of the information processing device can receive information around the line-of-sight direction of the user while being at a position away from the user wearing the head-mounted display device. Therefore, according to this form of the head-mounted display device, information in the line-of-sight direction can be appropriately and efficiently presented to the operator of the information processing device at a distant position.

(2) In the head-mounted display device of the above-described embodiment, the head-mounted display device includes a camera that captures a direction that the user visually recognizes, and the processing unit captures at least a part of the image captured by the camera in the wireless communication unit. It may be configured to transmit to the information processing apparatus via the above. According to this form of the head-mounted display device, the image captured by the camera can present the external view visually recognized by the user. Therefore, according to this form of the head-mounted display device, the presentation to the operator of the information processing device can be made more appropriate and efficient.

(3) In the head-mounted display device of the above-described form, the data received from the short-range wireless communication terminal existing in the predetermined range may be data for each type of product displayed in the shopping facility. Good. According to this form of head-mounted display device, data for each product type can be received from short-range wireless communication terminals arranged for each product type in the shopping facility, so that external information processing can be performed. It is possible to present information for each type of product to the device. Therefore, according to this form of the head-mounted display device, it is possible to support the operator of the information processing device to purchase the product.

(4) In the head-mounted display device of the above-described embodiment, the processing unit may specify a portion of the image for transmission that overlaps with the predetermined range, and perform image processing on the overlapping portion. According to this form of the head-mounted display device, a predetermined range can be clearly indicated to the operator of the information processing device at a distant position.

(5) In the head-mounted display device of the above-described embodiment, the external information processing device may be another head-mounted display device different from the head-mounted display device. According to this form of the head-mounted display device, it is possible to provide a user who wears another head-mounted display device with a usage environment similar to that of the user.

(6) According to the second embodiment of the present invention, a head-mounted display device, comprising an information processing apparatus, a display system that helps shop is provided. The head-mounted display device of this display system includes a line-of-sight direction detection unit that detects the line-of-sight direction of the user, a camera that captures the direction that the user sees, and a voice interface that transmits information to the user. A wireless communication unit, and a processing unit that presents information to the information processing device via the wireless communication unit. The processing unit receives a movement destination instruction from the information processing device via the wireless communication unit, transmits the movement destination information to the user via the voice interface, and is imaged by the camera. At least a part of the image is transmitted as an image for transmission to the information processing apparatus via the wireless communication unit, and a predetermined range including the line-of-sight direction detected by the line-of-sight direction detection unit is specified. presented via the wireless communication unit performs communication with the short-range wireless communication terminal existing in a predetermined range which is the specified, receives data from the short-range wireless communication terminal, based on the received data Information is transmitted to the information processing device, the instruction of the moving destination is received, the information of the moving destination is transmitted, the captured image for transmission is transmitted to the information processing device, and the predetermined range is specified. And, the reception of the data from the short-range wireless communication terminal and the transmission of the presented information to the information processing device are repeatedly executed . The information processing device of this display system transmits an instruction of the movement destination to the head-mounted display device, receives the transmission captured image from the head-mounted display device, and receives the transmission image to display the head-mounted display. The presentation information sent from the device is received, and a display is performed based on the received captured image for transmission and the presentation information. According to this form of display system, presentation information based on data received from a short-range wireless communication terminal existing in a predetermined range including the line-of-sight direction of a user wearing a head-mounted display device is transmitted to an information processing device. Can be sent. Therefore, the operator of the information processing device can receive information around the line-of-sight direction of the user while being at a position away from the user wearing the head-mounted display device. Therefore, this form of display system can appropriately and efficiently present information on the line-of-sight direction of the user wearing the head-mounted display device to the operator of the information processing device at a remote position. it can.

The present invention can also be realized in various forms other than the head-mounted display device. For example, it can be realized by a control method of a head-mounted display device, a computer program for realizing the functions of each component included in the head-mounted display device, a recording medium on which the computer program is recorded, or the like.

It is explanatory drawing which shows the schematic structure of the display system in 1st Embodiment of this invention. It is a main part plan view which shows the structure of the optical system provided in the image display part. It is a figure which shows the main part structure of the image display part seen from the user. It is a figure for demonstrating the angle of view of a camera. It is a block diagram which shows the structure of HMD functionally. It is a block diagram which shows the structure of a control device functionally. It is explanatory drawing which shows an example of the augmented reality display by HMD. It is a block diagram which functionally shows the structure of the control device of a supporter HMD. It is explanatory drawing which shows an example of augmented reality display by a supporter HMD. It is explanatory drawing which shows the process to execute on the supporter HMD side and the supporter HMD side at the time of shopping. It is a flowchart which shows the shopping support process. It is explanatory drawing which shows an example of the fruit section in a store. It is explanatory drawing which illustrates the predetermined range. It is explanatory drawing which shows each product attribute data received from each BLE terminal existing in a predetermined range. It is explanatory drawing which shows an example of the field of view of the supporter at the time of execution of a shopping support process. It is explanatory drawing which shows another example of the field of view of the supporter at the time of execution of a shopping support process. It is a main part plan view which shows the structure of the optical system provided in the image display part of the modification.

A. Overall configuration of display system:
FIG. 1 is an explanatory diagram showing a schematic configuration of a display system according to an embodiment of the present invention. The display system 1 includes two head-mounted display devices 100 and 400. On the other hand, the head-mounted display device 100 is used by a shopping supporter in a store (shopping facility) such as a supermarket. The other head-mounted display device 400 is used by a shopping supporter, for example, in a home away from the store.

Each of the two head-mounted display devices 100 and 400 is connected to the Internet INT by wireless communication via the communication carrier BS. The communication carrier BS includes a transmission / reception antenna, a radio base station, and an exchange. The Internet INT enables communication between the two head-mounted display devices 100 and 400. First, the configuration of one head-mounted display device 100, that is, the HMD 100 used by the shopping supporter in the store will be described.

B. Configuration of head-mounted display device:
The head-mounted display device 100 is a display device worn on the head of a shopping supporter (hereinafter, also referred to as “user”), and is also referred to as a head-mounted display (HMD). The HMD 100 is a see-through type (transmissive type) head-mounted display device in which an image emerges in the outside world that is visually recognized through the glass.

The HMD 100 includes an image display unit 20 that allows the user to visually recognize an image, and a control device (controller) 10 that controls the image display unit 20.

The image display unit 20 is a wearer that is worn on the user's head, and has a spectacle shape in the present embodiment. The image display unit 20 has a right display unit 21, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27. And.

The right holding portion 21 and the left holding portion 23 extend rearward from both ends of the front frame 27, respectively, and hold the image display portion 20 on the user's head like a temple (vine) of eyeglasses. Here, of both ends of the front frame 27, the end located on the right side of the user when the image display unit 20 is attached is referred to as the end ER, and the end located on the left side of the user is referred to as the end EL. To do. The right holding portion 21 extends from the end ER of the front frame 27 to a position corresponding to the right head of the user in the mounted state of the image display portion 20. The left holding portion 23 is provided extending from the end EL of the front frame 27 to a position corresponding to the left head of the user in the mounted state of the image display portion 20.

The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is located in front of the user's right eye when the image display unit 20 is attached, and allows the right eye to visually recognize the image. The left light guide plate 28 is located in front of the left eye of the user when the image display unit 20 is attached, and allows the left eye to visually recognize the image.

The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other. This connection position corresponds to the position between the user's eyebrows when the image display unit 20 is worn. The front frame 27 may be provided with a nose pad that comes into contact with the user's nose when the image display unit 20 is attached at the connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad portion, the right holding portion 21, and the left holding portion 23. Further, a belt in contact with the back of the head of the user may be connected to the right holding portion 21 and the left holding portion 23 while the image display unit 20 is attached. In this case, the image display unit 20 can be firmly held on the user's head by the belt.

The right display unit 22 displays an image by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located near the right head of the user in the mounted state of the image display unit 20. The left display unit 24 displays an image by the left light guide plate 28. The left display unit 24 is provided on the left holding unit 23 and is located near the left side head of the user when the image display unit 20 is attached. The right display unit 22 and the left display unit 24 are also collectively referred to as a "display drive unit".

The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical portions (for example, prisms) formed of a light-transmitting resin or the like, and use the image light output by the right display unit 22 and the left display unit 24. Lead to the eyes of others. A dimming plate may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The dimmer is a thin plate-shaped optical element having different transmittance depending on the wavelength range of light, and functions as a so-called wavelength filter. The dimmer is arranged so as to cover, for example, the surface of the front frame 27 (the surface opposite to the surface facing the user's eyes). By appropriately selecting the optical characteristics of the dimmer, the transmittance of light in any wavelength range such as visible light, infrared light, and ultraviolet light can be adjusted, and the right light guide plate 26 and left guide can be adjusted from the outside. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively, and creates an image (augmented reality (AR) image) by the image light. Make it visible to the user (also called "displaying an image"). When external light is incident on the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light constituting the image and the external light are incident on the user's eyes. To do. Therefore, the visibility of the image by the user is affected by the intensity of external light.

Therefore, for example, by attaching a dimmer to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the dimmer, the ease of viewing the image can be adjusted. In a typical example, it is possible to select a dimmer plate having at least a light transmittance that allows a user wearing the HMD 100 to visually recognize the outside scenery. When the dimming plate is used, the effect of protecting the right light guide plate 26 and the left light guide plate 28 and suppressing damage to the right light guide plate 26 and the left light guide plate 28 and adhesion of dirt can be expected. The light control plate may be detachable from the front frame 27 or the right light guide plate 26 and the left light guide plate 28, respectively. Further, a plurality of types of dimmers may be exchanged so that they can be attached and detached, and the dimmers may be omitted.

The camera 61 is arranged in the front frame 27 of the image display unit 20. The camera 61 is provided on the front surface of the front frame 27 at a position that does not block the external light transmitted through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is arranged at the connecting portion between the right light guide plate 26 and the left light guide plate 28. The camera 61 may be arranged on the end ER side of the front frame 27, or may be arranged on the end EL side of the front frame 27.

The camera 61 is a digital camera including an image pickup element such as a CCD or CMOS, an image pickup lens, or the like. Although the camera 61 of the present embodiment is a monocular camera, a stereo camera may be adopted. The camera 61 captures at least a part of the outside view (real space) in the front side direction of the HMD 100, in other words, in the field of view direction visually recognized by the user when the image display unit 20 is attached. In other words, the camera 61 captures a range or direction that overlaps with the user's field of view, and captures the direction that the user sees. The width of the angle of view of the camera 61 can be set as appropriate. In the present embodiment, the wide angle of view of the camera 61 is set so that the user can image the entire field of view of the user who can see through the right light guide plate 26 and the left light guide plate 28. The camera 61 executes imaging under the control of the control function unit 150 (FIG. 5), and outputs the obtained imaging data to the control function unit 150.

The distance measuring sensor 62 is arranged above the camera 61 at the connecting portion between the right light guide plate 26 and the left light guide plate 28 of the front frame 27. The distance measuring sensor 62 detects the distance to the measurement object located in the preset measurement direction. The measurement direction can be the front side direction of the MD 100 (the direction overlapping the imaging direction of the camera 61). The distance measuring sensor can be composed of, for example, a light emitting unit such as an LED or a laser diode, and a light receiving unit that receives the reflected light that the light emitted from the light source reflects on the object to be measured. In this case, the distance is obtained by triangular distance measurement processing or distance measurement processing based on the time difference. The distance measuring sensor may be composed of, for example, a transmitting unit that emits ultrasonic waves and a receiving unit that receives ultrasonic waves reflected by the object to be measured. In this case, the distance is obtained by distance measurement processing based on the time difference. Like the camera 61, the distance measuring sensor is controlled by the control function unit 150, and the detection result is output to the control function unit 150.

FIG. 2 is a plan view of a main part showing the configuration of the optical system included in the image display unit 20. For convenience of explanation, FIG. 2 illustrates the user's right eye RE and left eye LE. As shown in FIG. 2, the right display unit 22 and the left display unit 24 are symmetrically configured.

The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 and a right optical system 251 so that the right eye RE can visually recognize an image (AR image). The OLED unit 221 emits image light. The right optical system 251 includes a lens group and the like, and guides the image light L emitted by the OLED unit 221 to the right light guide plate 26.

The OLED unit 221 has an OLED panel 223 and an OLED drive circuit 225 for driving the OLED panel 223. The OLED panel 223 is a self-luminous display panel composed of light emitting elements that emit light by organic electroluminescence and emit R (red), G (green), and B (blue) colored lights, respectively. In the OLED panel 223, a plurality of pixels having a unit including one R, G, and B element as one pixel are arranged in a matrix.

The OLED drive circuit 225 selects and energizes the light emitting element included in the OLED panel 223 according to the control of the control function unit 150 (FIG. 5), and causes the light emitting element to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 225 is composed of, for example, a semiconductor device that drives the OLED panel 223, and may be mounted on a substrate fixed to the back surface of the OLED panel 223. A temperature sensor 217 (FIG. 5), which will be described later, is mounted on this substrate. The OLED panel 223 may adopt a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to each of the R, G, and B colors are arranged in an overlapping manner. Further, an OLED panel 223 having a WRGB configuration including a light emitting element that emits W (white) light in addition to a light emitting element that emits R, G, and B colored lights may be adopted.

The right optical system 251 has a collimating lens that converts the image light L emitted from the OLED panel 223 into a light flux in a parallel state. The image light L converted into a light flux in a parallel state by the collimating lens is incident on the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in the optical path that guides the light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. After being reflected by the half mirror 261, the image light L is emitted from the right light guide plate 26 to the right eye RE, and the image light L forms an image on the retina of the right eye RE so that the user can visually recognize the image.

The left display unit 24 includes an OLED unit 241 and a left optical system 252 as a configuration for allowing the left eye LE to visually recognize an image (AR image). The OLED unit 241 emits image light. The left optical system 252 includes a lens group and the like, and guides the image light L emitted by the OLED unit 241 to the left light guide plate 28. The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The details of each part are the same as those of the OLED unit 221 and the OLED panel 223, and the OLED drive circuit 225. The temperature sensor 239 is mounted on the substrate fixed to the back surface of the OLED panel 243. The details of the left optical system 252 are the same as those of the right optical system 251.

According to the configuration described above, the HMD 100 can function as a see-through type display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the user's right eye RE. The image light L reflected by the half mirror 281 and the external light OL transmitted through the left light guide plate 28 are incident on the left eye LE of the user. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL and causes them to enter the user's eye. As a result, the user can see the outside view (real world) through the right light guide plate 26 and the left light guide plate 28, and can visually recognize the image (AR image) by the image light L so as to overlap the outside view.

The half mirror 261 and the half mirror 281 function as an "image extraction unit" that reflects the image light output by the right display unit 22 and the left display unit 24 to extract an image. Further, the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide unit”, and the left optical system 252 and the left light guide plate 28 are collectively referred to as a “left light guide unit”. The configuration of the right light guide unit and the left light guide unit is not limited to the above-mentioned example, and any method can be used as long as an image is formed in front of the user's eyes using image light. For example, a diffraction grating may be used for the right light guide portion and the left light guide portion, or a semitransmissive reflective film may be used.

In FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector provided at the bottom of the control device 10, and is connected to various circuits inside the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable or an optical fiber cable for transmitting digital data. The connection cable 40 may further include a metal cable for transmitting analog data. A connector 46 is provided in the middle of the connection cable 40.

The connector 46 is a jack for connecting a stereo mini plug, and the connector 46 and the control device 10 are connected by, for example, a line for transmitting an analog audio signal. In the example of the present embodiment shown in FIG. 1, the right earphone 32 and the left earphone 34 constituting the stereo headphones and the headset 30 having the microphone 63 are connected to the connector 46.

As shown in FIG. 1, for example, the microphone 63 is arranged so that the sound collecting portion of the microphone 63 faces the line-of-sight direction of the user. The microphone 63 collects sound and outputs a voice signal to the voice interface 182 (FIG. 5). The microphone 63 may be a monaural microphone or a stereo microphone, and may be a directional microphone or an omnidirectional microphone.

The control device 10 is a device for controlling the HMD 100. The control device 10 includes a lighting unit 12, a touch pad 14, a direction key 16, a decision key 17, and a power switch 18. The lighting unit 12 notifies the operating state of the HMD 100 (for example, ON / OFF of the power supply, etc.) according to its light emitting mode. As the lighting unit 12, for example, an LED (Light Emitting Diode) can be used.

The touch pad 14 detects a contact operation on the operation surface of the touch pad 14 and outputs a signal according to the detected content. As the touch pad 14, various touch pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The direction key 16 detects a pressing operation on the key corresponding to the up / down / left / right directions, and outputs a signal according to the detected content. The decision key 17 detects the pressing operation and outputs a signal for determining the content operated by the control device 10. The power switch 18 switches the power state of the HMD 100 by detecting the sliding operation of the switch.

FIG. 3 is a diagram showing a configuration of a main part of the image display unit 20 as seen from the user. In FIG. 3, the connection cable 40, the right earphone 32, and the left earphone 34 are not shown. In the state of FIG. 3, the back side of the right light guide plate 26 and the left light guide plate 28 can be visually recognized, and the half mirror 261 for irradiating the right eye RE with image light and the left eye LE for irradiating the image light. The half mirror 281 can be visually recognized as a substantially quadrangular area. The user visually recognizes the outside view through the entire left and right light guide plates 26 and 28 including the half mirrors 261 and 281 and visually recognizes a rectangular display image at the positions of the half mirrors 261 and 281.

FIG. 4 is a diagram for explaining the angle of view of the camera 61. In FIG. 4, the camera 61 and the user's right eye RE and left eye LE are schematically shown in a plan view, and the angle of view (imaging range) of the camera 61 is shown by θ. The angle of view θ of the camera 61 is widened in the horizontal direction as shown in the drawing, and is also widened in the vertical direction like a general digital camera.

As described above, the camera 61 is arranged at the connecting portion between the light guide plate 26 and the left light guide plate 28 in the front frame 27 of the image display unit 20, and images the direction of the user's line of sight (that is, in front of the user). .. Therefore, the optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The external view that the user can see while wearing the HMD100 is not always infinity. For example, when the user gazes at the object OB with both eyes, the user's line of sight is directed to the object OB as shown by the symbols RD and LD in the figure. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often 1 m to 4 m. Therefore, for the HMD 100, a guideline for the upper limit and the lower limit of the distance from the user to the object OB during normal use may be set. This guideline may be obtained in advance and preset in the HMD 100, or may be set by the user. The optical axis and angle of view of the camera 61 are set so that the object OB is included in the angle of view when the distance to the object OB during such normal use corresponds to the set upper limit and lower limit. It is preferable to be done.

Generally, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective visual field having excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. The stable gazing field at which a human gaze can be seen quickly and stably is 60 to 90 degrees in the horizontal direction and 45 to 70 degrees in the vertical direction. In this case, when the gazing point is the object OB (FIG. 4), the effective field of view is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction with the lines of sight RD and LD as the center. The stable gaze is 60 to 90 degrees in the horizontal direction and 45 to 70 degrees in the vertical direction. The actual field of view that the user sees through the image display unit 20 and through the right light guide plate 26 and the left light guide plate 28 is called an actual field of view (FOV). The actual field of view is narrower than the viewing angle and stable gaze, but wider than the effective field of view.

The angle of view θ of the camera 61 of the present embodiment is set so that a range wider than the field of view of the user can be imaged. The angle of view θ of the camera 61 is preferably set so that at least a range wider than the user's effective field of view can be imaged, and more preferably a range wider than the actual field of view can be imaged. The angle of view θ of the camera 61 is more preferably set so as to be able to image a wider range than the user's stable gazing field, and is set so as to be able to image a wider range than the viewing angle of both eyes of the user. Most preferred. Therefore, the camera 61 may be provided with a so-called wide-angle lens as an image pickup lens, and may be configured to be able to take a wide angle of view. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens. Further, the camera 61 may include a single focus lens, a zoom lens, or a lens group including a plurality of lenses.

FIG. 5 is a block diagram functionally showing the configuration of the HMD 100. The control device 10 includes a main processor 140 that executes a program to control the HMD 100, a storage unit, an input / output unit, sensors, an interface, and a power supply unit 130. These storage units, input / output units, sensors, interfaces, and power supply units 130 are connected to the main processor 140, respectively. The main processor 140 is mounted on the controller board 120 in which the control device 10 is built.

The storage unit includes a memory 118 and a non-volatile storage unit 121. The memory 118 constitutes a computer program executed by the main processor 140 and a work area for temporarily storing data to be processed. The non-volatile storage unit 121 is composed of a flash memory and an eMMC (embedded Multi Media Card). The non-volatile storage unit 121 stores a computer program executed by the main processor 140 and various data processed by the main processor 140. In this embodiment, these storage units are mounted on the controller board 120.

The input / output unit includes a touch pad 14 and an operation unit 110. The operation unit 110 includes a direction key 16 provided in the control device 10, a decision key 17, and a power switch 18. The main processor 140 controls each of these input / output units and acquires signals output from each input / output unit.

The sensors include a 6-axis sensor 111, a magnetic sensor 113, and a GPS (Global Positioning System) receiver 115. The 6-axis sensor 111 is a motion sensor (inertia sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Inertial Measurement Unit) in which these sensors are modularized. The magnetic sensor 113 is, for example, a three-axis geomagnetic sensor. The GPS receiver 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects the coordinates of the current position of the control device 10. These sensors (6-axis sensor 111, magnetic sensor 113, GPS receiver 115) output the detected value to the main processor 140 according to a sampling frequency specified in advance. The timing at which each sensor outputs the detected value may be in response to an instruction from the main processor 140.

The interface includes a wireless communication unit 117, a voice codec 180, an external connector 184, an external memory interface 186, a USB (Universal Serial Bus) connector 188, a sensor hub 192, an FPGA 194, and an interface 196. ing. These function as an interface with the outside. The wireless communication unit 117 executes wireless communication between the HMD 100 and an external device. The wireless communication unit 117 is configured to include an antenna (not shown), an RF circuit, a baseband circuit, a communication control circuit, and the like, or is configured as a device in which these are integrated. The wireless communication unit 117 performs wireless communication conforming to standards such as a wireless LAN including Wi-Fi (registered trademark), Bluetooth (registered trademark), and iBeacon (registered trademark).

The voice codec 180 is connected to the voice interface 182 and encodes / decodes a voice signal input / output via the voice interface 182. The voice interface 182 is an interface for inputting / outputting a voice signal. The voice codec 180 may include an A / D converter that converts an analog voice signal into digital voice data, and a D / A converter that performs the reverse conversion. The HMD 100 of the present embodiment outputs sound from the right earphone 32 and the left earphone 34, and collects the sound by the microphone 63. The voice codec 180 converts the digital voice data output by the main processor 140 into an analog voice signal and outputs it via the voice interface 182. Further, the voice codec 180 converts the analog voice signal input to the voice interface 182 into digital voice data and outputs it to the main processor 140.

The external connector 184 is a connector for connecting an external device (for example, a personal computer, a smartphone, a game device, etc.) that communicates with the main processor 140 to the main processor 140. The external device connected to the external connector 184 can be used as a source of contents, debug a computer program executed by the main processor 140, and collect an operation log of the HMD 100. The external connector 184 can adopt various aspects. As the external connector 184, for example, an interface compatible with a wired connection such as a USB interface, a micro USB interface, and an interface for a memory card, or an interface compatible with a wireless connection such as a wireless LAN interface and a Bluetooth interface can be adopted.

The external memory interface 186 is an interface to which a portable memory device can be connected. The external memory interface 186 includes, for example, a memory card slot in which a card-type recording medium is attached to read / write data, and an interface circuit. The size, shape, standard, etc. of the card-type recording medium can be appropriately selected. The USB connector 188 is an interface to which a memory device, a smartphone, a mobile phone terminal, a personal computer, or the like conforming to the USB standard can be connected. The USB connector 188 includes, for example, a connector conforming to the USB standard and an interface circuit. The size, shape, USB standard version, etc. of the USB connector 188 can be appropriately selected.

The HMD 100 also includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor, and the like, and emits vibration under the control of the main processor 140. The HMD 100 generates vibration by the vibrator 19 in a predetermined vibration pattern, for example, when an operation on the operation unit 110 is detected or when the power of the HMD 100 is turned on / off.

The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via the interface (I / F) 196. The sensor hub 192 acquires the detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted / received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196. The interface 196 is connected to the right display unit 22 and the left display unit 24 of the image display unit 20, respectively. In the example of the present embodiment, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and the right display unit 22 and the left display unit 24 are respectively. It is connected to the interface 196 of the control device 10.

The power supply unit 130 includes a battery 132 and a power supply control circuit 134. The power supply unit 130 supplies electric power for operating the control device 10. The battery 132 is a rechargeable battery. The power supply control circuit 134 detects the remaining capacity of the battery 132 and controls the charging of the OS 143. The power supply control circuit 134 is connected to the main processor 140, and outputs the detected value of the remaining capacity of the battery 132 and the detected value of the voltage of the battery 132 to the main processor 140. The electric power may be supplied from the control device 10 to the image display unit 20 based on the electric power supplied by the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be controlled by the main processor 140.

The right display unit 22 includes a display unit board 210, an OLED unit 221, a camera 61, an illuminance sensor 65, an LED indicator 67, and a temperature sensor 217. An interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213, and an EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 are mounted on the display unit board 210. The receiving unit 213 receives the data input from the control device 10 via the interface 211. When the receiving unit 213 receives the image data of the image displayed by the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

The EEPROM 215 stores various data in a manner readable by the main processor 140. The EEPROM 215 stores, for example, data on the light emission characteristics and display characteristics of the OLED units 221 and 241 of the image display unit 20, data on the sensor characteristics of the right display unit 22 or the left display unit 24, and the like. Specifically, for example, parameters related to gamma correction of the OLED units 221 and 241 and data for compensating for the detected values of the temperature sensors 217 and 239 described later are stored. These data are generated by the factory inspection of HMD100 and written to EEPROM 215. After shipment, the main processor 140 reads the data of the EEPROM 215 and uses it for various processes.

The camera 61 executes imaging according to a signal input via the interface 211, and outputs captured image data or a signal representing the imaging result to the control device 10. As shown in FIG. 1, the illuminance sensor 65 is provided at the end ER of the front frame 27 and is arranged so as to receive external light from the front of the user who wears the image display unit 20. The illuminance sensor 65 outputs a detected value corresponding to the amount of received light (light receiving intensity). As shown in FIG. 1, the LED indicator 67 is arranged in the vicinity of the camera 61 at the end ER of the front frame 27. The LED indicator 67 lights up during the image pickup by the camera 61 to notify that the image pickup is in progress.

The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 217 is mounted on the back surface side of the OLED panel 223 (FIG. 3). The temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225, for example. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223. The temperature sensor 217 may be built in the OLED panel 223 or the OLED drive circuit 225. For example, when the OLED panel 223 is mounted as a Si-OLED together with the OLED drive circuit 225 as an integrated circuit on an integrated semiconductor chip, the temperature sensor 217 may be mounted on the semiconductor chip.

The left display unit 24 includes a display unit board 230, an OLED unit 241 and a temperature sensor 239. An interface (I / F) 231 connected to the interface 196, a receiving unit (Rx) 233, a 6-axis sensor 235, and a magnetic sensor 237 are mounted on the display unit board 230. The receiving unit 233 receives the data input from the control device 10 via the interface 231. When the receiving unit 233 receives the image data of the image displayed by the OLED unit 241, the receiving unit 233 outputs the received image data to the OLED drive circuit 245 (FIG. 2).

The 6-axis sensor 235 is a motion sensor (inertia sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensor is modularized. Since the 6-axis sensor 235 is provided in the image display unit 20, when the image display unit 20 is attached to the user's head, the movement of the user's head is detected. The orientation of the image display unit 20 can be known from the detected movement of the head of the user US. The magnetic sensor 237 is, for example, a three-axis geomagnetic sensor. The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 239 is mounted on the back surface side of the OLED panel 243 (FIG. 3). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243. The temperature sensor 239 may be built in the OLED panel 243 or the OLED drive circuit 245. The details are the same as those of the temperature sensor 217.

The camera 61, the illuminance sensor 65, and the temperature sensor 217 of the right display unit 22, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 of the left display unit 24 are connected to the sensor hub 192 of the control device 10. The sensor hub 192 sets and initializes the sampling cycle of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization of each sensor, transmission of control data, acquisition of detected values, and the like in accordance with the sampling cycle of each sensor. The sensor hub 192 outputs the detection value of each sensor included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a cache function that temporarily holds the detected value of each sensor. The sensor hub 192 may include a signal format or data format conversion function (for example, a conversion function to a unified format) of the detection value of each sensor. The sensor hub 192 turns on or off the LED indicator 67 by starting and stopping the energization of the LED indicator 67 according to the control of the main processor 140.

FIG. 6 is a block diagram functionally showing the configuration of the control device 10. The control device 10 functionally includes a storage function unit 122 and a control function unit 150. The storage function unit 122 is a logical storage unit composed of the non-volatile storage unit 121 (FIG. 5). The storage function unit 122 may be configured to use the EEPROM 215 or the memory 118 in combination with the non-volatile storage unit 121 instead of the configuration in which only the storage function unit 122 is used. The control function unit 150 is configured by the main processor 140 executing a computer program, that is, by the cooperation of hardware and software.

The storage function unit 122 stores various data to be processed by the control function unit 150. Specifically, the storage function unit 122 of the present embodiment stores the setting data 123 and the content data 124. The setting data 123 includes various setting values related to the operation of the HMD 100. For example, the setting data 123 includes parameters, a determinant, an arithmetic expression, a LUT (Look Up Table), and the like when the control function unit 150 controls the HMD 100.

The content data 124 includes content data (image data, video data, audio data, etc.) including images and videos displayed by the image display unit 20 under the control of the control function unit 150. The content data 124 may include bidirectional content data. The two-way content means that the operation unit 110 acquires the user's operation, the control function unit 150 executes the processing according to the acquired operation content, and the content according to the processing content is displayed on the image display unit 20. Means the type of content to display. In this case, the content data may include image data of the menu screen for acquiring the operation of the user, data for defining the process corresponding to the item included in the menu screen, and the like.

The control function unit 150 executes various processes using the data stored in the storage function unit 122, so that the OS 143, the image processing unit 145, the display control unit 147, the image pickup control unit 149, and the input / output control unit 151 , Communication control unit 153, shopping support processing unit 155. In this embodiment, each functional unit other than OS143 is configured as a computer program executed on OS143.

The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on the image data of the image / video displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like. The image processing unit 145 may be configured by hardware other than the main processor 140 (for example, DSP (Digital Signal Processor)) in addition to the configuration realized by the main processor 140 executing a computer program.

The image processing unit 145 may execute resolution conversion processing, image adjustment processing, 2D / 3D conversion processing, and the like, if necessary. The resolution conversion process is a process of converting the resolution of the image data into a resolution suitable for the right display unit 22 and the left display unit 24. The image adjustment process is a process for adjusting the brightness and saturation of image data. The 2D / 3D conversion process is a process of generating two-dimensional image data from three-dimensional image data or generating three-dimensional image data from two-dimensional image data. When these processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data and transmits the signal to the image display unit 20 via the connection cable 40.

The display control unit 147 generates a control signal for controlling the right display unit 22 and the left display unit 24, and controls the generation and emission of image light by each of the right display unit 22 and the left display unit 24 by the control signal. To do. Specifically, the display control unit 147 controls the OLED drive circuits 225 and 245 to display an image on the OLED panels 223 and 243. The display control unit 147 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243, and controls the brightness of the OLED panels 223 and 243, based on the signal output by the image processing unit 145.

The image pickup control unit 149 controls the camera 61 to execute image pickup, generate captured image data, and temporarily store it in the storage function unit 122. When the camera 61 is configured as a camera unit including a circuit for generating captured image data, the imaging control unit 149 acquires the captured image data from the camera 61 and temporarily stores the captured image data in the storage function unit 122.

The input / output control unit 151 appropriately controls the touch pad 14 (FIG. 1), the direction key 16, and the decision key 17, and acquires an input command from them. The acquired command is output to OS143 or a computer program running on OS143 together with OS143. The communication control unit 153 controls the wireless communication unit 117 to perform wireless communication with, for example, an external HMD 400.

The shopping support processing unit 155 is a function realized according to an application program running on the OS 143. The shopping support processing unit 155 identifies a predetermined range including the detected line-of-sight direction of the user, and connects the BLE (Bluetooth Low Energy) terminal and the wireless communication unit 117 (FIG. 5) existing in the specified predetermined range. Communication is performed via the BLE terminal, data is received from the BLE terminal, and presentation information based on the received data is transmitted to the HMD 400 worn by the shopping supporter via the wireless communication unit 117. The details of the shopping support processing unit 155 will be described later. The shopping support processing unit 155 corresponds to the "processing unit" in the first aspect of the present invention described in the [Summary of Invention] column.

FIG. 7 is an explanatory diagram showing an example of augmented reality display by the HMD 100. FIG. 7 illustrates the user's field of view VR. As described above, the image light guided to both eyes of the user of the HMD 100 is imaged on the retina of the user, so that the user visually recognizes the image AI as augmented reality (AR). In the example of FIG. 7, the image AI is the menu screen of the OS of the HMD100. The menu screen includes, for example, an icon IC for starting each application program of "message", "telephone", "camera", "browser", and "shopping support". Further, the light guide plates 26 and 28 on the left and right transmit the light from the outside view SC so that the user can visually recognize the outside view SC. In this way, the user of the HMD 100 can see the image AI so that the portion of the field of view VR where the image VI is displayed overlaps with the outside view SC. In addition, only the outside view SC can be seen in the portion of the field of view VR where the image AI is not displayed. Although the illustrated outside view SC is in the city, the outside view SC is a view inside the store because the HMD100 is used in the store as described above.

As described above, the above-mentioned HMD100 is used by the shopping supporter in the store, while the HMD400 is used by the shopping supporter in the home away from the store. Hereinafter, the HMD100 used by the shopping supporter will be referred to as a “supporter HMD100”, and the HMD400 used by the shopping supporter will be referred to as a “supporter HMD400”. The supporter HMD400 is the same model as the supporter HMD100, and differs only in some installed application programs. Due to the difference in some application programs, the supporter HMD400 has a different control function unit of the control device 10. The same components as the supporter HMD100 in the supporter HMD400 will be described below using the same reference numerals as those of the supporter HMD100.

FIG. 8 is a block diagram functionally showing the configuration of the control device 10 of the supporter HMD400. The control device 10 functionally includes a storage function unit 122 and a control function unit 450. The memory function unit 122 is the same as the memory function unit 122 of the supporter HMD100. The control function unit 450 is different from the control function unit 150 of the supporter HMD100 in that a shopping processing unit 455 is provided instead of the shopping support processing unit 155, and the remaining components (143 in the figure). ~ 153) is the same.

The shopping processing unit 455 is a function realized according to an application program running on the OS 143. The shopping processing unit 455 requests shopping support and displays the presentation information sent from the supporter HMD100. The details of the shopping processing unit 455 will be described later.

FIG. 9 is an explanatory diagram showing an example of augmented reality display by the supporter HMD400. FIG. 9 illustrates the field of view VR2 of the shopping supporter (hereinafter, also referred to as “user”). As described above, the image light guided to both eyes of the user of the supporter HMD400 is imaged on the retina of the user, so that the user visually recognizes the image AI2 as augmented reality (AR). .. In the example of FIG. 7, the image AI2 is the OS menu screen of the supporter HMD400. The menu screen includes, for example, an icon IC2 for activating each application program of "message", "telephone", "camera", "browser", and "remote shopping". Further, the right and left light guide plates 26 and 28 transmit the light from the outside view SC2 so that the user can visually recognize the outside view SC2. In this way, the user of the supporter HMD400 can see the image AI2 so as to overlap the external view SC2 with respect to the portion of the field of view VR2 where the image VI2 is displayed. Further, only the external view SC2 can be seen in the portion of the field of view VR2 where the image AI2 is not displayed. Although the illustrated outside view SC2 is in the city, since the supporter HMD400 is used in the home as described above, the outside view SC2 is a view inside the home.

C. About the process related to shopping:
FIG. 10 is an explanatory diagram showing a process executed on the supporter HMD400 side and the supporter HMD100 side when shopping. When shopping, first, the shopping supporter wearing the supporter HMD400 (hereinafter, also simply referred to as “supporter”) requests the store to support the shopping (process P1). Specifically, the supporter instructs the icon IC2 of "remote shopping" prepared on the menu screen by the direction key 16 (FIG. 1) and the enter key 17 (FIG. 1), thereby causing the USB connector 188 (FIG. 1). Communicate with the mobile phone terminal via 5) and call the pre-registered store. Use this phone to ask the store to help you with your shopping. Requests for assistance can be made by other means of communication, such as email, instead of telephone.

At the store that receives the request for shopping support, the shopping supporter wearing the supporter HMD100 (hereinafter, also simply referred to as “supporter”), who is waiting in advance in the store, is instructed to provide shopping support. The supporter who received the instruction establishes communication between the supporter HMD100 and the supporter HMD400, and then entrusts the shopping support to the supporter HMD400 worn by the shopping supporter. Transmit (step P2). Communication between the supporter HMD100 and the supporter HMD400 includes not only data transfer but also voice communication.

Further, the supporter HMD100 transmits the captured image captured by the camera 61 to the supporter HMD400 (step P3).

The supporter HMD400 receives the captured image sent from the supporter HMD100 and displays the received captured image (step P4). Specifically, the image processing unit 145 (FIG. 6) is controlled based on the captured image, and the display control unit 147 (FIG. 6) is made to display the captured image. As a result, the captured image captured by the supporter HMD100 is displayed on the supporter HMD400. As a result, the supporter wearing the supporter HMD400 can see the image in the store while staying at home. After that, the captured image is continuously sent from the supporter HMD100 to the supporter HMD400 until the shopping support is completed, and the supporter HMD400 can sequentially see the situation in the store.

The supporter looks at the store in order while looking at the products. At that time, the supporter uses the supporter HMD400 to transmit an instruction to the supporter HMD100 (step P5). Specifically, the supporter transmits to the supporter HMD100 that he / she will move to the desired sales floor by voice call. For example, send an instruction such as "I want fruit".

The supporter who received the instruction moves based on the instruction (step P6). For example, move to the fruit counter. Subsequently, the supporter HMD100 acquires presentation information for each type of product around the line-of-sight direction of the supporter (process P7), and transmits the acquired presentation information to the supporter HMD400 (process P8). How to obtain the presentation information will be described later.

The supporter HMD400 receives the presentation information sent from the supporter HMD100 and displays the presentation information (step P9). Specifically, the image processing unit 145 (FIG. 6) of the supporter HMD400 is controlled based on the presentation information, and the display control unit 147 (FIG. 6) of the supporter HMD400 is made to display the presentation information. This display is superimposed on the captured image displayed in step P4 . The steps P1, P4, and P9 executed by the supporter HMD400 correspond to the shopping processing unit 455 (FIG. 8).

The supporter wearing the supporter HMD400 sees the displayed information and orders the desired product. The order for the product is sent to the supporter HMD100 by voice call, for example.

D. About shopping support processing:
FIG. 11 is a flowchart showing the shopping support process. This shopping support processing corresponds to the shopping support processing unit 155 (FIG. 6), and in terms of FIG. 10, the processes P2, P3, P7, and P8 are realized. The shopping support process is a process routine that follows a predetermined computer program (application program) stored in the non-volatile storage unit 121 (FIG. 5), and is executed by the main processor 140 of the supporter HMD100. In response to the instruction of the "shopping support" icon IC on the menu screen illustrated in FIG. 7 by the direction key 16 (FIG. 1) and the enter key 17 (FIG. 1), the shopping support process is started.

When the process is started, the main processor 140 of the supporter HMD100 first establishes communication between the supporter HMD100 and the supporter HMD400, and then the supporter HMD400 worn by the shopping supporter. A message indicating that the shopping support has been accepted is sent to the user (step S110). This process corresponds to step P2 in FIG.

Next, the main processor 140 activates the camera 61 (step S120) and transmits the image captured by the camera 61 to the supporter HMD400 (step S130). As described above, the image captured by the camera 61 captures a wider range than the field of view of the supporter. Therefore, here, the periphery of the captured image is deleted so as to match the field of view of the supporter. , Send to the supporter HMD400. That is, an image that matches the field of view of the supporter is transmitted to the supporter HMD400. The process of step S130 corresponds to step P3 of FIG. After that, the captured image will continue to be sent until the shopping support is completed.

Subsequently, the main processor 140 receives the instruction when the instruction is sent from the supporter HMD400 (step S140). For example, you receive an instruction such as "I want fruits." This process is a process on the supporter HMD100 side corresponding to the process P5 of FIG.

FIG. 12 is an explanatory diagram showing an example of a fruit counter in a store. According to step P6 shown in FIG. 10, the supporter moves to, for example, a fruit counter. A display shelf DC is provided in the fruit counter. Fruits F1, F2, F3, and F4 are classified and displayed on the display shelf DC according to their types. For example, the first fruit F1 is a mandarin orange, the second fruit F2 is a banana, the third fruit F3 is an apple, and the fourth fruit F4 is a lemon. For example, from left to right of the display shelf DC, a group C1 of oranges, a group C2 of bananas, a group C3 of apples, and a group C4 of lemons are arranged. BLE terminals 501, 502, 503, 504 are provided at the upper ends of the groups C1, C2, C3, and C4, respectively. BLE (Bluetooth Low Energy) is one of the extended specifications of the short-range wireless communication technology Bluetooth (registered trademark), and is capable of communication with extremely low power. The wireless communication unit 117 (FIG. 5) of the supporter HMD100 is capable of communicating with the BLE terminal. The supporter wearing the supporter HMD100 moves in front of the display shelf DC according to the movement instruction from the supporter.

Returning to FIG. 11, after that, the main processor 140 obtains the line-of-sight direction of the user (supporter) wearing the image display unit 20 (step S150). The "line-of-sight direction" referred to here is the direction in which the user sees with both eyes. In step S150, assuming that the direction of the image display unit 20 matches the line-of-sight direction of the user, the line-of-sight is specified from the detection signal of the 6-axis sensor 235 provided in the image display unit 20 to identify the direction of the image display unit 20. Find the direction. It can be said that the "line-of-sight direction" is the direction connecting the center between the left eye and the right eye and the object (one point) seen by the user with both eyes. The 6-axis sensor 235 corresponds to the "line-of-sight direction detection unit" in one embodiment of the present invention described in the [Summary of Invention] column.

Subsequently, the main processor 140 specifies a predetermined range including the line-of-sight direction obtained in step S150 (step S160). In the present embodiment, one point in the line-of-sight direction is specified, and a predetermined range centered on this one point is specified. "One point in the line-of-sight direction" is a point on the surface of an object that intersects the line-of-sight direction when an object exists in the line-of-sight direction, and when there is no object in the line-of-sight direction, the user in the line-of-sight direction. It is a point separated by a predetermined distance (for example, 5 m) from.

FIG. 13 is an explanatory diagram illustrating a predetermined range VA. The predetermined range VA is a rectangular parallelepiped region, and a single point VP in the line-of-sight direction is located at the center of the rectangular parallelepiped. In step S160, specifically, the main processor 140 obtains one point VP seen by the user based on the line-of-sight direction obtained in step S150 and the distance detected by the distance measuring sensor 62, and the one point VP is obtained. The area extended from 1 / 2Ax on the left side, 1 / 2Ax on the right side, 1 / 2Ay on the upper side, 1 / 2Ay on the lower side, 1/2 Az on the back side, and 1/2 Az on the front side is defined as a predetermined range VA. .. Ax, Ay, and Az are predetermined values. These values may be values that change depending on the location in the store. For example, at the position of the booth where large products are displayed, Ax, Ay, and Az may be large values, and at the position of the booth where small products are displayed, Ax, Ay, and Az may be small values. In this embodiment, Ax, Ay, and Az are stored in the non-volatile storage unit 121 (FIG. 5). In the example of FIG. 13, one point VP is located near the lower part of the apple group C3, and the predetermined range VA is a range including the banana group C2, the apple group C3, and the lemon group C4.

The predetermined range VA does not have to be limited to a predetermined range. For example, a predetermined range may be displayed on the image display unit 20, and the range may be expanded, contracted, or moved based on a user's operation command from the control device 10. ..

Returning to FIG. 11, the main processor 140 subsequently requested a connection from a plurality of BLE terminals (501 to 504 in the example of FIG. 12) existing around the supporter HMD100, and was connected after the connection was established. By transmitting / receiving (communication) data with each BLE terminal 501-504, the relative position of each BLE terminal 501-504 with respect to the supporter HMD100 is obtained (step S170). Subsequently, the main processor 140 selects BLE terminals located within the predetermined range obtained in step S160 from the plurality of BLE terminals 501 to 504 based on the relative positions of the obtained BLE terminals 501 to 504. , Data transmission / reception is performed with each selected BLE terminal to receive product attribute data (step S180). In the example of FIG. 13, since the predetermined range VA includes the BLE terminal 502 of the banana group C2, the BLE terminal 503 of the apple group C3, and the BLE terminal 504 of the lemon group C4, these are included. Receives product attribute data from BLE terminals 502 to 504.

FIG. 14 is an explanatory diagram showing each product attribute data MJ received from each BLE terminal 502 to 504 existing within a predetermined range. The product attribute data MJ is numerical data prepared individually for each type of product, and each item d1 to d5 of "product name code", "production area code", "price", "eating time", and "feature code". including. The item d1 of the "product name code" is a code indicating the name of the product, the item d2 of the "production area code" is a code indicating the production area, and the item d3 of the "price" is the price of the product. Item d4 of "eating time" indicates the period when the product is ready to eat, and item d5 of "feature code" indicates the characteristics of the product. The types of these items d1 to d5 may be arbitrarily determined. In step S180 of FIG. 11, individual product attribute data MJs are received from each of the BLE terminals 502 to 504 located within a predetermined range.

The non-volatile storage unit 121 (FIG. 5) contains a table showing the correspondence between the product name code and the product name (character string), a table showing the correspondence between the production area code and the production area name (character string), and a feature code. A table or the like showing the correspondence between the character and the feature (character string) is stored in advance. Subsequent step S190, the code items d1, d2, d5 in each received product attribute data were converted into a character string by comparing with the above table, and the code portion thus obtained was converted into a character string. The product attribute data is used as the presentation information, and the data indicating the position on the captured image of each BLE terminal 502 to 504 and each presentation information are set and transmitted to the supporter HMD400. Specifically, the position on the captured image of each BLE terminal 502 to 504 existing in the predetermined range VA is obtained from the relative position of each BLE terminal selected in step S170, and the product attribute received from each BLE terminal 502 to 504. The position on the captured image of each BLE terminal 502 to 504 is associated with each presentation information generated based on the data, and the data of each associated set is transmitted to the supporter HMD400.

Subsequently, the main processor 140 activates the microphone 63 (FIG. 1), acquires the voice of the supporter by the microphone 63, and transmits the acquired voice data to the supporter HMD400 (step S200). If the supporter has felt the product by touch, taste, or smell, the supporter will say what he / she feels by voice. For example, if you smell a scent and it has a characteristic, you can give an impression about the scent by voice. If the tactile sensation is characteristic when touched, the impression about the tactile sensation is uttered by voice. For example, if you can taste it, you can taste it and give a voice impression about the taste. In step S200, the voice emitted by the supporter is acquired by the microphone, and the acquired voice data is transmitted to the supporter HMD400. The process of acquiring sound by the microphone 63 and transmitting the acquired sound data to the supporter HMD400 does not necessarily have to be executed at this timing. For example, when the captured image is started to be transmitted in step S130. , The sound may also be started to be transmitted, and thereafter, the sound may be acquired and transmitted until the shopping support process is completed.

After executing step S200, the main processor 140 returns the process to step S130 and repeatedly executes the processes after step S130. The main processor 140 repeats this process until the shopping is completed.

FIG. 15 is an explanatory diagram showing an example of the field of view of the supporter when the shopping support process is executed. When the shopping support process is executed in the supporter HMD100, for example, an image shown in the figure is displayed in the field of view VR2a of the supporter wearing the supporter HMD400 that operates in cooperation with the supporter HMD100. To. This image is an captured image transmitted from the supporter HMD100 in step S130 (FIG. 11), and is an image that the supporter sees through the light guide plates 26 and 28 on the left and right of the supporter HMD100 in the field of view VR2a. The same image as (see FIG. 12) appears.

Further, the field of view VR2a displays the presentation information about the fruit groups C2 to C4 included in the predetermined range VA (see FIG. 13) described above. This presentation information is created based on the set data transmitted from the supporter HMD100 in step S190 (FIG. 11). Specifically, the presentation information included in the set data is displayed at the position on the captured image of each BLE terminal 502 to 504 included in the set data. That is, the presentation information M2 obtained from the BLE terminal 502 of the banana group C2 is displayed at the position of the BLE terminal 502. The presentation information M3 obtained from the BLE terminal 503 of the group C3 of apples is displayed at the position of the BLE terminal 503. The presentation information M4 obtained from the BLE terminal 504 of the group C4 of lemons is displayed at the position of the BLE terminal 504. Each of the presented information M2 to M4 indicates a product name, a place of origin, a price, a time to eat, and a feature.

FIG. 16 is an explanatory diagram showing another example of the field of view of the supporter when the shopping support process is executed. When the supporter wearing the supporter HMD100 moves to the golf club section, the field of view (field of view) VR2b of the supporter wearing the supporter HMD400 that operates in cooperation with the supporter HMD100 is illustrated, for example. The image is displayed. This image is an image showing a golf club section. In the field of view VR2b, the same image as the image (see FIG. 12) that the supporter sees through the light guide plates 26 and 28 on the left and right of the supporter HMD100 appears.

Further, in the field of view VR2a, presentation information about the golf club group included in the predetermined range (not shown) including the line-of-sight direction, the group C12 of the “iron” and the group C13 of the “driver” in the illustrated example is displayed. Will be done. This presentation information is created based on the set data transmitted from the supporter HMD100. Specifically, the presentation information included in the set data is displayed at a position associated with the position on the captured image of each BLE terminal 602, 603 included in the set data. Specifically, based on the position on the captured image of the BLE terminal 602 of the iron group C12, the position of the golf club group corresponding to the BLE terminal 602 is specified by the pattern recognition of the image, and the upper part of the specified position is displayed. The presentation information M12 is displayed as the "associated position". Similarly, the presentation information M13 obtained from the BLE terminal 603 of the driver group C13 is also displayed at a position based on the position of the BLE terminal 603.

E. About the effect of the embodiment:
According to the supporter HMD100 of the present embodiment configured as described above, the presentation information based on the product attribute data received from the BLE terminals 502 to 504 existing in the predetermined range VA including the one-point VP in the line-of-sight direction of the user is provided. , Can be sent to the supporter HMD400. Therefore, the supporter wearing the supporter HMD400 can receive information around the line-of-sight direction of the user while being at a position away from the user wearing the supporter HMD100. Therefore, the supporter HMD100 of the present embodiment can appropriately and efficiently present information in the line-of-sight direction to the supporter at a distant position.

Further, in the present embodiment, since the information processing device used by the supporter is an HMD, the supporter can visually recognize an image that matches the image visually recognized by the supporter who uses the supporter HMD100. Therefore, the supporter can purchase the product as if he / she actually made a purchase in the store. In particular, in the present embodiment, since the HMD is a binocular type, the supporter can share a three-dimensional 3D sensation such as a sense of depth with the supporter.

F. Modification example:
The present invention is not limited to the above-described embodiment and its modifications, and can be implemented in various embodiments without departing from the gist thereof. For example, the following modifications are also possible.

-Modification example 1:
In the embodiment and the modified example, the line-of-sight direction detection unit is composed of the 6-axis sensor 235 provided in the image display unit 20, but the line-of-sight direction detection unit is not limited to this and can be variously deformed. For example, the line-of-sight direction detection unit may be a camera for eyeball photography. The direction of the user's line of sight is detected by imaging the left and right eyeballs of the user with the head-mounted display device attached using an eyeball photographing camera and analyzing the obtained eyeball images. For example, the line-of-sight direction is detected by obtaining the center position of the pupil from the eyeball image. In analyzing the image of the eyeball, even if the human being is staring at the gazing point, the small involuntary eye shaking and the fixation tremor may be taken into consideration.

-Modification example 2:
In the embodiment and the modified example, the region of the rectangular parallelepiped centered on the one-point VP (see FIG. 13) in the line-of-sight direction is specified as the predetermined range VA, but the method for specifying the predetermined range is not limited to this, and various modifications are possible. For example, a region in which one point in the line-of-sight direction is one vertex of a rectangular parallelepiped may be specified as a predetermined range. In short, any configuration can be used as long as one point in the line-of-sight direction is included in a predetermined range. Further, the predetermined range does not necessarily have to be a rectangular parallelepiped, and may be another solid (three-dimensional shape) such as a cube, a sphere, an ellipsoid, a cylinder, a polygonal prism (triangular prism, pentagonal prism, hexagonal prism, etc.). Further, it may be a two-dimensional shape having no depth, a circle, or a polygon (triangle, pentagon, hexagon, etc.). Further, it is not necessary to specify the predetermined range based on one point in the line-of-sight direction, and the predetermined range may be specified based on a part or all of the line-of-sight direction.

-Modification example 3:
In the embodiments and modifications, the display system is suitable for applications that support shopping such as fruits and golf clubs, but the applications are not limited to this and can be variously modified. For example, it may be used for borrowing an article or for discarding an article, and in short, it can be applied when it is desired to make some request for an article from outside the facility based on the situation inside the facility.

-Modification example 4:
In the embodiment and the modified example, the product name, production area, price, eating time, characteristics, etc. are displayed as the presented information, but the presented information is not limited to these and can be variously modified. For example, in the case of food, calories, ingredients, content, unit price per milliliter, traceability information, and the like may be used as presentation information. Traceability is to clarify the processes from cultivation and breeding to processing, manufacturing and distribution in order to ensure food safety. In the case of clothes and shoes, the size, color, inventory, target age group (for example, for Mrs., for young people) and the like may be presented information. In the case of sports equipment, the target skill (for example, for advanced users, beginners, ladies, etc.) may be added as presentation information. Further, location information, store information, payment information (point card, credit card, etc.) and the like may be added to the presentation information.

-Modification example 5:
In the embodiment and the modified example, the image captured by the camera 61 is configured to be transmitted to the supporter HMD 400 after deleting the periphery of the captured image so as to match the field of view of the supporter. On the other hand, in the captured image (captured image for transmission) in which the periphery of the captured image is deleted, a portion overlapping the predetermined range VA specified in step S160 is specified, and the brightness of the specified overlapping portion is determined. The image may be enhanced and the processed image may be transmitted to the supporter HMD400. The image processing is not limited to the configuration for increasing the brightness, and may be processing such as providing a frame around the periphery. According to this configuration, the predetermined range VA in the captured image for transmission can be emphasized as compared with other parts and displayed on the supporter HMD400.

-Modification example 6:
In the embodiment and the modified example, both the supporter and the supporter are equipped with the HMD100 and 400, but the HMD400 on the supporter side is another type such as a tablet computer, a smartphone, a personal computer, a projector, and a television. It can also be used as an information processing device.

-Modification example 7:
In the embodiments and modifications, the short-range wireless communication terminal was a BLE terminal. On the other hand, as a modification, other short-range wireless communication terminals such as a wireless LAN and an infrared communication terminal may be used.

-Modification example 8:
In each embodiment and modification, a part of the configuration realized by hardware may be replaced with software, and conversely, a part of the configuration realized by software may be replaced with hardware. It may be.

-Modification example 9:
In the above embodiment, the configuration of the HMD has been illustrated. However, the configuration of the HMD can be arbitrarily determined within a range that does not deviate from the gist of the present invention, and for example, components can be added, deleted, converted, and the like.

In the above embodiment, the so-called transmissive HMD100 in which the right light guide plate 26 and the left light guide plate 28 transmit external light has been described. However, the present invention can also be applied to, for example, a so-called non-transparent HMD 100 that displays an image in a state in which the outside view cannot be transmitted. Further, in these HMD 100s, in addition to the AR (Augmented Reality) display that displays an image superimposed on the real space described in the above embodiment, MR (Mixed) that displays the captured real space image and the virtual image in combination is displayed. Reality) display or VR (Virtual Reality) display that displays the virtual space can also be performed.

In the above embodiment, the functional units of the control device 10 and the image display unit 20 have been described, but these can be arbitrarily changed. For example, the following aspects may be adopted. A mode in which a storage function unit 122 and a control function unit 150 are mounted on the control device 10, and only a display function is mounted on the image display unit 20. A mode in which a storage function unit 122 and a control function unit 150 are mounted on both the control device 10 and the image display unit 20. A mode in which the control device 10 and the image display unit 20 are integrated. In this case, for example, the image display unit 20 includes all the components of the control device 10 and is configured as a glasses-type wearable computer. A mode in which a smartphone or a portable game device is used instead of the control device 10. A mode in which the control device 10 and the image display unit 20 are connected by wireless communication and a connection cable 40 is arranged. In this case, for example, power supply to the control device 10 and the image display unit 20 may be performed wirelessly.

-Modification example 10:
In the above embodiment, the configuration of the control device has been illustrated. However, the configuration of the control device can be arbitrarily determined within a range that does not deviate from the gist of the present invention, and for example, components can be added, deleted, converted, and the like.

In the above embodiment, an example of the input means included in the control device 10 has been described. However, the control device 10 may be configured by omitting some of the illustrated input means, or may include other input means not described above. For example, the control device 10 may include an operation stick, a keyboard, a mouse, and the like. For example, the control device 10 may include an input means for interpreting a command associated with the movement of the user's body or the like. The movement of the user's body can be acquired by, for example, a line-of-sight detection for detecting the line of sight, a gesture detection for detecting the movement of the hand, a foot switch for detecting the movement of the foot, or the like. The line-of-sight detection can be realized by, for example, a camera that images the inside of the image display unit 20. Gesture detection can be realized, for example, by image analysis of an image taken with time by the camera 61.

In the above embodiment, the control function unit 150 operates by the main processor 140 executing a computer program in the storage function unit 122. However, the control function unit 150 can adopt various configurations. For example, the computer program may store the non-volatile storage unit 121, the EEPROM 215, the memory 118, and other external storage devices (such as a USB memory inserted in various interfaces) in place of the storage function unit 122 or together with the storage function unit 122. It may be stored in a device or an external device such as a server connected via a network). Further, each function of the control function unit 150 may be realized by using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

-Modification example 11:
In the above embodiment, the configuration of the image display unit has been illustrated. However, the configuration of the image display unit can be arbitrarily determined within a range that does not deviate from the gist of the present invention, and for example, components can be added, deleted, converted, and the like.

FIG. 17 is a plan view of a main part showing the configuration of the optical system included in the image display unit of the modified example. In the image display unit of the modified example, an OLED unit 221a corresponding to the user's right eye RE and an OLED unit 241a corresponding to the left eye LE are provided. The OLED unit 221a corresponding to the right eye RE includes an OLED panel 223a that develops a white color and an OLED drive circuit 225 that drives the OLED panel 223a to emit light. A modulation element 227 (modulation device) is arranged between the OLED panel 223a and the right optical system 251. The modulation element 227 is composed of, for example, a transmissive liquid crystal panel, and modulates the light emitted by the OLED panel 223a to generate image light L. The image light L transmitted through the modulation element 227 and modulated is guided to the right eye RE by the right light guide plate 26.

The OLED unit 241a corresponding to the left eye LE includes an OLED panel 243a that emits light in white and an OLED drive circuit 245 that drives the OLED panel 243a to emit light. A modulation element 247 (modulation device) is arranged between the OLED panel 243a and the left optical system 252. The modulation element 247 is composed of, for example, a transmissive liquid crystal panel, and modulates the light emitted by the OLED panel 243a to generate image light L. The image light L transmitted through the modulation element 247 and modulated is guided to the left eye LE by the left light guide plate 28. The modulation elements 227 and 247 are connected to a liquid crystal driver circuit (not shown). This liquid crystal driver circuit (modulation device driving unit) is mounted on a substrate arranged in the vicinity of, for example, the modulation elements 227 and 247.

According to the image display unit of the modified example, the right display unit 22 and the left display unit 24 have the OLED panels 223a and 243a as the light source unit and the image light including a plurality of colored lights by modulating the light emitted by the light source unit, respectively. It is configured as a video element including a modulation element 227 and 247 that output the light source. The modulation device that modulates the light emitted by the OLED panels 223a and 243a is not limited to the configuration in which the transmissive liquid crystal panel is adopted. For example, a reflective liquid crystal panel may be used instead of the transmissive liquid crystal panel, a digital micromirror device may be used, or a laser retinal projection type HMD100 may be used.

In the above embodiment, the eyeglass-shaped image display unit 20 has been described, but the aspect of the image display unit 20 can be arbitrarily changed. For example, the image display unit 20 may be worn like a hat, or may be built into a body protective device such as a helmet. Further, the image display unit 20 may be configured as a HUD (Head Up Display) mounted on a vehicle such as an automobile or an airplane, or other means of transportation.

In the above embodiment, as an optical system that guides the image light to the eyes of the user, a configuration in which a virtual image is formed on a part of the right light guide plate 26 and the left light guide plate 28 by the half mirrors 261 and 281 is illustrated. However, this configuration can be changed arbitrarily. For example, a virtual image may be formed in a region that occupies the entire surface (or most) of the right light guide plate 26 and the left light guide plate 28. In this case, the image may be reduced by the operation of changing the display position of the image. Further, the optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having half mirrors 261 and 281, and an optical component (for example, a diffraction grating, a prism, etc.) that allows image light to enter the user's eye. Any aspect can be adopted as long as holography or the like is used.

-Modification example 12:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations within a range not deviating from the gist thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the column of the outline of the invention may be used to solve some or all of the above-mentioned problems. , It is possible to replace or combine as appropriate in order to achieve some or all of the above effects. Further, if the technical feature is not described as essential in the present specification, it can be appropriately deleted.

1 ... Display system, 10 ... Control device, 12 ... Lighting unit, 14 ... Touch pad, 16 ... Direction key, 17 ... Enter key, 18 ... Power switch, 19 ... Vibrator, 20 ... Image display unit, 21 ... Right holding unit , 22 ... right display unit, 23 ... left holding unit, 24 ... left display unit, 26 ... right light guide plate, 27 ... front frame, 28 ... left light guide plate, 30 ... headset, 32 ... right earphone, 34 ... left Earphones, 40 ... connection cables, 46 ... connectors, 61 ... cameras, 62 ... distance measuring sensors, 63 ... microphones, 65 ... illuminance sensors, 67 ... LED indicators, 100 ... head-mounted display devices (HMD), 110 ... operations Unit, 111 ... 6-axis sensor, 113 ... Magnetic sensor, 115 ... GPS receiver, 117 ... Wireless communication unit, 118 ... Memory, 120 ... Controller board, 121 ... Non-volatile storage unit, 122 ... Storage function unit, 123 ... Setting data , 124 ... content data, 130 ... power supply unit, 132 ... battery, 134 ... power supply control circuit, 140 ... main processor, 145 ... image processing unit, 147 ... display control unit, 149 ... imaging control unit, 150 ... control function unit, 151 ... Input / output control unit, 153 ... Communication control unit, 155 ... Shopping support processing unit, 180 ... Voice codec, 182 ... Voice interface, 184 ... External connector, 186 ... External memory interface, 188 ... USB connector, 192 ... Sensor hub , 196 ... interface, 210 ... display unit board, 211 ... interface, 213 ... receiver, 215 ... OLED, 217 ... temperature sensor, 221.221a ... OLED unit, 223 ... 223a ... OLED panel, 225 ... OLED drive circuit, 227. ... Modulation element, 230 ... Display unit board, 231 ... Interface, 233 ... Receiver, 235 ... 6-axis sensor, 237 ... Magnetic sensor, 239 ... Temperature sensor, 241, 241a ... OLED unit, 243, 243a ... OLED panel, 245 ... OLED drive circuit, 247 ... Modulator, 251 ... Right optical system, 252 ... Left optical system, 261 ... Half mirror, 281 ... Half mirror, 400 ... Head-mounted display device (HMD), 450 ... Control function unit, 455 ... Shopping processing unit, 501-504 ... BLE terminal, 601-603 ... BLE terminal, MJ ... Product attribute data, VA ... Predetermined range, VP ... One point in the line-of-sight direction

Claims (9)

A head-mounted display device that supports shopping
A line-of-sight direction detection unit that detects the line-of-sight direction of the user,
A camera that captures the direction that the user sees, and
A voice interface that conveys information to the user,
With the wireless communication department
Via the wireless communication unit, and a processing unit that had line transmitting and receiving information to and from the external information processing apparatus, to convey information to the user via the voice interface,
With
The processing unit
Upon receiving the instruction of the destination from the information processing device via the wireless communication unit,
Information on the destination is transmitted to the user via the voice interface.
At least a part of the image captured by the camera is transmitted as a transmission image to the information processing apparatus via the wireless communication unit.
A predetermined range including the line-of-sight direction detected by the line-of-sight direction detection unit is specified.
Communication is performed with the short-range wireless communication terminal existing in the specified predetermined range via the wireless communication unit, and data is received from the short-range wireless communication terminal.
The presentation information based on the received data is transmitted to the information processing device, and the presentation information is transmitted .
Receiving the instruction of the moving destination, transmitting the information of the moving destination, transmitting the captured image for transmission to the information processing device, specifying the predetermined range, and displaying the data from the short-range wireless communication terminal. Receiving and transmitting the presented information to the information processing device are repeatedly executed .
Head-mounted display device. The head-mounted display device according to claim 1 .
The processing unit
A portion of the captured image for transmission that overlaps with the predetermined range is specified, and image processing is performed on the overlapping portion.
Head-mounted display device. The head-mounted display device according to any one of claims 1 to 2.
The data received from the short-range wireless communication terminal existing in the predetermined range is data for each type of product displayed in the shopping facility.
Head-mounted display device. The head-mounted display device according to any one of claims 1 to 3.
The external information processing device is another head-mounted display device different from the head-mounted display device.
Head-mounted display device. The head-mounted display device according to any one of claims 1 to 4.
The processing unit
The position of the short-range wireless communication terminal existing in the predetermined range on the captured image is obtained.
A head-mounted display device that associates the presented information with the position information and transmits the information to the information processing device. Includes a head-mounted display device, and an information processing apparatus, and a display system that helps you shopping,
The head-mounted display device is
A line-of-sight direction detection unit that detects the line-of-sight direction of the user,
A camera that captures the direction that the user sees, and
A voice interface that conveys information to the user,
With the wireless communication department
And via said wireless communication unit, the have line transmission and reception of information with the information processing apparatus, the processing unit for transmitting information to the user via the voice interface,
With
The processing unit
Upon receiving the instruction of the destination from the information processing device via the wireless communication unit,
Information on the destination is transmitted to the user via the voice interface.
At least a part of the image captured by the camera is transmitted as a transmission captured image to the information processing apparatus via the wireless communication unit.
A predetermined range including the line-of-sight direction detected by the line-of-sight direction detection unit is specified.
Communication is performed with the short-range wireless communication terminal existing in the specified predetermined range via the wireless communication unit, and data is received from the short-range wireless communication terminal.
The presentation information based on the received data is transmitted to the information processing device, and the presentation information is transmitted.
Receiving the instruction of the moving destination, transmitting the information of the moving destination, transmitting the captured image for transmission to the information processing device, specifying the predetermined range, and data from the short-range wireless communication terminal. Receiving and transmitting the presented information to the information processing device are repeatedly executed.
The information processing device
The instruction of the destination is transmitted to the head-mounted display device, and the instruction is transmitted to the head-mounted display device.
The image for transmission is received from the head-mounted display device, and the image is received.
Upon receiving the presentation information sent from the head-mounted display device,
Display is performed based on the received captured image for transmission and the presented information.
Display system. The display system according to claim 6.
The processing unit
Obtaining the position of the short-range wireless communication terminal existing in the predetermined range,
The information of the position is associated with the presented information and transmitted to the information processing apparatus.
The information processing device
A display system that displays the presented information at the position of the short-range wireless communication terminal in the display based on the captured image for transmission based on the position information. It is equipped with a line-of-sight direction detection unit that detects the line-of-sight direction of the user, a camera that captures the direction that the user sees, a voice interface that transmits information to the user, and a wireless communication unit to support shopping. This is a control method for a head-mounted display device.
A process of receiving a movement destination instruction from an external information processing device via the wireless communication unit, and
A step of transmitting information on the destination to the user via the voice interface, and
A step of transmitting at least a part of the image captured by the camera as a transmission captured image to the information processing apparatus via the wireless communication unit.
Via the wireless communication unit, and performing presentation of information to the information processing apparatus,
A step of repeatedly executing each of the steps is provided.
The step of presenting information to the information processing device is
A predetermined range including the line-of-sight direction detected by the line-of-sight direction detection unit is specified.
Communication is performed with the short-range wireless communication terminal existing in the specified predetermined range via the wireless communication unit, and data is received from the short-range wireless communication terminal.
The presentation information based on the received data is transmitted to the information processing apparatus.
How to control a head-mounted display device. It is equipped with a line-of-sight direction detection unit that detects the line-of-sight direction of the user, a camera that captures the direction that the user sees, a voice interface that transmits information to the user, and a wireless communication unit to support shopping. A computer program for controlling a head-mounted display device.
A function of receiving a movement destination instruction from an external information processing device via the wireless communication unit, and
A function of transmitting information on the destination to the user via the voice interface, and
A function of transmitting at least a part of the image captured by the camera as a transmission captured image to the information processing device via the wireless communication unit.
Via the wireless communication unit, a function for presenting information to the information processing apparatus,
A computer that repeatedly executes each of the above functions is realized.
The function of presenting information to the information processing device is
A predetermined range including the line-of-sight direction detected by the line-of-sight direction detection unit is specified.
Communication is performed with the short-range wireless communication terminal existing in the specified predetermined range via the wireless communication unit, and data is received from the short-range wireless communication terminal.
The presentation information based on the received data is transmitted to the information processing apparatus.
Computer program.

Images