WO2018147084A1

WO2018147084A1 - Line-of-sight information sharing method and line-of-sight information sharing system

Info

Publication number: WO2018147084A1
Application number: PCT/JP2018/002142
Authority: WO
Inventors: 笠井　一郎; 平岡　三郎
Original assignee: コニカミノルタ株式会社
Priority date: 2017-02-07
Filing date: 2018-01-24
Publication date: 2018-08-16

Abstract

A line-of-sight information sharing method that includes: a first step for outputting information about a photographic image obtained with an imaging device of a first terminal (HMD) to a second terminal; a second step for displaying the photographic image at the second terminal so as to be visible to a user of the second terminal on the basis of the input photographic image information; a third step for detecting the line-of-sight direction of the user of the second terminal with respect to the displayed photographic image; a fourth step for outputting line-of-sight information associated with the detected line-of-sight direction of the user of the second terminal to the first terminal; and a fifth step for displaying the line-of-sight position of the user of the second terminal at a position corresponding to a position within the field of view of the user of the first terminal on the basis of the input line-of-sight information.

Description

Gaze information sharing method and gaze information sharing system

The present invention relates to a line-of-sight information sharing method and a line-of-sight information sharing system for sharing line-of-sight information of a user between a plurality of information input / output terminals connected via a communication line.

Conventionally, a head-mounted see-through information display device called a see-through HMD (Head Mounted Display) has been known. This see-through HMD is a device that can provide desired information to a wearer by adding desired information as an image when observing the outside world, and is roughly classified into a video see-through type and an optical see-through type.

In the video see-through type, a camera that captures the wearer's front view is added to a shielded (closed) display device, and a desired image is superimposed on the image that captures the wearer's front view, and these are integrated. It is a device that allows the wearer to visually recognize the image. On the other hand, the optical see-through type is a device that uses a combiner that superimposes light from the outside and image light, and guides the light to the wearer's pupil by the combiner, thereby allowing the wearer to visually recognize the outside world together with the image.

For any of the above-described types of see-through HMDs, there is a proposal of utilization that complements and assists the work of the wearer by superimposing a desired image on a visually recognized external image (hereinafter also referred to as an external image). By making the outside world observable in combination with the hands-free nature of the HMD, it prevents the wearer from interfering with normal work, and displays necessary information (for example, work instructions and manuals) superimposed on the outside world image. Thus, the workability of the wearer can be improved.

An example of supplementing the wearer's work using the HMD is proposed in Patent Document 1, for example. In Patent Document 1, an operator and an instructor wear a video see-through type HMD, and the outside world in front of the operator is imaged by a camera, and the image is transmitted to an instructor at a remote location and shared. Then, the instructor adds additional information to the display video, so that the instruction is transmitted to the worker and the collaborative work is performed. As a means for adding additional information, a hand-held type (hand-held type, handy type) input device is used.

However, when a handheld input device is used to add additional information, the hands are blocked by the input device, and the hands-free property of the original HMD is impaired. For this reason, when work by hand is required, such as actually tightening a screw of an object or removing a part using an HMD, the workability is deteriorated.

In addition, in collaborative work with multiple people as described above, or in the case where an operator performs an actual work in response to an instruction from the instructor, display by voice or voice input (display by converting the voice to type) Communication method). However, when specifying a work location for an actual object, it is difficult to specify the work location with an ambiguous voice or display instruction such as “this”, “that”, “there”. Also, when trying to communicate in detail, for example, when instructing a work on a specific bolt, the instruction content becomes long, such as “second bolt from the top in the third column from the right”. To be complicated. For this reason, it is difficult to accurately communicate only by display by voice or voice input.

Therefore, in the collaborative / collaborative work of multiple people using HMD, there is a need for a means for accurately communicating each other's intentions without compromising the hands-free nature of HMD.

On the other hand, in recent years, there is also a proposal for detecting the gaze of the instructor and utilizing the detected gaze for communication with the worker. For example, in Patent Document 2, the line of sight of a person (instructor) in the passenger seat of a vehicle is detected, and the see-through display (transmission head-up display) in front of the driver (operator) corresponds to the line of sight of the instructor. The line-of-sight marker is coordinate-transformed and presented (displayed) so as to be in an appropriate position when viewed from the operator's viewpoint, thereby achieving mutual communication.

For example, from the line-of-sight position when a person in the passenger's seat emits an instruction word such as “There” or “Ano”, the target position of the instruction word is obtained and the line-of-sight position of the speaker (passenger seat person) is determined by the driver. By converting to a display position on the head-up display that is visually recognized and superimposed, information on the line of sight of the speaker can be shared with the driver. As a result, the driver can accurately know the intention of the speaker. In addition, both the driver and the speaker do not need manual work for communication, and hands-freeness is ensured.

JP 2006-209664 A (refer to claim 1, paragraphs [0007], [0019], [0102], FIG. 1, FIG. 2, etc.) Japanese Patent Laying-Open No. 2015-41197 (refer to claim 1, paragraph [0010], FIG. 7 etc.)

However, it is difficult for the following two reasons to apply the system of Patent Document 2 to a purpose of performing a collaborative work using a plurality of information input / output terminals including an HMD.

(Reason 1)
The system of Patent Document 2 presents information on the line of sight of a speaker with respect to an object that can be directly and directly viewed by two nearby speakers and a driver. For this reason, the system of patent document 2 cannot be utilized in the situation where both cannot recognize a target object simultaneously or directly. For example, when one of the instructor and the worker who performs the collaborative work (for example, the instructor) is in a remote place, the instructor cannot directly see the same object. Even if the instructor and the worker are adjacent to each other, if the work place is small and the worker is close to the target object, the target object is hidden from the direct operator by the operator. It cannot be visually recognized. In these cases, since the system of Patent Document 2 cannot be used, accurate communication based on the line of sight cannot be performed among users who use a plurality of information input / output terminals.

(Reason 2)
In the system of Patent Document 2, the line of sight of the speaker for the same object and the line of sight of the driver are different. Need to be converted to For this purpose, a process of spatially grasping the relative positions of the speaker, the driver, and the object, and converting the direction of the speaker's line of sight with respect to the object into the line of sight of the driver (position coordinate conversion process). It is necessary to do. Therefore, when such a system of Patent Document 2 is applied to a use in which a collaborative work is performed by a plurality of people, the system becomes large and complicated, the processing load increases, and it corresponds to the line of sight of the operator or the instructor. It is considered that the display cannot be performed immediately. It is not desirable that the line-of-sight display is delayed because it interferes with communication based on the line-of-sight.

Therefore, in applications where collaborative work is performed using a plurality of information input / output terminals including the HMD, accurate communication based on the line of sight is performed even if any user cannot visually recognize the object. Realize a system that can communicate based on the line of sight with a simple method without requiring complicated processing such as coordinate conversion by grasping the mutual spatial position. It is desirable.

The present invention has been made to solve the above-described problems, and its purpose is that any user cannot visually recognize an object in a collaborative work using a plurality of information input / output terminals including an HMD. Gaze information sharing that enables accurate communication based on the line of sight even in such situations, and allows easy communication based on the line of sight without requiring complicated processing It is to provide a method and a line-of-sight information sharing system.

A line-of-sight information sharing method according to an aspect of the present invention is a line-of-sight information sharing method for sharing line-of-sight information of a user between a plurality of information input / output terminals connected via a communication line, wherein the plurality of pieces of information Each of the input / output terminals includes a display device that presents an image to a user, a line-of-sight detection device that detects a user's line-of-sight direction, and a communication unit that inputs and outputs information to and from each other. At least one of the input / output terminals is a head-mounted terminal that is mounted on a user's head and displays the video as a virtual image so as to be visible to the user, and the head-mounted terminal includes the display device. In addition to the line-of-sight detection device and the communication unit, a relative position is fixed with respect to the display device, and further includes an imaging device that images the external world in front of the user. Any of the head mounted end Is the first terminal, and at least one other information input / output terminal is the second terminal, the line-of-sight information sharing method uses the captured image information acquired by the imaging device of the first terminal as Based on the first step of outputting to the second terminal and the input information of the captured image, the second terminal displays the captured image so as to be visible to the user of the second terminal. The third step of detecting the line-of-sight direction of the user of the second terminal with respect to the displayed captured image, and the line-of-sight information relating to the detected line-of-sight direction of the user of the second terminal, Based on the fourth step of outputting to one terminal and the input line-of-sight information, the position of the line of sight of the user of the second terminal is set to a corresponding position in the display area of the video by the first terminal. And a fifth step of displaying.

A line-of-sight information sharing system according to another aspect of the present invention is a line-of-sight information sharing system for sharing line-of-sight information of a user among a plurality of information input / output terminals connected via a communication line, Each of the plurality of information input / output terminals includes a display device that presents an image to the user, a line-of-sight detection device that detects a user's line-of-sight direction, and a communication unit for inputting and outputting information to and from each other, At least one of the plurality of information input / output terminals is a head-mounted terminal that is mounted on a user's head and displays the video as a virtual image so as to be visible to the user. In addition to the display device and the line-of-sight detection device, the image processing device further includes an imaging device whose relative position is fixed with respect to the display device and images the external environment in front of the user, The head-mounted When the terminal is the first terminal and the other at least one information input / output terminal is the second terminal, the first terminal receives the information of the captured image acquired by the imaging device of the first terminal. The second terminal outputs the captured image to the second terminal based on the information of the captured image input by the display device of the second terminal via the communication unit of one terminal. Displayed so as to be visible to the user of the terminal, and the gaze detection device of the second terminal detects the gaze direction of the user of the second terminal with respect to the displayed captured image, and then gaze information related to the gaze direction is displayed. Output to the first terminal via the communication unit of the second terminal, and the first terminal uses the second terminal based on the line-of-sight information input by the display device of the first terminal. Display position of the video by the first terminal. The corresponding position displays within.

According to the above-described line-of-sight information sharing method and line-of-sight information sharing system, in a collaborative work using a plurality of information input / output terminals including the first terminal (HMD), any user cannot visually recognize an object. It is possible to communicate accurately based on the line of sight even in situations, and communicate based on the line of sight in a simple manner without requiring complicated processing such as conversion of position coordinates. Is possible.

It is explanatory drawing which shows the structure of the outline of the gaze information sharing system which concerns on one Embodiment of this invention. It is a front view which shows the outline structure of HMD which is an example of the information input / output terminal which comprises the said gaze information sharing system. It is sectional drawing which shows the optical structure of the video display apparatus with which said HMD is provided. It is a block diagram which shows the detailed structure of said HMD. It is explanatory drawing which shows typically a mode that an operator and an instruction | inspection supervisor perform a collaborative work. It is a flowchart which shows an example of the flow of operation | movement in the said gaze information sharing system. It is explanatory drawing which shows the image of an operator's visual field. It is explanatory drawing which shows a mode that the instruction | inspection supervisor observes a display image | video. It is explanatory drawing which shows the image of an operator's visual field when the marker which shows the instruction | inspection supervisor's eyes | visual_axis position is displayed. It is a flowchart which shows the other example of the flow of operation | movement in the said gaze information sharing system. It is explanatory drawing which shows the image of an operator's visual field when the position of the eyes | visual_axis of both an operator and an instruction | inspection supervisor is displayed. It is a flowchart which shows the further another example of the flow of operation | movement in the said gaze information sharing system. It is explanatory drawing which shows the image of the field of view of an instruction | inspection supervisor when the position of the eyes | visual_axis of both an operator and an instruction | indication supervisor is displayed. It is a block diagram which shows the other structure of the said HMD. It is explanatory drawing which shows a mode that a worker receives the instruction | indication based on line-of-sight information from a long-distance instruction | inspection supervisor. It is explanatory drawing which shows typically a mode that two workers perform a collaborative work in the gaze information sharing system which concerns on other embodiment of this invention. It is explanatory drawing which shows the image of one worker's visual field. It is explanatory drawing which shows the image of the visual field of the other operator. It is explanatory drawing which shows the image of the visual field of one worker when the visual field image of the other worker is displayed by one worker's HMD. It is explanatory drawing which shows the image of the visual field of the other worker when the visual field image of one worker is displayed by HMD of the other worker. It is explanatory drawing which shows the image of the visual field of one worker when the marker which shows the line-of-sight position of two workers is displayed. It is explanatory drawing which shows the image of the other worker's visual field when the marker which shows the visual line position of two workers is displayed. FIG. 10 is an explanatory diagram schematically showing a state in which an instruction supervisor observes an image sent and displayed by two workers in a line-of-sight information sharing system according to still another embodiment of the present invention. . It is explanatory drawing which shows the image of one worker's visual field when the marker which shows the visual line position of two workers and an instruction | inspection supervisor is displayed. It is explanatory drawing which shows the image of the field of view of the other operator when the marker which shows the gaze position of two workers and an instruction | inspection supervisor is displayed. It is explanatory drawing which shows the other structure of the information input / output terminal which comprises the said gaze information sharing system. In the line-of-sight information sharing system according to still another embodiment of the present invention, it is an explanatory view schematically showing a relative positional relationship between an image display device, an imaging device, and an object located at infinity. It is a block diagram which shows the structure of HMD which performs parallax correction. It is sectional drawing which shows the other optical structure of HMD. It is sectional drawing which shows other optical structure of HMD. It is sectional drawing which shows other optical structure of HMD.

[Embodiment 1]
An embodiment of the present invention will be described below with reference to the drawings. In this specification, when the numerical range is expressed as a to b, the numerical value range includes the values of the lower limit a and the upper limit b. The present invention is not limited to the following contents.

(Gaze information sharing system configuration)
FIG. 1 is an explanatory diagram illustrating a schematic configuration of a line-of-sight information sharing system 100 according to the present embodiment. The line-of-sight information sharing system 100 is configured by connecting a plurality of information input / output terminals 200 via a communication line 300 so that they can communicate with each other (so that information can be linked). The communication line 300 is realized by a wireless communication environment such as Wi-Fi (registered trademark), but may be a wired communication line using a cable such as an optical fiber.

The plurality of information input / output terminals 200 can input / output (transmit / receive) information to / from each other. Examples of information to be input / output include information on captured images (image data) acquired by each information input / output terminal 200 and information on the line-of-sight direction and the position of the line of sight of the user of each information input / output terminal 200 (Hereinafter also simply referred to as line-of-sight information). As a result, the user's line-of-sight information is shared among the plurality of information input / output terminals 200, and communication based on the user's line-of-sight can be performed as described later.

At least one of the plurality of information input / output terminals 200 is a head-mounted display (HMD) that is mounted on the user's head and displays the image as a virtual image so as to be visible to the user. . Details of the HMD will be described below.

(Configuration of HMD)
FIG. 2 is a front view illustrating a schematic configuration of the HMD 201 which is an example of the information input / output terminal 200 of the present embodiment. The HMD 201 includes an image display device 202 that presents an image to the user and a support member 203. The support member 203 is a member that supports the video display device 202 in front of an observer (user of the HMD 201) (for example, in front of the right eye ER). The support member 203 corresponds to a frame or temple of glasses, and a support unit. 203a includes a nose pad 203b attached to 203a and abutting the user's nose when worn.

The HMD 201 may further include a right-eye lens and a left-eye lens, and these may be supported by the support member 203. Further, the HMD 201 may have two video display devices 202 and support each video display device 202 in front of the right and left eyes of the user by the support member 203. Furthermore, the HMD 201 may have a position adjustment mechanism that adjusts the position (for example, each position in the left-right direction and the up-down direction) and the attachment angle (tilt angle) of the video display device 202.

The video display device 202 is an optical see-through display that allows a user to directly observe the outside world together with video. In other words, the video display device 202 guides light from the outside world to the user's pupil, thereby allowing the user to observe the outside world and displaying the video as a virtual image in a part of the user's field of view. It is a system. Hereinafter, the optical configuration of the video display device 202 will be described.

FIG. 3 is a cross-sectional view showing an optical configuration of the image display device 202. The video display device 202 includes an illumination optical system 2, a polarizing plate 3, a polarization beam splitter (PBS) 4, a display element 5, and an eyepiece optical system 6. The upper ends of the illumination optical system 2, the polarizing plate 3, the PBS 4, the display element 5, and the eyepiece optical system 6 are located in the housing 202a shown in FIG.

For convenience of explanation below, each direction is defined as follows. In FIG. 3, an axis that optically connects the center of the optical pupil P formed by the eyepiece optical system 6 (the user's pupil center at the time of image observation) and the center of the display surface of the display element 5, and an extension of the axis. Is the optical axis (observation central axis). A direction perpendicular to the optical axis plane of the HOE (Holographic Optical Element) 23 of the eyepiece optical system 6 is defined as an X direction. Note that the optical axis plane of the HOE 23 refers to a plane including incident light and reflected light when a light beam that coincides with the optical axis enters the HOE 23. In addition, the direction perpendicular to the X direction in the plane perpendicular to the surface normal at the intersection of each optical member with the optical axis is defined as the Y direction. A direction perpendicular to the X direction and the Y direction is taken as a Z direction. When such a definition is used, for example, the normal line of the display element 5 and the normal lines of two

parallel surfaces

21b and 21c described later of the eyepiece optical system 6 are included, and the center of the display surface of the display element 5 is defined. The included cross section is a YZ cross section.

The illumination optical system 2 illuminates the display element 5 and has a light source 11, an illumination mirror 12, and a diffusion plate 13.

The light source 11 is composed of RGB integrated LEDs that emit light corresponding to each color of R (red), G (green), and B (blue). A plurality of light emission points (each light emission point of RGB) are arranged in a substantially straight line in the horizontal direction (X direction). The wavelength of light emitted from the light source 11 is, for example, a peak wavelength of light intensity and a wavelength width of half value of light intensity, 462 ± 12 nm (B light), 525 ± 17 nm (G light), 635 ± 11 nm (R light). It is. The light source 11 may be a laser light source.

The illumination mirror 12 reflects light (illumination light) emitted from the light source 11 toward the diffuser plate 13 and bends the illumination light so that the optical pupil P and the light source 11 are substantially conjugate with respect to the Y direction. It is an optical element.

The diffusing plate 13 is a unidirectional diffusing plate that diffuses incident light, for example, 40 ° in the X direction in which a plurality of light emitting points of the light source 11 are arranged and does not diffuse incident light in the Y direction. The diffusion plate 13 is held on the surface of the polarizing plate 3.

The polarizing plate 3 transmits light having a predetermined polarization direction out of light incident through the diffusion plate 13 and guides it to the PBS 4.

The PBS 4 reflects the light transmitted through the polarizing plate 3 in the direction of the reflective display element 5, while out of the light reflected by the display element 5, the light corresponding to the image signal ON (transmitted through the polarizing plate 3). The light is a flat plate-shaped polarization separating element that transmits light whose polarization direction is orthogonal, and is attached to a light incident surface 21a of an eyepiece prism 21 (to be described later) of the eyepiece optical system 6.

The display element 5 is a display element that modulates the light from the illumination optical system 2 and displays an image. In the present embodiment, the display element 5 is composed of a reflective liquid crystal display element. The display element 5 may have a configuration having a color filter, and is driven in a time division manner so that an RGB image corresponding to the emission color is displayed in synchronization with the time division emission for each RGB of the light source 11. It may be a configuration.

The display element 5 is arranged so that light incident from the PBS 4 substantially vertically is reflected almost vertically and directed toward the PBS 4. This facilitates optical design that increases the resolution as compared with a configuration in which light is incident on the reflective display element at a large incident angle. The display surface of the display element 5 is rectangular, and is arranged so that the longitudinal direction of the display surface is the X direction and the short direction is the Y direction.

The eyepiece optical system 6 is an optical system for guiding the image light from the display element 5 to the user's pupil (optical pupil P), and has non-axisymmetric (non-rotationally symmetric) positive optical power. . The eyepiece optical system 6 includes an eyepiece prism 21, a deflection prism 22, and a HOE 23.

The eyepiece prism 21 guides the image light incident from the display element 5 through the PBS 4 inside, and transmits light from the outside (external light). The upper end of the parallel plate is directed toward the upper end. It is configured to be thicker and thinner at the lower end toward the lower end.

In the eyepiece prism 21, the surface to which the PBS 4 is attached is a light incident surface 21 a on which the image light from the display element 5 is incident, and two

surfaces

21 b and 21 c that are positioned substantially parallel to the optical pupil P and face each other are The total reflection surface guides the image light by total reflection. Among them, the surface 21b on the optical pupil P side also serves as an image light exit surface that is diffracted and reflected by the HOE 23.

The eyepiece prism 21 is joined to the deflection prism 22 with an adhesive so as to sandwich the HOE 23 arranged at the lower end thereof. In the present embodiment, among the surfaces constituting the eyepiece prism 21, the surfaces (light incident surface 21 a and surface 21 b) that transmit image light other than the surface 21 d in contact with the HOE 23 are flat. In the eyepiece prism 21, the surface 21d with which the HOE 23 comes into contact may be a flat surface, a curved surface, or a surface combining a flat surface and a curved surface.

The deflection prism 22 is bonded to the eyepiece prism 21 via the HOE 23 to form a substantially parallel plate. By attaching the deflecting prism 22 to the eyepiece prism 21, refraction when external light passes through the wedge-shaped lower end of the eyepiece prism 21 can be canceled by the deflecting prism 22, and an image observed (viewed) as the outside world. It is possible to prevent distortion in the (external world image).

The HOE 23 is a volume phase type reflective holographic optical element that is provided in contact with the eyepiece prism 21 and diffracts and reflects the image light guided inside the eyepiece prism 21. The HOE 23 diffracts light in three wavelength ranges of, for example, 465 ± 5 nm (B light), 521 ± 5 nm (G light), and 634 ± 5 nm (R light) with a peak wavelength of diffraction efficiency and a half width of the diffraction efficiency. (Reflect). That is, the RGB diffraction wavelength of the HOE 23 substantially corresponds to the wavelength of RGB image light (the emission wavelength of the light source 11).

In the above configuration, the light emitted from the light source 11 of the illumination optical system 2 is reflected by the illumination mirror 12 and diffused only in the X direction by the diffusion plate 13, and then only the light having a predetermined polarization direction is a polarizing plate. 3 is transmitted. The light transmitted through the polarizing plate 3 is reflected by the PBS 4 and enters the display element 5.

In the display element 5, incident light is modulated in accordance with an image signal. At this time, since the image light corresponding to the image signal ON is converted by the display element 5 into light having a polarization direction orthogonal to the incident light and emitted, the light is incident on the eyepiece prism 21 through the PBS 4. Incident from the surface 21a. On the other hand, the image light corresponding to the image signal being turned off is emitted by the display element 5 without changing the polarization direction, and thus is blocked by the PBS 4 and does not enter the eyepiece prism 21.

In the eyepiece prism 21, the incident image light is totally reflected once by the two opposing

surfaces

21 c and 21 b of the eyepiece prism 21, then enters the HOE 23, where it is diffracted and reflected and emitted from the surface 21 b. Reach pupil P. Therefore, at the position of the optical pupil P, the user can observe the image displayed on the display element 5 as a virtual image.

On the other hand, the eyepiece prism 21, the deflecting prism 22, and the HOE 23 transmit almost all of the external light, so that the user can observe the outside world see-through. Therefore, the virtual image of the image displayed on the display element 5 is observed while overlapping a part of the outside world in the user's field of view (view).

In this embodiment, a reflective image display element is used as the display element 5, but a transmissive liquid crystal display element may be used, and the image display device 202 may be configured with an optical design corresponding thereto. The eyepiece optical system 6 may be configured integrally with the right-eye lens or the left-eye lens of the HMD 201, or may be configured separately.

(Detailed configuration of HMD)
FIG. 4 is a block diagram illustrating a detailed configuration of the HMD 201. In addition to the video display device 202 and the support member 203 described above, the HMD 201 includes an imaging device 31, a line-of-sight detection device 32, an image processing unit 33, a communication unit 34, a storage unit 35, a voice input device 36, a voice output device 37, and A control unit 38 is further provided. The control unit 38 is composed of, for example, a central processing unit (CPU) and controls the operation of each unit of the HMD 201 including the video display device 202. Therefore, the control of the control unit 38 includes, for example, switching on / off of the light source 11 of the video display device 202 and control regarding display of information on the display element 5.

The imaging device 31 is composed of, for example, a camera (digital camera, digital video camera) capable of shooting a moving image, and images the outside world ahead of the user's line of sight. For example, the imaging device 31 is provided on the outer surface of the housing 202a illustrated in FIG. 2, but may be provided inside the housing 202a. In the latter case, an opening penetrating the housing 202a or a transparent window is provided in front of the image pickup device 31, so that the image pickup device 31 can shoot in front.

As shown in FIG. 2, the imaging center Co (intersection between the imaging optical axis and the imaging element) of the imaging device 31 is located almost immediately above the observation center Vo (pupil center) of the video display device 202. The relative position of the imaging device 31 with respect to the video display device 202 is fixed. Further, the imaging optical axis of the imaging device 31 substantially coincides with the viewing direction of the user when observing at infinity (see imaging optical axis A ₁ and viewing direction (observation central axis) A _{2 in} FIG. 27). . Thereby, the imaging device 31 can image a user's normal field of view (front outside).

The gaze detection device 32 is a sensor that detects the user's observation gaze (gaze direction). The line-of-sight detection device 32 includes an infrared LED and a camera, and is provided at the left and right edges of the eyepiece optical system 6, for example, as shown in FIG. The line-of-sight detection device 32 directly detects the user's line-of-sight direction by irradiating a safe ray (for example, infrared rays) from the infrared LED toward the user's pupil or retina and measuring the reflected light with a camera. More specifically, when the infrared LED illuminates the user's face, the position where the infrared light is reflected on the user's cornea (the position of the corneal reflection) is used as a reference point, and the position corresponding to this corneal reflection is determined. The direction of the user's line of sight is detected by detecting the position of the pupil with a camera (by receiving infrared rays). For example, if the pupil is closer to the corneal reflection of the right eye, the user is looking to the right (the user's line of sight is pointing to the right), and the pupil is larger than the corneal reflection. If it is on the side, it is possible to detect that the user is looking at the left side (the user's line of sight is facing left).

Although the number of the line-of-sight detection devices 32 may be one or plural, it is possible to perform line-of-sight detection based on a plurality of pieces of information and improve detection accuracy. This is desirable. For this reason, in this embodiment, as shown in FIG. 2, the gaze detection device 32 is provided at the left and right edges (two places in total) of the eyepiece optical system 6 to perform gaze detection.

Note that Tobe Technology has proposed a spectacle-type gaze detection device that detects gaze by imaging a pupil illuminated by a plurality of infrared light sources with a plurality of cameras. The detection device 32 can have a similar configuration.

The image processing unit 33 is a processing unit that generates an image to be displayed on the display element 5 by image processing, and includes, for example, a specific arithmetic processing circuit such as an ASIC (application specific integrated circuit). In the above image processing, in addition to normal image processing (image processing such as color correction, enlargement / reduction, and edge enhancement), an index (marker) indicating the user's line-of-sight position, which will be described later, is input from another terminal. Image processing for displaying the picked-up images to be displayed on the display element 5 individually or in combination is also included.

The communication unit 34 is an interface for inputting and outputting information between the HMD 201 and another information input / output terminal 200, and includes a transmission circuit, a reception circuit, an antenna, and the like. The storage unit 35 is configured by a non-volatile memory such as a flash memory, for example, and stores various types of information (including captured image information (image data) and information indicating the line-of-sight position) output from other terminals.

The voice input device 36 is a voice information input device including, for example, a microphone. The audio information input to the audio input device 36 is output to another terminal via the communication unit 34. The audio output device 37 is an audio information output device including, for example, a speaker or an earphone. Audio information input from another terminal via the communication unit 34 is output from the audio output device 37.

The plurality of information input / output terminals 200 may include information input / output terminals other than the head-mounted type. For example, the plurality of information input / output terminals 200 are information input / output terminals in which the above-described line-of-sight detection device 32 is combined with a display device installed in a room (for example, a liquid crystal display device or an organic EL (Electro-Luminescence) display device). May be included.

(Examples of using eye-gaze information sharing system)
Next, an application example of the above-described line-of-sight information sharing system 100 will be described. FIG. 5 schematically shows how the worker PA and the instruction supervisor PB perform the collaborative work by wearing the

HMDs

201A and 201B on their heads. The worker PA and the instruction supervisor PB are users of the

HMDs

201A and 201B, respectively. The configurations of the

HMDs

201A and 201B are exactly the same as the HMD 201 described above. Further, in order to distinguish the constituent elements included in the

HMDs

201A and 201B from each other, a symbol “A” or “B” is added after the reference numerals of the constituent elements as in the

video display devices

202A and 202B, for example.

Here, as an example, it is assumed that the device OB, which is an object placed on the table T, is removed for maintenance. Further, the three sides around the table T (back side, left side and right side when viewed from the worker PA) are surrounded by a built-in wall W, and the worker PA stands in front of the table T and works on the device OB. It is assumed that it is difficult for the instruction supervisor PB standing behind the worker PA to directly access and visually recognize the device OB in the situation where the operator PA is performed.

In addition, the

HMDs

201A and 201B are in a communication state (for example, in a state in which communication is possible using Wi-Fi (registered trademark)) so that information can be linked to each other, and can transmit and receive mutual information. At this time, it is assumed that communication between the

HMDs

201A and 201B is performed via each communication unit 34 under the control of each control unit 38.

FIG. 6 is a flowchart showing an example of an operation flow in the line-of-sight information sharing system 100 of the present embodiment. Hereinafter, description will be given with reference to this flowchart.

First, the HMD 201A (first terminal) of the worker PA receives information on the captured image acquired by the imaging device 31 that is the visual line camera of the HMD 201A via the communication unit 34 of the HMD 201A. 2 terminals) (S1; first step).

FIG. 7 shows an image of the field of view of the worker PA. A rectangular area IM (201A) indicated by a broken line indicates an imaging range of the imaging device 31 of the HMD 201A worn by the worker PA. The worker PA can see the device OB, the hand of the worker PA (right hand RH (PA), left hand LH (PA)), the tool held in the hand of the worker PA (driver DR (PA)), The field of view in front of the worker PA is imaged by the imaging device 31 of the HMD 201A. The HMD 201A outputs information (image data) of the captured image of the imaging device 31 viewed from the viewpoint of the worker PA to the HMD 201B.

Subsequently, the HMD 201B causes the video display device 202B to display the captured image so as to be visible to the instruction supervisor PB based on the captured image information input from the HMD 201A (S2; second step).

FIG. 8 shows a state where the instruction supervisor PB observes the video (virtual image) V (202B) displayed by the video display device 202B of the HMD 201B of the instruction supervisor PB. Since the instruction supervisor PB stands behind the worker PA who is actually working toward the device OB as shown in FIG. 5, as described above, the work and device of the actual worker PA It is difficult to observe OB directly. However, the instruction supervisor PB can confirm the actual work of the worker PA by observing the video V (202B) imaged by the HMD 201A, transmitted to the HMD 201B, and displayed.

Next, the line-of-sight direction of the instruction supervisor PB with respect to the captured image (video V (202B)) displayed by the video display device 202B is detected by the line-of-sight detection device 32 of the HMD 201B (S3; third step).

As shown in FIG. 2, the line-of-sight detection device 32 is attached to the video display device 202 (eyepiece optical system 6), and the relative position with respect to the video display device 202 is fixed. Therefore, the line-of-sight detection device 32 of the HMD 201B can detect which part of the video V (202B) displayed on the video display device 202B is being viewed by the instruction supervisor PB.

Next, the HMD 201B outputs the line-of-sight information regarding the line-of-sight direction of the instruction supervisor PB detected by the line-of-sight detection device 32 to the HMD 201A via the communication unit 34 of the HMD 201B (S4; fourth step). Then, the HMD 201A visually recognizes the position of the line of sight of the instruction supervisor PB to the worker PA at a corresponding position in the video display area by the HMD 201A based on the line-of-sight information input from the HMD 201B by the video display device 202A. Display is possible (S5; fifth step).

In the HMD 201A of the worker PA, the relative position of the imaging device 31 with respect to the video display device 202 is fixed, and the captured image (video V (202B)) displayed on the HMD 201B of the instruction supervisor PB is originally the worker. It is the captured image acquired by HMD201A of PA. For this reason, the point of sight of the instruction supervisor PB with respect to the captured image displayed on the HMD 201B (the point ahead of the direction of the line of sight of the instruction supervisor PB) is known relative to the imaging device 31 in the HMD 201A of the worker PA. Corresponding to an arbitrary position in the entire display area of a video (virtual image) of a video display device 202A. Therefore, the HMD 201A of the worker PA immediately displays the position of the line of sight of the instruction supervisor PB at a predetermined position (a position corresponding to the point of sight of the instruction supervisor PB) in the display area of the video display device 202A. It becomes possible.

FIG. 9 shows an image of the field of view of the worker PA when the worker PA is looking at the outside world through the HMD 201A. In the HMD 201A, by displaying the marker MB indicating the position of the line of sight of the instruction supervisor PB, it is possible to present to the operator PA which part of the device OB the instruction supervisor PB is looking at. The worker PA can perform the work of loosening the screw indicated by the marker MB by looking at the displayed marker MB (the position of the line of sight of the instruction supervisor PB). Thus, by displaying the position of the line of sight of the instruction supervisor PB on the HMD 201A, the worker PA and the instruction supervisor PB can communicate based on the line of sight, and perform the work accurately and efficiently. Is possible.

At this time, since the worker PA and the instruction supervisor PB are adjacent to each other (see FIG. 5), for example, the instruction supervisor PB may emit a voice such as “Loosen this screw!”. Thereby, the worker PA can easily recognize that “this screw should be loosened” while looking at the marker MB displayed on the HMD 201A. That is, the instruction supervisor PB can give an accurate instruction to the worker PA by the line of sight and the voice, and the worker PA can accurately perform the work based on this instruction.

As described above, in the present embodiment, a captured image indicating the field of view of the HMD 201A of the worker PA is output to the HMD 201B of the instruction supervisor PB, the captured image is displayed on the HMD 201B, and the instruction supervisor PB for the captured image is displayed. Is detected and the line-of-sight information is output to the HMD 201A. Accordingly, the HMD 201A can immediately display the position of the line of sight of the instruction supervisor PB with the marker MB at a position corresponding to the position of the line of sight of the instruction supervisor PB with respect to the captured image in the display image of the HMD 201A. Therefore, in collaborative work using a plurality of

HMDs

201A and 201B, complicated processing such as grasping the mutual spatial positions of the worker PA and the instruction supervisor PB and conversion of line-of-sight information (conversion of position coordinates) is required. Therefore, communication based on the line of sight can be performed with a simple method.

Further, even in a situation where the instruction supervisor PB cannot directly recognize the device OB, the instruction supervisor PB uses the captured image (displayed video V (202B)) output from the HMD 201A of the worker PA. OB can be grasped. Therefore, by outputting the line-of-sight information of the instruction supervisor PB with respect to the captured image to the HMD 201A of the worker PA and displaying the position of the line of sight of the instruction supervisor PB on the HMD 201A, the worker PA displays the displayed line-of-sight position. Based on the above, it is possible to accurately perform the operation on the device OB. That is, even if the instruction supervisor PB cannot visually recognize the device OB, accurate communication and work based on the line of sight can be performed.

FIG. 10 is a flowchart showing another example of the operation flow in the line-of-sight information sharing system 100 of the present embodiment. The HMD 201A may detect the line-of-sight direction of the worker PA by the line-of-sight detection device 32 of the HMD 201A (S6; sixth step). In the step S5 described above, the position of the line of sight of the worker PA based on the detection result in S6 may be displayed on the HMD 201A together with the position of the line of sight of the instruction supervisor PB.

FIG. 11 shows an image of the field of view of the worker PA when the line-of-sight positions of both the worker PA and the instruction supervisor PB are displayed on the HMD 201A. In FIG. 11, the position of the line of sight of the worker PA is indicated by a marker MA, and the position of the line of sight of the instruction supervisor PB is indicated by a marker MB. However, in order to distinguish the markers MA and MB from each other, here, the shapes of the markers MA and MB are different from each other. More specifically, the marker MA indicating the line of sight of the operator (worker PA) is displayed as a rectangle, and the marker MB indicating the line of sight of the other person (instructor supervisor PB) is displayed as a circle. The display method of the marker MA / MB is not limited to this. For example, the colors may be different from each other, the line types (solid line, broken line, etc.) may be different from each other, “instruction supervisor”, “worker”, etc. The markers MA and MB may be distinguished from each other by displaying the characters in parallel with the markers, or by combining any of shapes, line types, colors, and characters.

In this way, in the HMD 201A of the worker PA, the worker PA displays the position of the worker PA's line of sight (marker MA) together with the position of the line of sight of the instruction supervisor PB (marker MB). At the same time, it is possible to grasp the part of the OB that he / she sees (the part where the work is to be performed) and the part where the instruction supervisor PB is viewing (the part where the work is instructed), and immediately recognizes the match / mismatch. I can grasp it. In the case of a mismatch, the worker PA can take appropriate measures such as confirming by sounding. Therefore, the worker PA can more accurately communicate with the instruction supervisor PB.

FIG. 12 is a flowchart showing still another example of the operation flow in the line-of-sight information sharing system 100 of the present embodiment. The HMD 201A may output the line-of-sight information of the worker PA detected in the step S6 to the HMD 201B of the instruction supervisor PB (S7; seventh step). Then, in the HMD 201B, the position of the line of sight of the worker PA acquired in step S7 and the position of the line of sight of the instruction supervisor PB based on the detection result in step S3 are combined with the captured image output from the HMD 201A. May be displayed (S8; eighth step).

FIG. 13 shows an image of the field of view of the instruction supervisor PB when the line-of-sight positions of both the worker PA and the instruction supervisor PB are displayed on the HMD 201B. In FIG. 13, the marker MB indicating the line of sight of the self (instructor supervisor PB) is displayed as a rectangle, and the marker MB indicating the line of sight of the other person (worker PA) is displayed as a circle. In FIGS. 11 and 13, since the subject viewing the video is different between the worker PA and the instruction supervisor PB, for example, even if the marker MA indicates the line-of-sight position of the same worker PA, the shape thereof is as shown in FIGS. 13 and different. However, FIG. 11 and FIG. 13 are common in that the marker corresponding to the subject viewing the video is displayed as a rectangle and the other's marker is displayed as a circle.

As described above, in the HMD 201B of the instruction supervisor PB, the captured image output from the HMD 201A of the worker PA includes the position of the line of sight of the worker PA (marker MA) and the position of the line of sight of the instruction supervisor PB (marker MB). Are displayed together with the instruction supervisor PB, the part that the device OB is looking at (the place where the work is to be instructed) and the place where the worker PA is looking (the place where the work is to be performed) Can be grasped at the same time, and the coincidence / mismatch can be grasped immediately. In the case of mismatch, the instruction supervisor PB can take appropriate measures such as adding an instruction by emitting a voice. Therefore, the instruction supervisor PB can more accurately communicate with the worker PA. In addition, the instruction supervisor PB can accurately give an instruction based on the line of sight to the worker PA while checking the position where he / she wants to designate the work for the device OB by looking at the marker MB.

Also, as shown in FIG. 11, in the HMD 201A of the worker PA, the position of the line of sight of the worker PA and the position of the line of sight of the instruction supervisor PB are displayed in different patterns (markers MA and MB). Similarly, in the HMD 201B of the instruction supervisor PB, the position of the line of sight of the worker PA and the position of the line of sight of the instruction supervisor PB are displayed in different patterns (markers MA and MB) as shown in FIG. The With such a display, the worker PA and the instruction supervisor PB can easily distinguish and grasp each other's line-of-sight positions in the respective HMDs 210A and 201B, thereby enabling quick communication.

(Gaze information output timing)
The HMD 201 B is instructed by the instruction supervisor PB only when the line of sight of the instruction supervisor PB is located within the display area of the captured image output from the HMD 201 A (one information input / output terminal) to the HMD 201 B (other information input / output terminal). It is desirable to output the line-of-sight information to the HMD 201A of the worker PA.

For example, when the instruction supervisor PB does not see the captured image (video V (202B)) output from the HMD 201A on the HMD 201B, the instruction supervisor PB gives some instruction to the device OB included in the captured image. And no advice. In this case, communication with the line of sight is not required between the instruction supervisor PB and the worker PA. Although the gaze detection of the instruction supervisor PB is always performed by the gaze detection device 32, gaze information (gaze information when looking outside the display area of the photographed image) when communication is unnecessary is obtained from the HMD 201B to the HMD 201A. Even if it outputs to, it will only perform unnecessary communication and is useless.

Therefore, only when the line of sight of the instruction supervisor PB is located within the display area of the captured image on the HMD 201B, that is, the instruction supervisor PB is viewing the captured image on the HDM 201B, and the line of sight with the worker PA Only when the communication based on this is actually necessary, the HMD 201B outputs the line-of-sight information of the instruction supervisor PB to the HMD 201A of the worker PA, thereby reducing the output of useless information to the HMD 201A and reducing the system load. Can do.

Here, as described above, the system in which the HMD 201B outputs the line-of-sight information of the instruction supervisor PB to the HMD 201A only when the line of sight of the instruction supervisor PB is located within the display area of the captured image, specifically, This can be realized by adopting the following configuration.

FIG. 14 is a block diagram showing another configuration of the HMD 201. The HMD 201 further includes an input unit 39 in addition to the configuration shown in FIG. The input unit 39 is a device for designating timing for outputting line-of-sight information of the user of the HMD 201 (for example, the instruction supervisor PB in the above example). Such an input unit 39 can be configured by, for example, a voice recognition device, a pupil observation device, a small touch device, or the like.

The voice recognition device outputs a signal instructing the output of the line-of-sight information to the control unit 38 when recognizing a specific voice such as “this” or “that”, for example. The pupil observation device observes the user's pupil, and when the characteristic blink of the user is detected, for example, three consecutive blinks, a blink larger than normal, or a blink longer than usual, A signal for instructing output is output to the control unit 38. Note that the line-of-sight detection device 32 may have the function of the above-described pupil observation device. The small touch device has, for example, a small ring-shaped member that can be attached to the finger of the user's hand, and outputs a signal instructing the output of the line-of-sight information to the control unit 38 when the ring-shaped member is touched. The control unit 38 receives the instruction signal from the input unit 39 and controls the communication unit 34 to output the user's line-of-sight information detected by the line-of-sight detection device 32 from the communication unit 34 to the outside.

In this way, by using the input unit 39, the HMD 201B can cause the HMD 201A to output the line-of-sight information at the detection timing (at the timing specified by the input unit 39) triggered by detection of a specific sound or the like. it can. That is, when the instruction supervisor PB is observing a captured image, the instruction supervisor PB makes a sound such as “this”, makes a specific blink, or touches a ring-shaped member, thereby changing the HMD 201B to the HMD 201A. Gaze information can be output. Therefore, as described above, it is possible to reliably realize the configuration in which the line-of-sight information of the instruction supervisor PB is output to the HMD 201A only when the line of sight of the instruction supervisor PB is located within the captured image display area. Further, the line-of-sight information output from the HMD 201B can be surely used as the line-of-sight information when the instruction supervisor PB is looking at the display area of the captured image. Communication can be realized. Further, by using the input unit 39 together, intended designation for the device OB becomes simple, and further improvement in usability can be expected.

In the small touch device which is an example of the input unit 39, the ring-shaped member is attached to, for example, the little finger of the user's hand and can be touched with the thumb of the same hand. For this reason, the user's hands-free performance is not impaired only by wearing the small touch device as the input unit 39 by the user. That is, even if a small touch device is used as the input unit 39, the hands-free property peculiar to the HMD 201 is ensured.

In this embodiment, the HMD 201B of the instruction supervisor PB is configured using the optical see-through video display device 202, but the instruction supervisor PB stands behind the worker PA and directly sees the device OB. It is not necessary to be optically see-through because it is difficult to do. Therefore, the HMD 201B may be configured by using a video see-through type (see FIGS. 30 and 31) video display device 202 described later.

In the above, it is assumed that the worker PA and the instruction supervisor PB are located at a short distance as shown in FIG. 5, but the worker PA and the instruction supervisor PB are far away as shown in FIG. It may be located at a distance. For example, even when the worker PA performs work at a location several hundred kilometers away from the instruction supervisor PB, the worker PA communicates an instruction based on the line-of-sight information from the instruction supervisor PB by applying the system of this embodiment. By receiving via the line 300 (see FIG. 1), the same effect as in the present embodiment can be obtained.

At this time, the voice input device 36 and the voice output device 37 shown in FIG. 4 are used between the

HMDs

201A and 201B, such as “this”, “that”, “loosen this screw!”, “I understand!” Audio information may be input / output. In this case, since communication using voice information can be performed in addition to the line-of-sight information of the instruction supervisor PB, communication is further simplified. In addition, since the content of the line-of-sight information can be supplemented with audio information, the accuracy of communication is further improved.

In the present embodiment, the case where the output of the captured image is unidirectional from the HMD 201A to the HMD 201B has been described, but the same effect can be obtained by applying the configuration of the present embodiment even when the output is bidirectional. . Hereinafter, a specific example thereof will be described as a second embodiment.

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to the drawings. FIG. 16 schematically shows a state where the worker PA and the worker PC perform the collaborative work by wearing the

HMDs

201A and 201C on their heads, respectively. Worker PA and worker PC are users of

HMDs

201A and 201C, respectively. The configurations of the

HMDs

201A and 201C are exactly the same as those of the HMD 201 (see FIGS. 2 to 4) described in the first embodiment. Further, in order to distinguish the constituent elements included in the

HMDs

201A and 201C from each other, a symbol “A” or “C” is added after the reference numerals of the constituent elements as in the

video display devices

202A and 202C, for example.

Here, as an example, a device OB that is an object is placed on a table T in the center of the work room, and a worker PA and a worker PC, which are a plurality of workers, are placed on the device OB. Assume that maintenance is performed.

In addition, the

HMDs

201A and 201C are in a communication state (for example, in a state where communication is possible using Wi-Fi (registered trademark)) so that information can be linked to each other, and the mutual information can be transmitted and received. At this time, communication between the HMDs 201 A and 201 C is assumed to be performed via each communication unit 34 under the control of each control unit 38.

In this embodiment, the steps S1 to S5 shown in FIG. 6 are performed between the HMD 201A of the worker PA and the HMD 201C of the worker PC. More specific description will be given below.

(Output of captured image information)
The HMD 201A, which is one information input / output terminal, outputs information of the captured image acquired by the imaging device 31 of the HMD 201A to the HMD 201C, which is another information input / output terminal. Similarly, the HMD 201C outputs information of the captured image acquired by the imaging device 31 of the HMD 201C to the HMD 201A.

FIG. 17 shows an image of the field of view of the worker PA. A rectangular area IM (201A) indicated by a broken line indicates an imaging range of the imaging device 31 of the HMD 201A worn by the worker PA. The worker PA includes the device OB, the hand of the worker PA (right hand RH (PA), the left hand LH (PA)), the tool held in the hand of the worker PA (driver DR (PA)), and the worker PC. The hands (right hand RH (PC), left hand LH (PC)) are visible, and the field of view of the worker PA is captured by the imaging device 31 of the HMD 201A. The HMD 201A outputs information (image data) of the captured image of the imaging device 31 viewed from the viewpoint of the worker PA to the HMD 201C via the communication unit 34 of the HMD 201A.

On the other hand, FIG. 18 shows an image of the field of view of the worker PC. A rectangular area IM (201C) indicated by a broken line indicates an imaging range of the imaging device 31 of the HMD 201C worn by the worker PC. The worker PC includes the device OB, the hand of the worker PA (right hand RH (PA), the left hand LH (PA)), the tool held in the hand of the worker PA (driver DR (PA)), the worker PC The hands (right hand RH (PC), left hand LH (PC)) are visible, and the field of view of the worker PC is captured by the imaging device 31 of the HMD 201C. The HMD 201C outputs information (image data) of the captured image of the imaging device 31 viewed from the viewpoint of the worker PC to the HMD 201A via the communication unit 34 of the HMD 201C.

(Display of captured image)
The HMD 201C displays the captured image on the worker PC of the HMD 201C so as to be visible based on the captured image information input from the HMD 201A by the video display device 202C. Similarly, the HMD 201A causes the video display device 202A to display the captured image so as to be visible to the worker PA of the HMD 201A based on information about the captured image input from the HMD 201C.

FIG. 19 shows an operation when an image (view image) captured and output by the HMD 201C of the worker PC is displayed as a video (virtual image) V (202A) on the video display device 202A of the HMD 201A of the worker PA. The view of the person PA is shown. FIG. 20 shows a case where an image (view image) captured and output by the HMD 201A of the worker PA is displayed as a video (virtual image) V (202C) on the video display device 202C of the HMD 201C of the worker PC. The field of view of the worker PC is shown. As described above, the

HMDs

201A and 201C mutually input and output view images captured by the respective image capturing apparatuses 31, and display the input view images as videos (virtual images). As a result, the worker PA can observe the field-of-view image of the worker PC with the HMD 201A worn by the worker PA, and the worker PC can see the worker PC with the HMD 201C worn by the worker PA. A visual field image of PA can be observed.

In particular, as shown in FIG. 19, the device OB, the hand of the worker PA, the driver DR (PA), and the hand of the worker PC are real objects in the actual view of the worker PA. As shown in FIG. 20, the device OB, the hand of the worker PA, the driver DR (PA), and the hand of the worker PC are also real objects in the actual field of view of the worker PC. By displaying such visual field images by inputting / outputting them between the

HMDs

201A and 201C, the worker PA and the worker PC share their respective visual fields (visual fields in the line of sight) with each other, and A real object inside can be visually recognized.

Note that the video V (202A) displayed on the HMD 201A is a part of a region that can be displayed as a virtual image by the video display device 202A and is displayed in a region other than the center of the region (in the display element 5, the display element 5). The video is displayed on a part of the display surface other than the center of the display surface). When the worker PA works on the device OB, it is considered that the worker PA usually works by positioning the device OB in the center of the field of view. Therefore, it is possible to avoid that the video V (202A) and the device OB are visually overlapped and visually recognized during the work of the worker PA, and the workability with respect to the device OB is deteriorated (the work is difficult). From the same point of view, the video V (202C) displayed on the HMD 201C is also a part of an area that can be displayed as a virtual image by the video display device 202C and is displayed in a region other than the center of the area.

(Detection of gaze direction for captured image)
The HMD 202C detects the line-of-sight direction of the worker PC with respect to the captured image (video V (202C)) displayed by the video display device 202C by the line-of-sight detection device 32 of the HMD 202C. Similarly, the HMD 202A detects the line-of-sight direction of the worker PA with respect to the captured image (video V (202A)) displayed by the video display device 202A by the line-of-sight detection device 32 of the HMD 202A.

As shown in FIG. 2, the line-of-sight detection device 32 is attached to the video display device 202 (eyepiece optical system 6), and the relative position with respect to the video display device 202 is fixed. Therefore, the line-of-sight detection device 32 of the HMD 201C can detect which part of the video V (202C) displayed on the video display device 202C is being viewed by the worker PC. Similarly, the line-of-sight detection device 32 of the HMD 201A can detect which part of the video V (202A) displayed on the video display device 202A is being viewed by the worker PA.

(Gaze information output, gaze position display)
The HMD 201C outputs line-of-sight information regarding the line-of-sight direction of the detected worker PC to the HMD 201A via the communication unit 34 of the HMD 201C. Then, the HMD 201A displays the position of the line of sight of the worker PC on the worker PA so as to be visible to the worker PA based on the line-of-sight information input from the HMD 201C by the video display device 202A. Similarly, the HMD 201A outputs the detected line-of-sight information regarding the line-of-sight direction of the worker PA to the HMD 201C via the communication unit 34 of the HMD 201A. Then, the HMD 201C displays the position of the line of sight of the worker PA on the worker PC so as to be visible based on the line-of-sight information input from the HMD 201A by the video display device 202C.

In the HMD 201A of the worker PA, the relative position of the imaging device 31 with respect to the video display device 202 is fixed, and the captured image (video V (202C)) displayed on the HMD 201C of the worker PC is originally the worker PA. It is the picked-up image acquired by HMD201A. For this reason, the gaze point of the worker PC with respect to the captured image of the HMD 201A displayed on the HMD 201C (the point ahead of the worker PC's line-of-sight direction) has a known relative position with the imaging device 31 in the HMD 201A of the worker PA. Corresponding to an arbitrary position in the entire video display area of the video display device 202A (substantially equal to the field of view of the worker PA). Accordingly, the HMD 201A of the worker PA can immediately display the position of the line of sight of the worker PC at a predetermined position (a position corresponding to the gazing point of the worker PC) in the display area of the video display device 202A. It becomes possible.

Similarly, in the HMD 201 C of the worker PC, the relative position of the imaging device 31 with respect to the video display device 202 is fixed, and the captured image (video V (202 A)) displayed on the HMD 201 A of the worker PA is originally It is a picked-up image acquired by HMD201C of worker PC. For this reason, the point of attention of the worker PA (the point ahead of the line of sight of the worker PA) with respect to the captured image displayed on the HMD 201A is a video whose relative position is known with the imaging device 31 in the HMD 201C of the worker PC. Corresponding to an arbitrary position in the entire display area of the display device 202C (substantially equal to the field of view of the worker PC). Therefore, the HMD 201C of the worker PC can immediately display the position of the line of sight of the worker PA at a predetermined position (a position corresponding to the gazing point of the worker PA) in the display area of the video display device 202C. It becomes possible.

FIG. 21 shows an image of the field of view of the worker PA when the worker PA is looking at the outside world via the HMD 201A. The HMD 201A can present to the worker PA which part of the device OB the worker PC is looking at by displaying the marker MC indicating the position of the line of sight of the worker PC. FIG. 22 shows an image of the field of view of the worker PC when the worker PC is looking at the outside world through the HMD 201C. The HMD 201C can present to the worker PC which part of the device OB the worker PA is viewing by displaying the marker MA indicating the position of the line of sight of the worker PA.

At this time, since the worker PA and the worker PC are adjacent to each other (see FIG. 16), for example, when the worker PC is holding the hand of the worker PC, for example, the worker PC supports the housing of the device OB with both hands. In addition, a voice such as “Loosen this screw!” May be emitted while gazing at the intended device OB. Thereby, the worker PA can easily recognize that “this screw should be loosened” while looking at the marker MC (see FIG. 21) displayed on the HMD 201A, and can accurately perform the work. . Further, the worker PC can accurately transmit the instruction to the worker PA by using the line of sight and the voice without damaging the manual work. Hereinafter, a specific example of the collaborative work of the worker PA and the worker PC using the line of sight and voice will be described.

21 and 22, the numbers (1) to (4) given in the vicinity of the markers MA and MC indicate the display order of the markers. In addition, here, in order to distinguish the markers MA and MC from each other, the marker indicating the own line-of-sight position displayed on the own machine is represented by a rectangle, and the marker indicating the line-of-sight position of another worker displayed on the own machine is displayed. Represented by a circle.

First, when the captured image output from the HMD 201A of the worker PA is displayed as the video V (202C) on the video display device 202C of the HMD 201C, the worker PC is to be loosened in the video V (202C). While listening to the screw, give a voice instruction "Loosen this screw!" The visual line direction of the worker PC with respect to the video V (202C) is detected by the visual line detection device 32 of the HMD 201C. Then, the video display device 202C displays a marker MC indicating the line of sight of the worker PC along with the captured image of the HMD 201A in the video V (202C) (see (1) in FIG. 22). This clearly identifies which screw should be loosened.

Next, the line-of-sight information indicating the line-of-sight position of the worker PC detected by the HMD 201C and the captured image information acquired by the HMD 201C are output to the HMD 201A. Then, in the entire video display area of the video display device 202A of the HMD 201A (corresponding to the field of view of the worker PA), the worker PC has a position corresponding to the gaze point of the worker PC with respect to the video V (202C). A marker MC indicating the line-of-sight position is displayed (see (2) in FIG. 21). The captured image acquired by the HMD 201C is displayed as the video V (202A) by the video display device 202A of the HMD 201A.

The worker PA grasps the screw (screw to be loosened) instructed by the worker PC by visually recognizing the marker MC displayed on his / her HMD 201A, and performs the work of loosening the screw. At this time, for example, when it is desired to request assistance from the worker PC, for example, when the worker wants to hold the housing cover to which the screw to be loosened is held for safety, the worker PA displays the video V In (202A), while listening to the part you want to hold, make a voice request “Please support this cover!”. The visual line direction of the worker PA with respect to the video V (202A) is detected by the visual line detection device 32 of the HMD 201A. Then, the video display device 202A displays the marker MA indicating the line-of-sight position of the worker PA together with the captured image of the HMD 201C as the video V (202A) (see (3) in FIG. 21). Thereby, it is clearly identified which cover the operator PA wants to support.

Then, line-of-sight information indicating the line-of-sight position of the worker PA detected by the HMD 201A is output to the HMD 201C. Then, in the entire video display area of the video display device 202C of the HMD 201C (corresponding to the visual field range of the worker PC), the worker PA's position at the position corresponding to the gaze point of the worker PA with respect to the video V (202A). A marker MA indicating the line-of-sight position is displayed (see (4) in FIG. 22). As a result, the worker PC can immediately grasp the cover to be supported by looking at the marker MA, and can accurately assist the worker PA in the work.

As described above, the steps of outputting the information of the captured image, displaying the captured image, detecting the line-of-sight direction with respect to the captured image, outputting the line-of-sight information, and displaying the position of the line of sight are performed between the

HMDs

201A and 201C. As a result, the

HMDs

201A and 201C can share the line-of-sight information of the worker PA and the worker PC with each other and perform a collaborative work. In addition, it is possible to communicate smoothly without compromising manual work by visually indicating the voice instructions with ambiguous locations and objects such as “this” and “that” using the marker MA / MC indicating the line-of-sight position. Thus, it is possible to perform efficient collaborative work. In particular, by sharing the line-of-sight information among the plurality of

HMDs

201A and 201C, a collaborative work utilizing the hands-free property peculiar to the HMD becomes possible. Further, by using voice information in addition to line-of-sight information, more accurate communication is possible.

[Embodiment 3]
The following will describe still another embodiment of the present invention with reference to the drawings. In the above-described second embodiment, the example in which the worker PA and the worker PC work together is described, but in the present embodiment, a case where the work is performed by three people is described. Here, in the system of the second embodiment, an example will be described in which an instruction supervisor PD located at a distant place is added to the worker PA and the worker PC to perform work (including work instructions and support). .

In FIG. 23, the worker PA and the worker PC wear the HMD 201A / 201C on the head, the distant instruction supervisor PD wears the HMD 201D, and the images sent from the HMD 201A / 201C to the HMD 201D are displayed. A state in which the instruction supervisor PD is observing is schematically shown. The worker PA and the worker PC are users of the

HMDs

201A and 201C, respectively, and the instruction supervisor PD is a user of the HMD 201D. The HMD 201D is an information input / output terminal that can communicate with the HMD 201A and the HMD 201C via the communication line 300 (see FIG. 1). The configurations of the

HMDs

201A, 201C, and 201D are exactly the same as those of the HMD 201 described in Embodiment 1 (see FIGS. 2 to 4). In the following description, when the components included in the

HMDs

201A, 201C, and 201D are distinguished from each other, for example, as in the

video display devices

202A, 202C, and 202D, "A" and "C" "Or" D ".

Input / output of information between the HMD 201A and the HMD 201C is the same as that in the second embodiment. Hereinafter, input / output of information between the HMD 201A and the HMD 201D and between the HMD 201C and the HMD 201D will be described.

First, the information of the captured image acquired by the

HMDs

201A and 201C is output to the HMD 201D via each communication unit 34. In the HMD 201D, the captured images of the

HMDs

201A and 201C are displayed as video V (202D) -A and video V (202D) -C.

When the instruction supervisor PD is observing the video V (202D) -A on the HMD 201D, the visual line direction of the instruction supervisor PD at that time is detected by the visual line detection device 32 of the HMD 201D. The line-of-sight information regarding the detected line-of-sight direction is output from the HMD 201D to the HMD 201A via the communication unit 34. Thereafter, in the HMD 201A, the position of the line of sight of the instruction supervisor PD is displayed at a predetermined position in the entire display image of the video display device 202A based on the line-of-sight information.

On the other hand, when the instruction supervisor PD is observing the video V (202D) -C, the line-of-sight direction of the instruction supervisor PD at that time is detected by the line-of-sight detection device 32 of the HMD 201D. Then, the line-of-sight information regarding the detected line-of-sight direction is output from the HMD 201D to the HMD 201C via the communication unit 34. Thereafter, in the HMD 201C, the position of the line of sight of the instruction supervisor PD is displayed at a predetermined position in the entire display image of the video display device 202C based on the line-of-sight information.

FIG. 24 shows an image of the field of view of the worker PA when the worker PA is looking at the outside world through the HMD 201A. In the HMD 201A, in addition to the marker MC indicating the position of the line of sight of the worker PC, the marker PD indicating the position of the line of sight of the instruction supervisor PD is displayed so that the worker PC and the instruction supervisor PD can identify which part of the device OB. Can be shown to the worker PA. FIG. 25 shows an image of the field of view of the worker PC when the worker PC is looking at the outside world via the HMD 201C. In the HMD 201C, in addition to the marker MA indicating the position of the line of sight of the worker PA, the marker PD indicating the position of the line of sight of the instruction supervisor PD is displayed so that the worker PA and the instruction supervisor PD can identify which part of the device OB. Can be shown to the worker PC.

In this manner, the HMD 201A / 201C can display the positions of the three eyes of the worker PA, the worker PC, and the instruction supervisor PD so that the three parties can communicate and perform the work. It becomes. Hereinafter, specific examples of work by the worker PA, the worker PC, and the instruction supervisor PD will be described.

In FIG. 23 to FIG. 25, the numbers (1) to (10) in the vicinity of the markers MA, MC, and MD indicate the display order of the markers. In addition, here, in order to distinguish the markers MA, MC, and MD from each other, the marker indicating the position of the user's line of sight displayed on the own machine is represented by a rectangle, and the line of sight of another worker displayed on the own machine is indicated. Markers are represented by circles. In FIGS. 24 and 25, a marker indicating the line-of-sight position of the instruction supervisor PD is represented by a triangle.

First, the information of the captured image acquired by the HMD 201A of the worker PA is output to the HMD 201C / HMD 201D, and the captured image is displayed as the video V (202C) / V (202D) -A by the HMD 201C / HMD 201D, respectively. (See FIG. 25 and FIG. 23).

The worker PC gives a voice instruction “Loosen this screw!” While watching the screw to be loosened in the video V (202C) displayed on the HMD 201C. The visual line direction of the worker PC with respect to the video V (202C) is detected by the visual line detection device 32 of the HMD 201C. Then, the video display device 202C displays a marker MC indicating the line of sight of the worker PC in the video V (202C) together with the captured image of the HMD 201A (see (1) in FIG. 25).

Next, the line-of-sight information indicating the line-of-sight position of the worker PC detected by the HMD 201C and the captured image information acquired by the HMD 201C are output to the HMD 201A. Then, in the entire video display area of the video display device 202A of the HMD 201A (corresponding to the field of view of the worker PA), the worker PC has a position corresponding to the gaze point of the worker PC with respect to the video V (202C). A marker MC indicating the line-of-sight position is displayed (see (2) in FIG. 24). The captured image acquired by the HMD 201C is displayed as the video V (202A) by the video display device 202A of the HMD 201A (see FIG. 24).

Further, the line-of-sight information indicating the line-of-sight position of the worker PC detected by the HMD 201C is also output to the HMD 201D. Accordingly, in the HMD 201D, the marker MC indicating the line-of-sight position of the worker PC is displayed in the video V (202D) -A by the video display device 201D (see (3) in FIG. 23). Note that the video V (202C) and the video V (202D) -A are both images taken by the HMD 201A, and therefore, the operator PC notes the video V (202C) in the video V (202D) -A. It is possible to easily display the marker MC indicating the line-of-sight position of the worker PC at a position corresponding to the viewpoint.

When the instruction supervisor PD looks at the marker MC in the video V (202D) -A and determines that the instruction of the worker PC is wrong, the screw to be loosened in the video V (202D) -A , Give a voice instruction saying “No, you should loosen this screw first!” The line-of-sight direction of the instruction supervisor PD with respect to the video V (202D) -A is detected by the line-of-sight detection device 32 of the HMD 201D. The video display device 202D displays a marker MD indicating the line-of-sight position of the instruction supervisor PD in the video V (202D) -A (see (4) in FIG. 23).

The line-of-sight information regarding the line-of-sight direction of the instruction supervisor PD is output to the HMD 201A. Accordingly, in the HMD 201A, the entire video display area of the video display device 202A (corresponding to the field of view of the worker PA) is located at a position corresponding to the point of sight of the instruction supervisor PD with respect to the video V (202D) -A. A marker MD indicating the line-of-sight position of the instruction supervisor PD is displayed (see (5) in FIG. 24). Whether the instruction supervisor PD is viewing the video V (202D) -A or the video V (202D) -C can be determined based on the visual line detection result in the visual line detection device 32 of the HMD 201D. Therefore, when the instruction supervisor PD is watching the video V (202D) -A, the line-of-sight information of the instruction supervisor PD can be output to the HMD 201A that is the provider of the video V (202D) -A. .

The worker PA performs the work of loosening the screw indicated by the instruction supervisor PD, not the screw indicated by the worker PC, by visually recognizing the marker MD displayed on the own HMD 201A. At this time, the worker PA makes a voice request “Please support this cover!” While observing the portion to be held in the video V (202A). The visual line direction of the worker PA with respect to the video V (202A) is detected by the visual line detection device 32 of the HMD 201A. Then, the video display device 202A displays the marker MA indicating the line-of-sight position of the worker PA together with the captured image of the HMD 201C as the video V (202A) (see (6) in FIG. 24).

Then, line-of-sight information indicating the line-of-sight position of the worker PA detected by the HMD 201A is output to the HMD 201C. Then, in the entire video display area of the video display device 202C of the HMD 201C (corresponding to the visual field range of the worker PC), the worker PA's position at the position corresponding to the gaze point of the worker PA with respect to the video V (202A). A marker MA indicating the line-of-sight position is displayed (see (7) in FIG. 25).

Further, the line-of-sight information indicating the line-of-sight position of the worker PA detected by the HMD 201A is also output to the HMD 201D. Accordingly, in the HMD 201D, the marker MA indicating the line-of-sight position of the worker PA is displayed in the video V (202D) -C by the video display device 201D (see (8) in FIG. 23). Note that the video V (202A) and the video V (202D) -C are both images taken by the HMD 201C, and therefore the operator PA's note on the video V (202A) in the video V (202D) -C. Displaying the marker MA indicating the line-of-sight position of the worker PA at a position corresponding to the viewpoint can be easily realized.

The instruction supervisor PD looks at the marker MA in the video V (202D) -C, and determines that the instruction of the worker PA is wrong, the cover to be held in the video V (202D) -C. While watching, give a voice instruction "No, hold this cover as well as its cover!" The line-of-sight direction of the instruction supervisor PD with respect to the video V (202D) -C is detected by the line-of-sight detection device 32 of the HMD 201D. The video display device 202D displays the marker MD indicating the line-of-sight position of the instruction supervisor PD in the video V (202D) -C (see (9) in FIG. 23).

The line-of-sight information regarding the line-of-sight direction of the instruction supervisor PD is output to the HMD 201C. Accordingly, in the HMD 201C, the entire video display area of the video display device 202C (corresponding to the field of view of the worker PC) is positioned at the position corresponding to the point of sight of the instruction supervisor PD with respect to the video V (202D) -C. A marker MD indicating the line-of-sight position of the instruction supervisor PD is displayed (see (10) in FIG. 25). As a result, the worker PC can appropriately support the work of the worker PA holding the cover designated by the marker MD and loosening the screw. As described above, whether the instruction supervisor PD is viewing the video V (202D) -A or the video V (202D) -C depends on the visual line detection result in the visual line detection device 32 of the HMD 201D. Therefore, when the instruction supervisor PD is watching the video V (202D) -C, the line-of-sight information of the instruction supervisor PD is output to the HMD 201C that is the provider of the video V (202D) -C. be able to.

As described above, by sharing the line-of-sight information of the worker PA, the worker PC, and the instruction supervisor PD among the three information input / output terminals of the

HMDs

201A, 201C, and 201D, without damaging the manual work, Communication is good and collaborative work can be done efficiently. In particular, by further utilizing voice information in addition to line-of-sight information, more accurate communication is possible, and more efficient work can be performed based on the communication.

By the way, although this embodiment demonstrated the example using HMD201D which can be mounted | worn to a head as the information input / output terminal 200 which instruction supervisor PD uses, instruction supervisor PD does not require special hands-free and mobility. In this case, the information input / output terminal 200 may have another form.

FIG. 26 is an explanatory diagram showing another configuration of the information input / output terminal 200. As shown in the figure, the information input / output terminal 200 may include at least a display device 200a and a line-of-sight detection device 32 fixed to the display device 200a. The display device 200a is a direct view display (monitor) such as a television set installed in a room. The line-of-sight detection device 32 detects the line-of-sight direction of the instruction supervisor PD who observes the video V (200) -A and the video V (200) -C displayed on the display device 200a. The position of the line-of-sight detection device 32 is identified with respect to the display device 200a. The line-of-sight detection device 32 having such a configuration has been proposed by Toby Technology, Inc. and can be used. Note that the video V (200) -A is a captured image of the HMD 201A acquired by the HMD 201A of the worker PA, input to the information input / output terminal 200, and displayed. The video V (200) -C is a captured image of the HMD 201C that is acquired by the HMD 201C of the worker PC, input to the information input / output terminal 200, and displayed.

When the instruction supervisor PD only gives instructions to the worker PA and the worker PC as in the present embodiment (when the instruction supervisor PD does not actually perform manual work on the device OB), the above configuration The information input / output device 200 displays the other person's line-of-sight image, detects the observation line of sight of the instruction supervisor PD with respect to the display image, and outputs the line-of-sight information to the other person. Similarly, communication based on line of sight is possible. Therefore, the information input / output device 200 of the instruction supervisor PD is not necessarily an HMD.

[Embodiment 4]
The following will describe still another embodiment of the present invention with reference to the drawings. In the present embodiment, a configuration incidental to or replaceable with each of the first to third embodiments described above will be described.

(About parallax correction)
The video display device 202 of the HMD 201 described in each of the above embodiments is an optical see-through display that allows a user to directly observe the outside world together with video. In this case, a user (for example, worker PA) can perform work while directly observing an object (for example, device OB) in the outside world, and workability can be improved.

Here, FIG. 27 schematically shows the relative positional relationship between the video display device 202 (optical see-through display), the imaging device 31, and the object OB ′ located at infinity. As described in the first embodiment, the imaging optical axis A ₁ of the imaging device 31 is substantially the same as the viewing direction (observation central axis) A ₂ of the user when observing the object OB ′ located at infinity. I am doing (almost parallel). In such an optical design, if the vertical direction (vertical direction) distance D between the video display device 202 and the imaging device 31 is too large relative to the distance L between the video display device 202 and the object OB ′. , Parallax occurs. As described above, for example, in the HMD 201A, the video display device 202A displays the marker MB indicating the line-of-sight position of the instruction supervisor PB on the device OB that is the object. The display of the MB deviates from the object (device OB) that the worker PA is actually looking at. That is, in the relative positional relationship shown in FIG. 27, the display position of the marker MB is shifted as the object OB ′ approaches the video display device 202.

Therefore, in order to correct such parallax, as shown in FIG. 28, the HMD 201 desirably includes a measurement unit 41 and a parallax correction unit 42 in addition to the above-described configuration. The measurement unit 41 measures the distance between the video display device 202 and the object OB ′ that the user observes with see-through. For example, in the first embodiment, the captured image displayed by the HMD 201B is an image captured by the imaging device 31 of the HMD 201A, and the object OB ′ is identified by the line-of-sight detection result of the HMD 201B with respect to the captured image. It is easy to measure the distance L from the object OB ′ by the imaging device 31 of the HMD 201A (for example, the distance L can be measured by adopting a contrast method in which the imaging device 31 itself performs distance measurement).

The parallax correction unit 42 is based on a relationship between a preset relative position of the imaging device 31 with respect to the video display device 201A (corresponding to the distance D in FIG. 27) and the distance L measured by the measurement unit 41. The parallax (angle θ) between the viewing direction (observation central axis) A ₂ of the user observing the object OB ′ and the imaging optical axis A ₁ of the imaging device 31 that images the object OB ′ is calculated and calculated. The position of the line of sight to be displayed is corrected based on the parallax. Such a parallax correction unit 42 can be configured by a CPU or a circuit for performing a specific calculation process.

For example, in FIG. 27, the distance D is known in advance by design. When the measurement unit 41 measures the distance L (measurement step), the parallax correction unit 42 calculates an angle θ (°) corresponding to the parallax based on the relationship between the distance D and the distance L (calculation step). That is, since tan θ = (D / L) according to the geometrical relationship shown in FIG. 27, the parallax correction unit 42 can calculate the angle θ from this equation.

The parallax correction unit 42 corrects the position of the line of sight to be displayed based on the calculated parallax (correction step). For example, assuming that the total angle of view in the vertical direction of the display area of the video (virtual image) of the video display device 202 is 10 °, the angle θ corresponding to the parallax obtained above is 1 °, and the number of display pixels in the vertical direction of the video If the display element 5 (corresponding to the number of pixels in the vertical direction of the display element 5 (see FIGS. 3 and 5)) is N1 (number), the display position of the marker is normalized by the number of pixels of (1/10) × N1. (For example, a position corresponding to the point of gaze of the instruction supervisor PB in the entire video display area of the video display device 202A) may be shifted in the vertical direction during video observation. In addition, when the imaging device 31 is positioned so as to be shifted in the horizontal direction (left-right direction) with respect to the video display device 202, the marker display position is set by the predetermined number of pixels calculated by the same method as described above. What is necessary is just to shift to the horizontal direction at the time of video observation from a regular position.

Thus, even when parallax occurs depending on the distance between the video display device 202 and the object OB ′, the parallax is calculated and the position of the line of sight to be displayed is corrected. Can be displayed. In other words, it is possible to avoid a situation where the line of sight is displayed with a deviation from a predetermined position (a position where the work is instructed) of the object OB ', and it is possible to reliably realize appropriate communication with the line of sight.

(Other configuration examples of HMD)
FIG. 29 is a cross-sectional view showing another optical configuration of the HMD 201 of each embodiment described above. As shown in the figure, a half mirror 24 is provided on the surface of the HOE 23 on the deflection prism 22 side, and imaging is performed so that the imaging optical axis A ₁ is bent by the half mirror 24 and is coaxial with the line-of-sight direction (observation central axis) A _2. The device 31 may be arranged. In this configuration, since no parallax is generated, a parallax correction mechanism such as the measurement unit 41 and the parallax correction unit 42 described above is unnecessary, and the configuration of the apparatus can be simplified by eliminating the need for such a mechanism. Can do.

FIG. 30 is a cross-sectional view showing still another optical configuration of the HMD 201. As shown in the figure, the video display device 202 of the HMD 201 may be a video see-through display. In other words, in the HMD 201 described in each of the above embodiments, the optical see-through display constituting the video display device 202 may be replaced with a video see-through display. The video see-through display is a display device that uses a shielded display and displays an external image (captured image) captured by the imaging device 31 so as to be visible to the user integrally with the video. The above-described shielding type display can be realized by arranging the shielding plate 25 on the outside side of the eyepiece optical system 6 described above.

As described above, even when the video display device 202 is configured with a video see-through display, communication can be achieved through line-of-sight display. In addition, the operator can work on the object in the outside world while observing the captured image.

FIG. 31 is a cross-sectional view showing still another optical configuration of the HMD 201. As shown in the figure, in the case where the video display device 202 is configured by a video see-through display, the outside of the shielding plate 25 (on the outside side) (so that the imaging optical axis A ₁ is coaxial with the viewing direction (observation central axis) A _2. The imaging device 31 may be disposed on a side opposite to the eyepiece optical system 6 with respect to the shielding plate 25. In this configuration, since no parallax is generated, a parallax correction mechanism such as the measurement unit 41 and the parallax correction unit 42 described above is unnecessary, and the configuration of the apparatus can be simplified by eliminating the need for such a mechanism. Can do. Further, in the video see-through display, since the shielding plate 25 is positioned in front of the user, the imaging device 31 is visually recognized by the user even if the imaging device 31 is arranged further outside the shielding plate 25. This does not interfere with the user's observation of the outside world (video). Therefore, it is possible to easily realize the configuration in which the imaging device 31 is arranged on the outside of the shielding plate 25 as described above so that the imaging optical axis A ₁ and the line-of-sight direction A ₂ are coaxial.

(Other)
Of course, it is possible to appropriately combine the configurations and methods of the embodiments described above. For example, in the system described in the second and third embodiments, the line-of-sight information may be output from the HMD at an arbitrary timing by applying the input unit 39 described in the first embodiment.

In the above description, an example in which a maximum of three people work together has been described. However, even when four or more people work, the system, configuration, and method described in each embodiment are applied to each other so that line-of-sight information can be obtained. Of course, it is possible to share and work together.

The information input / output terminal according to the present embodiment includes a plurality of GPS (Global Positioning System) sensors, geomagnetic sensors, acceleration sensors, temperature sensors, and the like from the viewpoint of improving workability by grasping the worker's work state and work environment. The sensor may be provided. For example, the GPS sensor acquires worker position information. The geomagnetic sensor detects the direction in which the worker is facing. The acceleration sensor detects the movement (posture) of the worker. The temperature sensor detects the temperature of the work environment or the body temperature of the worker himself. By transmitting the information acquired by these sensors to an external input / output terminal via the communication unit 34, the work instruction supervisor understands the work environment outside and appropriate instructions (if necessary) ( For example, it is possible to give the operator (HMD) a work stoppage or the like.

The embodiment of the present invention has been described above, but the scope of the present invention is not limited to this, and various modifications can be made without departing from the spirit of the invention.

Note that the line-of-sight information sharing method and line-of-sight information sharing system described above can be expressed as follows, thereby producing the following operational effects.

The line-of-sight information sharing method described in the embodiment of the present invention is a line-of-sight information sharing method for sharing user's line-of-sight information among a plurality of information input / output terminals connected via a communication line. Each of the information input / output terminals includes a display device that presents an image to a user, a line-of-sight detection device that detects a user's line-of-sight direction, and a communication unit that inputs and outputs information to and from each other, At least one of the information input / output terminals is a head-mounted terminal that is mounted on a user's head and displays the video as a virtual image so as to be visible to the user. In addition to the device, the line-of-sight detection device, and the communication unit, the display device further includes an imaging device that is fixed in a relative position with respect to the display device and that images the external world in front of the user. Any one of the head When the type terminal is the first terminal and the other at least one information input / output terminal is the second terminal, the line-of-sight information sharing method uses information on the captured image acquired by the imaging device of the first terminal. Based on the first step of outputting to the second terminal and the input information of the captured image, the captured image is displayed on the second terminal so as to be visible to the user of the second terminal. A second step, a third step of detecting a gaze direction of the user of the second terminal with respect to the displayed captured image, and gaze information relating to the detected gaze direction of the user of the second terminal, Based on the fourth step of outputting to the first terminal and the input line-of-sight information, the position of the line of sight of the user of the second terminal is associated within the display area of the video by the first terminal. And a fifth step of displaying the position.

The line-of-sight information sharing system described in the embodiment of the present invention is a line-of-sight information sharing system that shares line-of-sight information of a user between a plurality of information input / output terminals connected via a communication line, Each of the plurality of information input / output terminals includes a display device that presents an image to the user, a line-of-sight detection device that detects a user's line-of-sight direction, and a communication unit for inputting and outputting information to and from each other, At least one of the plurality of information input / output terminals is a head-mounted terminal that is mounted on a user's head and displays the video as a virtual image so as to be visible to the user. In addition to the display device and the line-of-sight detection device, the image processing device further includes an imaging device whose relative position is fixed with respect to the display device and images the external environment in front of the user, The head When the destination terminal is the first terminal and the other at least one information input / output terminal is the second terminal, the first terminal uses the captured image information acquired by the imaging device of the first terminal, The second terminal outputs the captured image to the second terminal via the communication unit of the first terminal based on the information of the captured image input by the display device of the second terminal. A line of sight regarding the line-of-sight direction is displayed after the line-of-sight detecting device of the second terminal displays the line-of-sight of the displayed image with respect to the displayed captured image. Information is output to the first terminal via the communication unit of the second terminal, and the first terminal is configured to output the second terminal based on the line-of-sight information input by the display device of the first terminal. The position of the user's line of sight is displayed in the table of the video by the first terminal. Displayed in the corresponding position in the area.

According to the above method and system, information of the captured image acquired by the imaging device of the first terminal (HMD) is output to the second terminal via the communication unit. In the second terminal, the captured image is displayed by the display device, and the visual direction of the user of the second terminal with respect to the displayed captured image is detected by the visual direction detection device. The detected line-of-sight information is output from the second terminal to the first terminal via the communication unit. Thereby, the line-of-sight information of the user of the second terminal is shared between the first terminal and the second terminal. In the first terminal, the display device displays the position of the line of sight of the user of the second terminal based on the line-of-sight information.

Here, in the first terminal, the imaging device has a fixed relative position with respect to the display device, and the captured image displayed on the second terminal is the captured image originally acquired by the first terminal. For this reason, the user's gaze point of the second terminal with respect to the captured image displayed on the second terminal is within the display area of the video (virtual image) by the display device whose relative position is fixed to the imaging device on the first terminal. Corresponding to any position. Therefore, the first terminal does not require complicated processing such as understanding the mutual spatial position of the user of the first terminal and the user of the second terminal and converting the position coordinates of the line of sight. It is possible to immediately display the position of the line of sight of the user of the two terminals at a predetermined position (the position corresponding to the gazing point of the user of the second terminal), and communicate with each other based on the display Is possible. That is, in collaborative work using a plurality of information input / output terminals, it is possible to communicate with each other based on the line of sight by a simple method without requiring the above complicated processing.

In addition, for example, even when the user of the second terminal cannot visually recognize the target object at the time of the collaborative work on the target object, the user of the second terminal outputs the captured image output from the first terminal. The object can be grasped based on the above. The user of the first terminal outputs the line-of-sight information of the user of the second terminal with respect to the captured image to the first terminal, and displays the line-of-sight position of the user of the second terminal on the first terminal. Thus, it is possible to work on the object based on the displayed line of sight. That is, even if any one of the users (the user of the second terminal in the above example) cannot visually recognize the target object, accurate communication based on the line of sight is possible.

The line-of-sight information sharing method further includes a sixth step of detecting a line-of-sight direction of the user of the first terminal. In the fifth step, the line-of-sight information sharing method is combined with the position of the line of sight of the user of the second terminal. The position of the line of sight of the user of the first terminal based on the detection result in the sixth step may be displayed. In the above-described line-of-sight information sharing system, the first terminal detects the line-of-sight direction of the user of the first terminal by using the line-of-sight detection device of the first terminal, and the display device of the first terminal Based on the detection result of the line-of-sight detection device, the position of the line of sight of the user of the first terminal may be displayed together with the position of the line of sight of the user of the second terminal.

In the first terminal, the user of the first terminal uses the second terminal by displaying the position of the user's line of sight along with the position of the user's line of sight of the second terminal. It is possible to more accurately communicate with the person based on the line of sight.

The line-of-sight information sharing method includes: a seventh step of outputting line-of-sight information related to the line-of-sight direction of the user of the first terminal to the second terminal; and the second terminal by the first step In the acquired captured image, the position of the line of sight of the user of the first terminal based on the line of sight information and the position of the line of sight of the user of the second terminal based on the detection result in the third step. And an eighth step of displaying together. In the above-described line-of-sight information sharing system, the first terminal outputs line-of-sight information regarding the line-of-sight direction of the user of the first terminal to the second terminal via the communication unit of the first terminal, The second terminal uses the display device of the second terminal to position the line of sight of the user of the first terminal based on the line-of-sight information in the captured image and the position of the line of sight of the user of the second terminal. And may be displayed together.

In the second terminal, the captured image output from the first terminal displays the position of the line of sight of the user of the first terminal and the position of the line of sight of the user of the second terminal, thereby displaying the second terminal. Can more accurately communicate with the user of the first terminal based on the line of sight.

In the line-of-sight information sharing method and the line-of-sight information sharing system, it is preferable that the line of sight of the user of the first terminal and the line of sight of the user of the second terminal are displayed in different patterns. . In this case, since it is possible to easily distinguish and grasp the mutual line-of-sight positions, it is possible to communicate quickly and accurately.

In the gaze information sharing method, at least two of the output of the captured image information, the display of the captured image, the detection of the gaze direction with respect to the captured image, the output of the gaze information, and the display of the position of the gaze You may perform mutually between information input / output terminals. In the above-described line-of-sight information sharing system, the output of the captured image information, the display of the captured image, the detection of the line-of-sight direction with respect to the captured image, the output of the line-of-sight information, and the display of the position of the line of sight are at least 2. The two information input / output terminals may be performed mutually. In this case, the line-of-sight information can be shared between at least two information input / output terminals so that the collaborative work can be performed efficiently.

In the above-described line-of-sight information sharing method and line-of-sight information sharing system, the line of sight of the user of the other information input / output terminal within the display area of the captured image output from one information input / output terminal to the other information input / output terminal It is preferable that the other information input / output terminal outputs line-of-sight information related to the user's line-of-sight direction to the one information input / output terminal only when the is located.

Only when a user of another information input / output terminal is looking at a captured image and actually needs communication based on the line of sight with the user of one information input / output terminal, By outputting the user's line-of-sight information to one information input / output terminal, it is possible to reduce the output of unnecessary (unnecessary) information to one information input / output terminal and reduce the system load.

In the line-of-sight information sharing method and line-of-sight information sharing system, the other information input / output terminal further includes an input unit for designating a timing for outputting the line-of-sight information, and at a timing designated by the input unit, The line-of-sight information of the user of the other information input / output terminal may be output to the one information input / output terminal.

For example, in another information input / output terminal, when the user of the other information input / output terminal views the display area of the captured image, the user specifies (identifies) the output timing of the line-of-sight information by the input unit. Thus, the line-of-sight information is output from another information input / output terminal to one information input / output terminal at that timing. Therefore, the configuration in which the other information input / output terminal outputs the line-of-sight information to one information input / output terminal is ensured only when the line of sight of the user of the other information input / output terminal is located within the display area of the captured image. Can be realized. In addition, the line-of-sight information output from another information input / output terminal can be surely set as the line-of-sight information when the user of the other information input / output terminal is looking at the display area of the captured image. It becomes possible to realize simple and highly accurate communication using the line-of-sight information.

In the above-described line-of-sight information sharing method and line-of-sight information sharing system, each of the plurality of information input / output terminals further includes a voice information input device and an output device, and the voice information is input between the plurality of information input / output terminals. It may be output. Since communication using voice information in addition to line-of-sight information is possible, communication is further simplified. In addition, since the content of the line-of-sight information can be supplemented with audio information, the accuracy of communication is further improved.

In the above-described line-of-sight information sharing method and line-of-sight information sharing system, the display device of the head-mounted terminal may be an optical see-through display that allows a user to directly observe the outside world together with an image. In this case, the user can perform work while directly observing the object in the outside world, and workability can be improved.

The line-of-sight information sharing method includes a step of measuring a distance between the display device of the head-mounted terminal and an object observed by a user through the see-through method, and a relative relationship of the imaging device with respect to the preset display device A step of calculating a parallax between an observation line of sight of a user observing the object and an imaging optical axis of the imaging apparatus that images the object based on a relationship between a position and the distance; And a step of correcting the position of the line of sight to be displayed based on the parallax. In the above-described line-of-sight information sharing system, the head-mounted terminal includes a measuring unit that measures a distance between the display device and an object that a user observes through, and the preset display device. Based on the relationship between the relative position of the imaging device and the distance, the parallax between the observation line of sight of the user observing the object and the imaging optical axis of the imaging device that images the object is calculated. And a parallax correction unit that corrects the position of the line of sight to be displayed based on the parallax.

Even when a parallax between the user's observation line of sight and the imaging optical axis of the imaging device occurs according to the distance between the display device and the object, the position of the line of sight to be displayed is corrected based on the parallax. The line of sight can be displayed at an appropriate position in the display image.

In the above-described line-of-sight information sharing method and line-of-sight information sharing system, the display device of the head-mounted terminal shields the front and displays a captured image of the outside world so as to be visible to the user integrally with the video. Type display. Even if the user cannot directly observe the object through the see-through, the user can grasp the object by observing the captured image of the outside world, thereby enabling the user to work on the object. .

In the above-described line-of-sight information sharing method and line-of-sight information sharing system, the display device includes a display element that displays the video, and optically displays a center of a display surface of the display element and a pupil center of a user during video observation. The imaging device may be arranged so that the observation center axis and the imaging optical axis are coaxial with each other when the axis connected to is the observation center axis. Since the parallax between the observation center axis and the imaging optical axis does not occur, a mechanism for correcting parallax is not required in order to improve the positional accuracy of the line of sight to be displayed. Can be simplified.

The present invention can be used in a system for sharing user's line-of-sight information among a plurality of information input / output terminals connected via a communication line.

5 Display Element 31 Imaging Device 32 Gaze Detection Device 36 Voice Input Device (Input Device)
37 Audio output device (output device)
39 Input unit 41 Measuring unit 42 Parallax correcting unit 100 Line-of-sight information sharing system 200 Information input / output terminal

200a Display device

201, 201A, 201B, 201C, 201D HMD (head-mounted terminal)
202, 202A, 202B, 202C, 202D Video display device (display device)
300 Communication line

Claims

A line-of-sight information sharing method for sharing line-of-sight information of a user between a plurality of information input / output terminals connected via a communication line,
Each of the plurality of information input / output terminals includes a display device that presents a video to the user, a line-of-sight detection device that detects a user's line-of-sight direction, and a communication unit for inputting and outputting information to and from each other.
At least one of the plurality of information input / output terminals is a head-mounted terminal that is mounted on a user's head and displays the video as a virtual image so as to be visible to the user,
In addition to the display device, the line-of-sight detection device, and the communication unit, the head-mounted terminal further includes an imaging device whose relative position is fixed with respect to the display device and images the external world in front of the user. ,
Among the plurality of information input / output terminals, when one of the head-mounted terminals is a first terminal and the other at least one information input / output terminal is a second terminal,
The line-of-sight information sharing method includes:
A first step of outputting information of a captured image acquired by the imaging device of the first terminal to the second terminal;
A second step of displaying, on the second terminal, the captured image so as to be visible to the user of the second terminal, based on the input information of the captured image;
A third step of detecting a line-of-sight direction of the user of the second terminal with respect to the displayed captured image;
A fourth step of outputting the detected line-of-sight information about the line-of-sight direction of the user of the second terminal to the first terminal;
A line of sight including a fifth step of displaying, based on the input line-of-sight information, the position of the line of sight of the user of the second terminal at a corresponding position in the display area of the video by the first terminal. Information sharing method.
A sixth step of detecting a gaze direction of the user of the first terminal;
The said 5th process displays the position of the user's eyes | visual_axis of the said 1st terminal based on the detection result in the said 6th process together with the position of the user's eyes | visual_axis of the said 2nd terminal. The line-of-sight information sharing method according to 1.
A seventh step of outputting gaze information related to the gaze direction of the user of the first terminal to the second terminal;
In the second terminal, based on the captured image acquired in the first step based on the position of the user's line of sight based on the line-of-sight information and the detection result in the third step. The line-of-sight information sharing method according to claim 2, further comprising an eighth step of displaying the position of the line of sight of the user of the second terminal together.
The line-of-sight information sharing method according to claim 2 or 3, wherein the position of the line of sight of the user of the first terminal and the position of the line of sight of the user of the second terminal are displayed in different patterns.
Output of the captured image information, display of the captured image, detection of the line-of-sight direction with respect to the captured image, output of the line-of-sight information, and display of the position of the line of sight are mutually performed between the at least two information input / output terminals. The line-of-sight information sharing method according to claim 1, wherein the line-of-sight information sharing method is performed.
The other information input / output terminal only when the line of sight of the user of the other information input / output terminal is located within the display area of the captured image output from one information input / output terminal to the other information input / output terminal. The line-of-sight information sharing method according to claim 1, wherein line-of-sight information relating to the direction of the line of sight of the user is output to the one information input / output terminal.
The other information input / output terminal further includes an input unit for designating a timing for outputting the line-of-sight information, and the line of sight of a user of the other information input / output terminal at a timing designated by the input unit. The line-of-sight information sharing method according to claim 6, wherein information is output to the one information input / output terminal.
Each of the plurality of information input / output terminals further includes an input device and an output device for audio information,
The line-of-sight information sharing method according to claim 1, wherein the audio information is input / output between the plurality of information input / output terminals.
The line-of-sight information sharing method according to any one of claims 1 to 8, wherein the display device of the head-mounted terminal is an optical see-through display that allows a user to directly observe the outside world together with an image.
Measuring the distance between the display device of the head-mounted terminal and an object observed by a user see-through,
Based on a relationship between the relative position of the imaging device with respect to the preset display device and the distance, an observation line of sight of a user observing the object, and imaging light of the imaging device that images the object Calculating a parallax with the axis;
The line-of-sight information sharing method according to claim 9, further comprising a step of correcting a position of the line of sight to be displayed based on the calculated parallax.
The display device of the head-mounted terminal is a video see-through display that is shielded from the front and displays a captured image of the outside world so as to be visible to the user integrally with the video. Line-of-sight information sharing method described in 1.
The display device includes a display element for displaying the video,
When the axis that optically connects the center of the display surface of the display element and the pupil center of the user at the time of video observation is the observation center axis,
The line-of-sight information sharing method according to claim 9 or 11, wherein the imaging device is arranged so that the observation center axis and the imaging optical axis are coaxial.
A line-of-sight information sharing system for sharing line-of-sight information of a user between a plurality of information input / output terminals connected via a communication line,
Each of the plurality of information input / output terminals includes a display device that presents a video to the user, a line-of-sight detection device that detects a user's line-of-sight direction, and a communication unit for inputting and outputting information to and from each other.
At least one of the plurality of information input / output terminals is a head-mounted terminal that is mounted on a user's head and displays the video as a virtual image so as to be visible to the user,
In addition to the display device and the line-of-sight detection device, the head-mounted terminal further includes an imaging device whose relative position is fixed with respect to the display device and images the external world ahead of the user,
Among the plurality of information input / output terminals, when one of the head-mounted terminals is a first terminal and the other at least one information input / output terminal is a second terminal,
The first terminal outputs information of a captured image acquired by the imaging device of the first terminal to the second terminal via a communication unit of the first terminal;
The second terminal displays the captured image so as to be visible to the user of the second terminal based on the information of the captured image input by the display device of the second terminal. After the gaze detection device detects the gaze direction of the user of the second terminal with respect to the displayed captured image, gaze information related to the gaze direction is transmitted to the first terminal via the communication unit of the second terminal. Output,
Based on the line-of-sight information input by the display device of the first terminal, the first terminal determines the position of the line of sight of the user of the second terminal within the display area of the video by the first terminal. A line-of-sight information sharing system that displays in the corresponding position.
The first terminal detects a gaze direction of a user of the first terminal by a gaze detection device of the first terminal, and based on a detection result of the gaze detection device by a display device of the first terminal The line-of-sight information sharing system according to claim 13, wherein the line-of-sight position of the user of the first terminal is displayed together with the line-of-sight position of the user of the second terminal.
The first terminal outputs line-of-sight information related to the line-of-sight direction of the user of the first terminal to the second terminal via the communication unit of the first terminal,
The second terminal uses the display device of the second terminal to position the line of sight of the user of the first terminal based on the line-of-sight information in the captured image and the position of the line of sight of the user of the second terminal. The line-of-sight information sharing system according to claim 14, which is displayed together.
The line-of-sight information sharing system according to claim 14 or 15, wherein the position of the line of sight of the user of the first terminal and the position of the line of sight of the user of the second terminal are displayed in different patterns.
Output of the captured image information, display of the captured image, detection of the line-of-sight direction with respect to the captured image, output of the line-of-sight information, display of the position of the line of sight are mutually performed between the at least two information input / output terminals. The line-of-sight information sharing system according to claim 13, wherein the line-of-sight information sharing system is performed.
The other information input / output terminal only when the line of sight of the user of the other information input / output terminal is located within the display area of the captured image output from one information input / output terminal to the other information input / output terminal. The line-of-sight information sharing system according to claim 13, wherein line-of-sight information related to the line-of-sight direction of the user is output to the one information input / output terminal.
The other information input / output terminal further includes an input unit for designating a timing for outputting the line-of-sight information, and the line of sight of a user of the other information input / output terminal at a timing designated by the input unit. The line-of-sight information sharing system according to claim 18, wherein information is output to the one information input / output terminal.
Each of the plurality of information input / output terminals further includes an input device and an output device for audio information,
The line-of-sight information sharing system according to claim 13, wherein the audio information is input / output between the plurality of information input / output terminals.
The line-of-sight information sharing system according to any one of claims 13 to 20, wherein the display device of the head-mounted terminal is an optical see-through display that allows a user to directly observe the outside world together with an image.
The head-mounted terminal is
A measuring unit for measuring a distance between the display device and an object observed by a user through the see-through;
Based on a relationship between the relative position of the imaging device with respect to the preset display device and the distance, an observation line of sight of a user observing the object, and imaging light of the imaging device that images the object The line-of-sight information sharing system according to claim 21, further comprising: a parallax correction unit that calculates a parallax with the axis and corrects a position of a line of sight to be displayed based on the calculated parallax.
The display device of the head-mounted terminal is a video see-through display that is shielded from the front and displays a captured image of the outside world so as to be visible to the user integrally with the video. Line-of-sight information sharing system described in 1.
The display device includes a display element for displaying the video,
When the axis that optically connects the center of the display surface of the display element and the pupil center of the user at the time of video observation is the observation center axis,
The line-of-sight information sharing system according to claim 21 or 23, wherein the imaging device is arranged such that the observation center axis and the imaging optical axis are coaxial.