JP2012008290A - Spectacle type display device and server - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a spectacle type display device capable of highly precisely specifying a gazing object at which a user gazes in a visual field, even if the posture of the user's head is changed, and capable of surely displaying related information related to the gazing object in real-time; and to provide a server.SOLUTION: A spectacle type display device 1 captures the image of a visual field, specifies the gazing object at which a user gazes in the visual field, and acquires and displays information related to the gazing object. The whole structure of a frame consists of a front rim 2, and left/right temples 3R and 3L. The rim 2 includes a visual field synchronous camera 9 detecting the sight line direction of the user in the visual field, and the rim 2 or the temples 3R and 3L include a posture change detection unit for detecting a posture change, a control unit, and an image display unit displaying the related information of the gazing object within the visual field. The control unit specifies the gazing object in the visual field based on the image information of the visual field and the detection result of the sight line direction of the visual field synchronous camera 9, and the detection result of the posture change detection unit, and acquires the related information.

Description

  The present invention relates to a glasses-type display device that can be worn and used in the same manner as normal glasses, and a server that provides information to be displayed on the glasses-type display device.

  Conventionally, as this type of eyeglass-type display device, a person's face is detected from an image taken in the user's field of view, and based on the line of sight detected by the line-of-sight sensor from a plurality of faces detected from the image. An eyeglass device that identifies a face to be searched is known (see Patent Document 1). The eyeglass device includes a record storage unit that stores, for each person, a record including a face image or a feature amount thereof and person information acquired for the person, searches for a record corresponding to the identified face, and When there is a record, the person information to be displayed is extracted from the person information of the corresponding record, and is displayed so as to be superimposed on the front view that the user is looking at.

  However, in the above conventional eyeglass device, when the posture of the user wearing the eyeglass device is tilted or moved up and down and the posture changes, the user's field of view changes and the use within the field of view is also performed. The person's line of sight also tends to fluctuate. Therefore, there is a possibility that the face to be searched cannot be specified accurately. Furthermore, when the posture of the user's head changes, the face of the gaze target that the user actually sees does not correspond to the person information that is displayed in the field of view and is related to the face of the gaze target. There is a possibility that the personal information cannot be displayed reliably and in real time.

  The present invention has been made in view of the above-described problems, and the object thereof is a gaze target in which the user is gazing in the field of view even if the posture of the head of the user wearing the glasses-type display device changes. A spectacle-type display device that can accurately identify related information related to the gaze target, and a server that provides information displayed on the spectacle-type display device. Is to provide.

In order to achieve the above object, a first aspect of the present invention is a glasses-type display device capable of displaying information superimposed on a field of view, a field imaging means for imaging the field of view, and a user's line-of-sight direction in the field of view. Eye-gaze direction detecting means for detecting a change in attitude of the eyeglass-type display device, image information of the field of view taken by the field-of-view imaging means, and the eye gaze detected by the eye-gaze direction detecting means Gaze target identifying means for identifying a gaze target that the user is gazing in the field of view based on a direction detection result and a posture change detection result detected by the posture change detection means; and the gaze Related information acquisition means for acquiring related information related to the gaze target specified by the target specifying means, and display means for displaying the related information acquired by the related information acquisition means in the field of view And it is characterized in and.
According to this glasses-type display device, the gaze direction detection unit detects the user's gaze direction, and based on the posture change detection result detected by the posture change detection unit, the user's gaze direction is detected. The detection result can be corrected quickly. Thereby, the precision at the time of pinpointing the gaze object which the user is gazing in a view field image can be raised. Furthermore, related information related to the gaze target can be displayed more reliably and in real time.

  In the eyeglass-type display device, the line-of-sight direction detection unit includes a user imaging unit that images the user's eyes, and the user's eye image captured by the user imaging unit is based on the image of the user's eyes. You may detect a user's gaze direction.

  In the glasses-type display device, the visual field imaging unit may also be used as the visual line direction detection unit. In this case, the direction of a predetermined position (for example, the center position of the captured image) in the image captured by the visual field imaging unit is estimated as the visual line direction of the user in the visual field.

Further, in the glasses-type display device, based on at least one of an imaging direction variable unit that changes an imaging direction of the visual field imaging unit, a detection result of the visual line direction detection unit, and a detection result of the posture change detection unit, The image pickup control unit may further include an image pickup control unit that controls the image pickup direction changing unit so that an image pickup direction of the visual field image pickup unit faces a user's line of sight. In this case, the visual field imaging unit is not used as the visual line direction detection unit and is provided separately from the visual line direction detection unit.
According to this glasses-type display device, it is possible to capture a field-of-view image centering on the line-of-sight direction in which the user is gazing, and the gaze target is located in the center of the captured image. Further, even when the captured image is enlarged so that the gaze target can be easily specified, it is difficult for the gaze target to protrude from the enlarged image. Therefore, the gaze target can be specified with higher accuracy.

The eyeglass-type display device further includes an orientation detection unit that detects an orientation of the field of view, and a current position information acquisition unit that acquires current position information of the eyeglass-type display device, and the gaze target specifying unit includes In addition, at least one of the detection result of the azimuth and the current position information may be further used for specifying the gaze target.
According to this glasses-type display device, in addition to the image information of the field of view, the detection result of the line-of-sight direction, and the detection result of the posture change of the glasses-type display device, the current position information of the glasses-type display device and the user gaze By using at least one of the azimuths in the direction in which the object is viewed to specify the gaze target, the gaze target can be specified more accurately.

Further, in the glasses-type display device, based on at least one of the image information of the field of view captured by the field of view imaging unit, the detection result of the line-of-sight direction detection unit, and the detection result of the posture change detection unit, You may further provide the display control means which controls the display position of the said related information so that the position of the said gaze target in a visual field and the display position of the said related information may become predetermined | prescribed positional relationship.
According to this glasses-type display device, when viewing a gaze target in the field of view, related information is displayed so as to be in a predetermined positional relationship with the gaze target. Therefore, the gaze target and the related information are displayed together. Visibility when viewing is improved.

Further, in the glasses-type display device, when the deviation between the position of the gaze target in the field of view and the display position of the related information is larger than a predetermined range, the display of the related information is stopped. Display control means may further be provided.
According to this glasses-type display device, it is possible to prevent related information related to the gaze target from being displayed at a position greatly deviated from the position of the gaze target in the field of view.

The eyeglass-type display device further includes gaze target distance detection means for detecting a gaze target distance between the gaze target and the user's eyes in the field of view, and the display control means includes the gaze target The display position of the related information may be controlled based on the detection result of the distance detection means.
According to this spectacles type display device, the gaze target distance between the gaze target in the field of view and the user's eyes is detected, and based on the detection result, the image forming position of the related information to be displayed superimposed on the field of view By changing, the image formation position of the related information can be adjusted to a position separated by the gaze target distance. Thereby, the user can suppress an increase in eye fatigue without causing a sense of incongruity when viewing the gaze target in the field of view and the information displayed in the field of view together.

Here, the gaze target distance detection unit may detect a relative gaze direction of the user in the field of view, and detect the gaze target distance with a target in the gaze direction as the gaze target. In this case, it is possible to more accurately detect the line-of-sight direction with respect to any gaze target that the user is actually looking at in focus in the field of view that is visible in front of the user. Therefore, the detection accuracy of the gaze target distance with respect to any gaze target actually viewed by the user in the field of view is increased. The relative line-of-sight direction of the user in the field of view is detected by, for example, the angle of the line-of-sight direction when an arbitrary gaze target in the field of view is viewed with reference to the center line-of-sight direction when viewing the center of the field of view. Can do. Further, the relative line-of-sight direction of the user in the field of view is viewed by the user in focus in the field of view that is visible in front of the user, with the center of the field of view being the origin of the coordinate axis for detection. It can detect by calculating | requiring the coordinate of arbitrary gaze objects.
In addition, the gaze target distance detection unit includes a unit that irradiates at least one eye of the user with reference light having a predetermined reference pattern and detects a reflected image from the eye irradiated with the reference light. The gaze target distance may be detected based on the degree of distortion of the reference pattern in the reflected image reflected from the eye. In this case, the relationship between the curvature of the lens surface that changes according to the thickness of the lens of the user's eye and the gaze target distance corresponding to the focal length of the eye when the user is gazing at the gaze target The curvature of the lens surface can be obtained from the degree of distortion of the reference pattern in the reflected image reflected from the eye, and the gaze target distance corresponding to the focal length of the eye can be calculated. Thus, the gaze target distance can be indirectly detected without directly measuring the distance between the user's eye and the gaze target.
In addition, the gaze target distance detection unit includes a unit that detects movement of the pupils of both eyes of the user, and detects the gaze target distance based on a degree of convergence obtained from the movement of the pupils of both eyes. May be. In this case, based on the relationship between the degree of convergence of both eyes of the user and the gaze target distance corresponding to the focal length of the eye when the user is gazing at the gaze target, The gaze target distance corresponding to the focal length of the eye can be calculated from the degree of convergence obtained from the pupil movement. Thus, the gaze target distance can be indirectly detected without directly measuring the distance between the user's eye and the gaze target. The degree of convergence can be obtained, for example, from the distance between the pupil centers of both eyes of the user, or from the relative position of the pupil (or black eye portion) in the overall shape of the eye.
A first display mode for adjusting and displaying the image forming position of the information according to the gaze target distance detected by the gaze target distance detecting unit; and the image forming position of the information regardless of the gaze target distance. May be configured to be switchable to the second display mode for displaying the image in a fixed manner. In this case, when it is desired to confirm the gaze target in the field of view being viewed by the user and related information related to the gaze target together, the first display mode is selected and executed, Both the gaze target and the related information can be confirmed with almost no change in focus. On the other hand, regardless of the field of view visible to the user, when it is desired to check the information displayed on the glasses-type display device, the second display mode is selected and executed, thereby being disturbed by the field of view image. The information can be confirmed using the image of the field of view as a background. Thus, an appropriate display mode can be selected according to the content of information that the user wants to confirm.

The glasses-type display device may further include means for receiving and acquiring information stored in a storage medium provided for the gaze target by short-range wireless communication. The information acquired from the storage medium provided for the gaze target may be the related information displayed on the display unit, or may be information that can be used for specifying the gaze target or searching for related information.
According to this glasses-type display device, short-range wireless communication can be performed without searching related information in the memory in the glasses-type display device or searching related information for a server on the communication network. The related information received in (1) can be displayed. Further, the gaze target can be specified based on the information that can be used to specify the gaze target received by the short-range wireless communication.

The eyeglass-type display device further includes communication means for communicating with a server via a communication network, wherein the related information acquisition means includes the image information of the gaze target, the feature extraction information of the gaze target, and the Transmitting at least one piece of information acquired from a storage medium provided for a gaze target to the server, and receiving and obtaining related information related to the gaze target searched by the server from the server. There may be.
According to this glasses-type display device, at least one of the image information of the gaze target, the feature extraction information of the gaze target, and the information acquired from the storage medium provided in the gaze target is transmitted via a communication network. Thus, the information is transmitted to a server having a relatively high processing capability and easily equipped with various functions such as a database function, and the related information retrieved by the server is received and acquired. By communicating with the server in this way, it is possible to efficiently acquire from the server related information that is improved in the accuracy of consistency between the identification of the gaze target and the related information. In addition, it is also possible to acquire related information that has more detailed information about the gaze target and a wide range of variations.
Here, instead of providing a communication means for communicating with the server via the communication network, a short-range communication means for communicating with a separate communication terminal device by wired or wireless short-range communication may be provided. In this case, a separate communication terminal device capable of communicating by wired or wireless short-range communication even if the glasses-type display device itself does not have a communication function for connecting to a communication network such as a mobile communication network. It is possible to connect to a communication network and communicate with a server on the communication network.

  The server according to the present invention is a server that can communicate with the glasses-type display device via a communication network, and is provided in the image information of the gaze target, the feature extraction information of the gaze target, and the gaze target. Information receiving means for receiving at least one piece of information acquired from the storage medium from the glasses-type display device, and identifying the gaze target based on the information received from the glasses-type display device; Search means for searching for related information related to the gaze target, and information transmission means for transmitting related information related to the gaze target searched by the search means to the glasses-type display device.

The glasses-type display device further includes a latest information acquisition unit that acquires at least one latest information of the user's schedule information, mail information addressed to the user, and distribution information addressed to the user, The display means may display the latest information acquired by the latest information acquisition means so as to overlap the field of view.
According to this glasses-type display device, the user can confirm the latest information such as his / her schedule information by a simple operation of wearing the glasses-type display device. In addition, the surrounding people are not aware that the user is checking the latest information.

A communication terminal system according to the present invention includes the glasses-type display device and a communication terminal device capable of communicating with the glasses-type display device by wired or wireless short-range communication.
According to this communication terminal system, even if the glasses-type display device itself does not have a communication function for connecting to a communication network such as a mobile communication network, the communication terminal device can communicate by wired or wireless short-range communication. It is possible to communicate with a server on the communication network by connecting to the communication network via the network.

  According to the present invention, the gaze direction of the user is detected by the gaze direction detection unit, and the gaze direction of the user is detected based on the detection result of the posture change of the glasses-type display device detected by the posture change detection unit. The direction can be corrected quickly. Thereby, it is possible to improve the accuracy when specifying the gaze target that the user is gazing in the visual field image captured by the visual field imaging means, and more reliably and in real time related information related to the gaze target. Can be displayed.

1 is a perspective view illustrating a configuration example of a glasses-type display device according to an embodiment of the present invention. (A) is a top view of the eyeglass-type display device. FIG. 6B is a top view of a glasses-type display device according to another configuration example. The functional block diagram which shows the example of 1 structure of spectacles type display apparatus. (A) And (b) is explanatory drawing for demonstrating a user's eyes | visual_axis direction and the picked-up image of a visual field synchronization camera, respectively. (A) is a schematic block diagram which shows the example of 1 structure of a gaze target distance detection part. (B) is explanatory drawing of the change of the striped pattern in a gaze target distance detection part. The graph which shows an example of the time change of the lens thickness of a user's left eye eyeball. The schematic block diagram which shows an example of an image generation part, an image projection display part, and a focal distance adjustment part. (A) is explanatory drawing in the case of displaying an image according to the gaze object far from the eye. (B) is explanatory drawing in the case of displaying an image according to the gaze object with a short distance from eyes. The schematic block diagram which shows an example of the whole structure of the information provision system which displays the information of the attention object with a spectacles type display apparatus. The sequence diagram for demonstrating the process which displays the information of the attention object which the user watched. Explanatory drawing of the state which is displaying the operation menu of household appliances on a spectacles type display apparatus. The sequence diagram for demonstrating the process which displays the operation menu of household appliances. Explanatory drawing of the state which is displaying the time and temperature on the spectacles type display apparatus. The sequence diagram for demonstrating the process which displays time and temperature. The perspective view which shows one structural example of the spectacles type display apparatus provided with the gaze detection part. (A) is a schematic block diagram which shows the example of 1 structure of a gaze detection part. (B) is a schematic diagram of the horizontal scanning output signal of the image sensor corresponding to each position of the eyeball. Explanatory drawing which shows the area | region which divided | segmented into the front view interface which a user sees through the right spectacles lens into plurality. Explanatory drawing of the gaze cursor and icon displayed on the front view. The functional block diagram of the spectacles type display apparatus which concerns on the other structural example. Explanatory drawing of the system which measures the gaze target distance using the convergence of both eyes.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a perspective view showing a configuration example of a glasses-type display device 1 according to an embodiment of the present invention. 2A and 2B are top views of the eyeglass-type display device 1 as viewed from above. The glasses-type display device 1 is mounted on the head so as to cover the front of both eyes of the user, so that the user can visually recognize various information such as related information on the gaze target that the user has watched. It can be displayed superimposed.

  The frame of the glasses-type display device 1 includes a rim 2. A pair of temples 3R, 3L, also called vines, are held on both the left and right sides of the rim 2 by hinges 4R, 4L (4L is not shown) so that they can be opened and closed by about 90 degrees. The pair of eyeglass lenses 5R and 5L, the pair of nose pads 6R and 6L, and the pair of temples 3R and 3L are provided with the tip cells 7R and 7L at the ends opposite to the side where the hinges 4R and 4L are provided. Yes. It should be noted that health information detecting means such as a temperature sensor, a pulse sensor, a blood pressure sensor, etc. are provided in the portion of the previous cell 7R, 7L that comes into contact with the user so that health information such as the user's body temperature, pulse, blood pressure, etc. can be detected. May be. The detected health information such as the body temperature, pulse, and blood pressure of the user may be displayed superimposed on the front view so that the user can visually recognize the health information.

  The glasses-type display device 1 also includes a power switch 8 for turning on / off the power inside the temple 3R on the right side, and a front view in the direction of the user's line of sight provided on the front side of the center upper portion of the rim 2. A visual field tuning camera 9 as a visual field imaging unit that captures the above image, and a gaze target distance detection unit 10 as a gaze target distance detection unit provided on the lower side of the lens 5L on the left side of the rim 2. Further, the glasses-type display device 1 includes an image display unit 12 as a display unit that displays information so that the user can visually recognize the rim 2 on the right end of the rim 2 so as to be superimposed on the front field of view seen through the glasses lens 5R. Yes.

  As an image display method for displaying information so that the user can visually recognize the image superimposed on the front field of view, various image display methods such as a method of directly scanning the retina and a method using a combiner optical system can be used. .

FIG. 2A shows a configuration example in the case of adopting a retinal scanning image display method. In the retinal scanning glasses-type display device 1, the scanning optical system 124 is disposed in front of the right lens 5R.
FIG. 2B shows a configuration example in which an image display method using the combiner optical system 125 is adopted. The combiner optical system 125 is formed of, for example, a flat transparent substrate having a polarization beam splitter, a quarter-wave plate, and a main reflection surface therein, and is provided so as to be embedded in the right lens 5R. The combiner optical system 125 is configured to guide the display light beam emitted from the image display surface of the light source inside the right temple 3R in the direction of the user's eyes.

FIG. 3 is a functional block diagram illustrating a configuration example of the glasses-type display device 1.
Inside the rim 2 or the temples 3R, 3L of the glasses-type display device 1, at least a part of the components constituting the image display unit 12, an image generation unit 11 that generates an image signal used in the image display unit 12, A display control unit 13 as a display control unit for controlling image display on the image display unit 12, a control unit 14 as a control unit constituted by a CPU, a ROM, etc., a memory 15 as a storage unit, and a power supply unit The battery 16 and the mounting detection unit 17 are stored. The glasses-type display device 1 further includes a communication unit 25 and a posture change detection unit 28.

  A program executed by the control unit 14 and various data used by the control unit 14 are stored in the memory 15. By reading and executing a predetermined program and data in the control unit 14, various controls and data processing described later are executed.

  The control unit 14 controls the control unit 14 based on the visual field image information and the visual line direction detection result of the visual field tuning camera 9 and the detection result of the posture change of the glasses-type display device 1 detected by the posture change detection unit 28. It also functions as a gaze target specifying means for specifying a gaze target that the user is gazes in the field of view by processing data in the network or receiving support from a server on the communication network. Furthermore, the control unit 14 can also be used as related information acquisition means for acquiring related information related to the specified gaze target by referring to data in the memory 15 or receiving support from a server on the communication network. Function.

The display control unit 13 relates to the position of the gaze target in the visual field based on at least one of the visual field image information captured by the visual field tuning camera 9 and the detection result of the visual line direction and the detection result of the posture change detection unit 28. The display position of the related information is controlled so that the display position of the information has a predetermined positional relationship.
Further, the display control unit 13 may perform control so as to stop the display of the related information when the shift between the position of the gaze target in the forward view and the display position of the related information becomes larger than a predetermined range.
Further, the display control unit 13 changes the image formation position (focus position) of the image to be displayed so that the line of sight when the user is looking at the gaze target is in focus. It also functions as a variable means (focal length adjustment unit).

  The communication unit 25 functions as a communication unit for communicating with various servers on a communication network via a public wireless LAN access point or a mobile communication terminal such as a mobile phone. The communication unit 25 also functions as the gaze target specifying unit and the related information acquiring unit by cooperating with the control unit 14. For example, by controlling the communication unit 25 by the control unit 14, the server on the communication network is accessed, and at least one of the gaze target image information and the gaze target feature extraction information extracted from the image information is Send to server. Information such as a gaze target image transmitted to the server may include a detection result of the posture change of the glasses-type display device 1. Further, the control unit 14 controls the communication unit 25 to receive, from the server, related information related to the gaze target searched based on at least one of the gaze target image information and the gaze target feature extraction information. And get.

  The eyeglass-type display device 1 can be used for specifying a gaze target and searching related information from a storage medium such as an RFID (Radio Frequency IDentification) tag (hereinafter referred to as “RF tag”) provided for the gaze target. When information (for example, information such as name, identification information, and type information) is received and acquired from the gaze target by short-range communication, the acquired information may be transmitted to the server. The information that can be used for specifying the gaze target and searching for related information may be transmitted to the server together with any of the gaze target image information, gaze target feature extraction information, and posture change detection result information. Alternatively, the information may be transmitted to the server instead of the information. By using the information acquired from the RF tag, the accuracy of the gaze target specification and related information search is improved. The storage medium provided for the gaze target is not limited to the RF tag as long as it has a function of wirelessly communicating with the eyeglass-type display device 1 and a function of storing information. When the gaze target includes a storage medium other than the RF tag, the communication when the glasses-type display device 1 receives information from the storage medium provided for the gaze target is, for example, communication using Bluetooth (registered trademark). Or near field communication such as infrared communication can be used.

The posture change detection unit 28 is composed of, for example, a uniaxial, biaxial, or triaxial acceleration sensor, and detects the acceleration acting on the spectacles-type display device 1 to thereby detect the spectacles accompanying the posture change of the user's head. It functions as a posture change detection means for detecting a posture change of the display device. As the acceleration sensor constituting the posture change detection unit 28, a sensor capable of detecting the direction of gravity (a sensor capable of detecting absolute acceleration) may be used. The posture change detection unit 28 uses, for example, the posture of the glasses-type display device detected at a predetermined timing as the reference posture, and changes the posture of the user's head from the reference posture (for example, the roll angle and pitch from the reference posture). The angle change value of each of the angle and the yaw angle, or the acceleration value generated due to the change in posture of the glasses-type display device) is output as a detection result. The detection timing of the reference posture may be, for example, a timing when the use of the glasses-type display device 1 is started, a timing when a predetermined operation is performed, or a timing when a gaze target that the user is gazing at is specified. .
Note that the posture change detection unit 28 includes a geomagnetic sensor together with or instead of the acceleration sensor, and is based on the direction information detected by the geomagnetic sensor with reference to the coordinates defined in the glasses-type display device 1. You may comprise so that the attitude | position change of the display apparatus 1 may be detected.

  The detection result of the posture change of the glasses-type display device 1 detected by the posture change detection unit 28 is, for example, the following (1) to (1) when specifying the gaze target that the user is gazing in the field of view. It can be used for various controls including 3).

(1) Control of blurring of user's head and line of sight:
Although the gaze target that the user intends to gaze is the same, for some reason, the head of the user wearing the glasses-type display device 1 may be shaken such as vibration or wobbling. . When the user's head is shaken, the line-of-sight direction in the field-of-view image captured by the field-of-view tuning camera 9 of the glasses-type display device 1 attached to the head is also blurred. There is a possibility that the accuracy of specifying the gaze target of the user specified based on the information may be lowered. Therefore, by performing control to correct the line-of-sight direction in the field-of-view image captured by the field-of-view tuning camera 9 based on the detection result of the position change of the eyeglass-type display device 1 detected by the position change detection unit 28, The target identification accuracy can be improved.

(2) Mitigation control for specific processing of gaze target:
If the process of specifying the gaze target based on the image of the field of view captured by the field-of-view tuning camera 9 and the detection result of the line-of-sight direction is frequently executed, the processing load on the glasses-type display device 1 may increase. Therefore, focusing on the point that there is a high possibility that the gaze target that the user wearing the spectacles-type display device 1 is gazing at has changed when the posture of the spectacles-type display device 1 is greatly changed. When the detection result of the posture change (posture change amount) of 1 is larger than a preset threshold range, the visual field tuning camera 9 performs the visual field image capturing process and the visual line direction detection process. Control is performed so as to perform a gaze target identification process based on the visual field image and the detection result of the visual line direction. As a result, it is possible to identify the user's gaze target while avoiding an increase in the processing load in the glasses-type display device 1.

(3) Complementary control of specific processing for gaze target:
There are cases where the imaging process of the image of the view and the detection process of the direction of the line of sight by the view tuning camera 9 cannot follow the posture change of the glasses-type display device (change in the posture of the user's head). In this case, there is a possibility that the user's gaze target cannot be specified in real time based on the captured image or the line-of-sight direction of the field of view, or the accuracy of specifying the gaze target of the user to be specified may be reduced. Therefore, based on the processing time required for the imaging process of the visual field image by the visual field tuning camera 9 and the detection process of the visual line direction, intervals for performing the imaging process and the visual line direction detection process are set in advance. Then, after specifying the user's gaze target based on the captured image and the line-of-sight direction of the field of view, until the next imaging / line-of-sight detection timing that arrives thereafter, based on the detection result of the posture change of the glasses-type display device 1 Control is performed so as to identify the gaze target by complementing the image of the most recently captured field of view and the detected line-of-sight direction. With this control, the user's gaze target can be identified in real time without lowering the gaze target identification accuracy.

  The power switch 8 is, for example, a 3P toggle switch for turning on / off the power of the glasses-type display device 1 and can take three positions of power OFF, AUTO, and power ON. Here, the AUTO position is such that a weak voltage is applied to the pair of front cells 7R, 7L with the pair of temples 3R, 3L open, and the user wears the glasses-type display device 1 on the head. When a weak current flowing between the pair of previous cells 7R and 7L is detected by the mounting detection unit 17 configured by a touch sensor or the like, the operation such as the image display function is started. On the other hand, when the user removes the glasses-type display device 1, the image display function or the like is controlled to stop. In the power ON position, the power display is turned on and the image display function and the like operate. However, power saving control is performed so that the power is automatically turned off when a predetermined time elapses with the user removing the glasses-type display device 1. The Note that the power supply of the glasses-type display device 1 may be automatically turned on / off in the same manner as the power supply operation at the AUTO position without providing the power switch 8.

  The visual field tuning camera 9 is composed of a solid-state image sensor such as a CCD camera or a CMOS camera, for example, and is disposed at the upper center of the rim 2 of the glasses-type display device 1 to capture the forward visual field viewed by the user. The visual field synchronization camera 9 can acquire image data of a forward visual field image and use it for specifying a gaze target and performing various analyses.

  In the configuration example of FIG. 1, the visual field synchronization camera 9 is used as a visual field imaging unit and also as a simple visual direction detection unit that detects a user's visual direction in the visual field. The field-of-view tuning camera 9 of this configuration example is provided so that the center of the captured image is positioned substantially at the center of the user's field of view, and the center of the captured image captured by the field-of-view camera 9 (approximately the center of the user's field of view). ) To detect the user's line-of-sight direction. Here, when the user changes the gaze direction by changing the posture of the head, the view synchronization camera 9 of the glasses-type display device 1 attached to the user's head is changed according to the change in the gaze direction. It can be estimated that the imaging direction changes and the direction toward the center of the captured image captured by the visual field tuning camera 9 (substantially the center of the user's visual field) is the user's visual line direction.

  FIG. 4 is a diagram for explaining a user's line-of-sight direction and a captured image of the visual field tuning camera 9. FIG. 4A is a diagram showing a state in which the user's head is horizontally inclined diagonally to the right and looking at the right tree Tr1 among the three trees, and FIG. 4B is a view tuning camera 9. It is the figure which superimposed and displayed the information of tree Tr1 on the image imaged in FIG.

  As shown in FIG. 4 (a), when a user wearing the glasses-type display device 1 gazes at the right tree Tr1 with his head horizontally and diagonally rightward, as shown in FIG. 4 (b), In synchronization with the movement, the view synchronization camera 9 images the front view. Since the user's line-of-sight direction substantially coincides with the direction of the line-of-sight mark 901 at the center of the field-of-view image captured by the field-of-view synchronization camera 9, the eyeglass-type display device 1 is that the user is gazing at the tree Tr1 on the right side. to decide. Then, the gaze target distance detection unit 10 calculates and detects a gaze target distance L1 from the user's eye to the right tree Tr1, and analyzes the right tree Tr1 to obtain a gaze target such as a name and type. Related related information is acquired from the memory 15 or a server on the communication network, and an image (virtual image) of the acquired related information 902 is displayed on the upper right side of the tree Tr1 on the right side as shown in the view screen of FIG. Are displayed on top of each other. The image (virtual image) of the related information 902 is displayed at the same focal position as the right tree Tr1 based on the detected gaze target distance L1. Thereby, in a state where the user's eye is focused on the tree Tr1 on the right side, the display image of the related information 902 of the gaze target is not blurred and can be read without a sense of incongruity. On the other hand, when the user gazes at the tree Tr3 on the left side with the head horizontally oriented diagonally to the left, the gaze target distance L3 is shortened and the user's eyes are in close focus. The formation position of the image of the relevant information is adjusted so that the display image of the related information of the tree Tr3 on the left side which is the gaze target is not blurred, and the user can read without discomfort.

  The line-of-sight mark 901 is displayed on the retina of the user by irradiation, but may be displayed in advance at the center of at least one of the pair of eyeglass lenses 5L and 5R by printing or the like.

FIGS. 5A and 5B are explanatory diagrams illustrating a configuration example of the gaze target distance detection unit 10, where FIG. 5A is a schematic configuration explanatory diagram and FIG. 5B is an explanatory diagram of a stripe pattern change. The gaze target distance detection unit 10 of the present configuration example irradiates the region including the crystalline lens of the user's left eyeball with the striped infrared light, and based on the change in the striped pattern of the infrared light reflected from the crystalline lens Is measured, and the gaze target distance from the user's eye to the gaze target is calculated and detected from this measurement result.
In FIG. 5A, the gaze target distance detection unit 10 includes, for example, a light source 101 such as a light emitting diode that emits infrared light, a polarizing filter 102, a light projecting lens 103, a light receiving lens 104, and a CCD or CMOS. An image sensor 105 as an imaging unit including a solid-state imaging device or the like, and a focal length calculation unit 106.

  The polarization filter 102 is a polarization filter that polarization filters infrared light emitted from the light source 101 into a lattice-like stripe pattern. In this grid-like striped pattern, a plurality of vertical lines and horizontal lines forming the grid have the same line spacing, and the line spacing is sufficiently narrower than the diameter of the crystalline lens (for example, 1 to a fraction of the diameter). Several tenths of an interval).

  Infrared light emitted from the light source 101 is polarized and filtered into a lattice-like striped pattern by the polarizing filter 102, and after passing through the projection lens 103, becomes infrared light C in the form of lattice stripes. An area including the crystalline lens is irradiated (see FIG. 5B). The infrared light C reflected by the crystalline lens forms a striped image I on the image sensor 105 via the light receiving lens 104. The image sensor 105 transmits the image I as an image signal to the focal length calculation means 106 at regular time intervals (for example, every 1/30 seconds).

As shown in FIG. 5B, the focal length calculation means 106 corrects lens distortion for the input image I and traps the tilt of the image every time an image signal from the image sensor 105 is input. Correction is performed to obtain a corrected image I ′. Lens distortion occurs according to the characteristics of the light receiving lens 104 and the like. Further, the inclination of the image is generated according to the positional relationship between the lens position of the user and the light receiving lens 104. The corrected image I ′ corresponds to an image in which there is no lens distortion and the lens is imaged from the front. Further, as preprocessing, noise removal or filtering may be performed on the image in order to make it easier to detect the lattice pattern from the input image.
Further, the focal length calculation means 106 traces the lattice pattern S in the corrected image I ′ and extracts the lattice pattern S. This lattice pattern S is distorted along the curved surface of the crystalline lens in the portion reflected in the crystalline lens, and is a lattice pattern composed of curves from the lattice pattern composed of straight lines at the time of irradiation.

  The focal length calculation means 106 calculates the curvature of each curve and the interval between the curves (vertical line interval, horizontal line interval) from the extracted lattice pattern S. Further, the focal length calculation means 106 calculates the curvature of the crystalline lens based on the curvature of the lattice pattern S and the line spacing, and calculates the thickness of the crystalline lens based on the curvature of the crystalline lens.

  The crystalline lens is a convex lens that refracts incoming light and focuses it on the retina, and the ciliary muscles are connected via the Chin band. When looking at nearby objects, the ciliary muscles contract and the chin band relaxes, resulting in a thicker lens (curvature increases). Also, when looking at a distant object, the ciliary muscle relaxes and the chin band is pulled, thereby reducing the thickness of the crystalline lens (decreasing the curvature). In this way, the person adjusts the focal length by unconsciously expanding and contracting the thickness of the crystalline lens according to the distance to the object to be viewed. The thickness of the crystalline lens is generally about 3.6 mm at the normal time and about 4.0 mm at the maximum. The diameter of the crystalline lens when viewed from the front is generally about 9 to 10 mm.

FIG. 6 shows an example of the temporal change in the thickness of the lens of the left eye eyeball of the user. Whether or not the user's eyes are focused on the gaze target may be determined as follows, for example. In the time period indicated at T ₀ in the figure, focus than the time period before and after, has duration thickness of the lens becomes thicker, a short focal length, that is, close to the fixation target It can be determined that On the other hand, when the time when the thickness of the crystalline lens is reduced continues for a predetermined time, it can be determined that the subject is focused on a distant gaze target.

  The image generation unit 11, the image display unit 12, and the display control unit (focal length adjustment unit) 13 are superposed on the front field of view that can be seen through the glasses lens 5 </ b> R based on the image signal from the control unit 14 so that the user can visually recognize the image. In addition, various information images (virtual images) are displayed in accordance with the focal position where the user's line of sight is in focus.

FIG. 7 shows the focal length adjustment unit 130 that constitutes part of the image generation unit 11, the image display unit 12, and the display control unit 13 when the retinal scanning image display method shown in FIG. It is schematic structure explanatory drawing which shows one structural example.
In FIG. 7, a light source unit 110 for processing an image signal supplied from the control unit 14 is provided. The light source unit 110 is provided with an image signal supply unit 111 that receives an image signal from the control unit 14 and generates an image signal for generating an image based on the image signal. The image signal supply unit 111 receives the image signal from the image signal supply unit 111. 112, a vertical synchronization signal 113 and a horizontal synchronization signal 114 are output. The light source unit 110 also displays a laser oscillation unit 115 serving as a light source that emits laser light whose intensity is modulated based on the image signal 112 transmitted from the image signal supply unit 111, and is superimposed on the front field of view. A focal length adjustment unit 130 that adjusts the focus of the image scanned on the retina is provided so that the formation position of the image (virtual image) to be matched with the focus position when the user is looking at the gaze target.

  Further, a vertical scanning system 121 as a scanning optical system that scans laser light guided from the light source unit 110 side in the vertical direction using the galvano mirror 121a, and laser light scanned by the vertical scanning system 121 will be described later. A first relay optical system (lens group) 123 guided to the horizontal scanning system 122 and a laser beam scanned by the vertical scanning system 121 and incident through the first relay optical system 123 are horizontally converted using a galvano mirror 122a. A horizontal scanning system 122 as a scanning optical system that scans in the direction, and a second relay optical system (lens group) 124 that causes the laser light scanned by the horizontal scanning system 122 to enter the pupil of the right eye 100R of the user. It has been.

  The vertical scanning system 121 is an optical system that performs vertical scanning for vertically scanning a laser beam in the vertical direction for each scanning line of an image to be displayed. The vertical scanning system 121 includes a galvano mirror 121a as an optical member that scans a laser beam in the vertical direction, and a vertical scanning control unit 121b that controls driving of the galvano mirror 121a.

  On the other hand, the horizontal scanning system 122 is an optical system that performs horizontal scanning in which a laser beam is scanned horizontally from the first scanning line toward the last scanning line for each frame of an image to be displayed. The horizontal scanning system 122 includes a galvano mirror 122a as an optical member that performs horizontal scanning, and a horizontal scanning control unit 122b that controls driving of the galvano mirror 122a.

  Further, as shown in FIG. 7, the vertical scanning system 121 and the horizontal scanning system 122 are connected to the image signal supply unit 111 and output to the vertical synchronization signal 113 and the horizontal synchronization signal 114 respectively output from the image signal supply unit 111. The laser beam is scanned in synchronization.

  The process of displaying an image on the user's retina by the scanning optical system having the above configuration is performed, for example, as follows. Here, as shown in FIG. 4B, the image displayed on the retina of the user is related information about the tree to be watched, such as “name: fir pine family fir genus evergreen conifer”. As shown in FIG. 7, in the eyeglass-type display device 1 of the present embodiment, when the image signal supply unit 111 provided in the light source unit 110 receives the supply of the image signal from the control unit 14, the image signal supply unit 111. Outputs, for example, an image signal 112 for outputting white laser light, a vertical synchronization signal 113, and a horizontal synchronization signal 114. Based on the image signal 112, the laser oscillator 115 generates intensity-modulated laser light and outputs it to the vertical scanning system 121. The laser light incident on the galvano mirror 121 a of the vertical scanning system 121 is scanned in the vertical direction in synchronization with the vertical synchronization signal 113 and is incident on the galvano mirror 122 a of the horizontal scanning system 122 via the first relay optical system 123. The galvanometer mirror 122a is reciprocally oscillated so as to reflect incident light in the horizontal direction in synchronization with the horizontal synchronization signal 114 in the same manner as the galvanometer mirror 121a is synchronized with the vertical synchronization signal. Are scanned horizontally. Laser light that is two-dimensionally scanned in the vertical and horizontal directions by the vertical scanning system 121 and the horizontal scanning system 122 is incident on the right eyeball of the user by the second relay optical system 124 and projected onto the retina. Thus, the user can recognize the image by the laser light that is two-dimensionally scanned and projected onto the retina.

  FIG. 8 is an explanatory diagram of a schematic configuration of the focal length adjustment unit 130. FIG. 8A is a configuration diagram when an image is displayed according to a gaze target that is far from the eye, and FIG. 8B is a configuration when an image is displayed according to a gaze target that is close to the eye. FIG. Inside the laser oscillating unit 115, a laser light source 116 is provided at the left end of the light source barrel 117, and a lens barrel 119 having a light projecting lens group 118 is arranged in the left-right direction inside the light source barrel 117. Are arranged so as to be linearly movable. The focal length adjusting unit 130 includes a micromotor 131 that is rotationally driven by the control unit 14, a worm gear 132 that is rotationally driven by the micromotor 131, and a rack gear 133 that is screwed with the worm gear 132 and linearly moves left and right in the drawing. Has been. The rack gear 133 is fixed to the lens barrel 119, and when the worm gear 132 is rotationally driven by the micromotor 131, the rack gear 133 and the lens barrel 119 are integrated to perform a linear motion in the left-right direction.

As shown in FIG. 8A, when the lens barrel 119 is at the rightmost position, that is, when the laser light source 116 and the light projecting lens group 118 are at the farthest positions, the laser light source 116 emits light. The laser beam is converted into parallel light by the projection lens group 118 and is incident on the galvanometer mirror 121a of the vertical scanning system 121. Finally, when parallel light enters the right eye of the user from the second relay optical system 124, it feels that an image is projected at infinity.
On the other hand, as shown in FIG. 8B, when the lens barrel 119 is at the leftmost position, that is, when the laser light source 116 and the light projecting lens group 118 are at the closest position, the laser light source 116 The emitted laser light is diffused by the light projection lens group 118 and is incident on the galvanometer mirror 121a of the vertical scanning system 121. Finally, the diffused light is incident on the user's right eye from the second relay optical system 124, so that the user feels that an image is displayed nearby (for example, 30 centimeters forward).

  In addition, although not shown about the case where the image display system using the combiner optical system 125 shown in FIG.2 (b) is employ | adopted, the image display part 12 has the light source which consists of a small-sized liquid crystal display surface, its An image forming optical system for guiding the image of the small liquid crystal display surface toward the combiner optical system 125 while forming an image at a predetermined focal length is used. The focal length adjusting unit 130 includes a micro motor and a worm gear that move a lens of the imaging optical system so as to change the focal length of the imaging image by the imaging optical system.

  Next, a configuration of an information providing system that identifies a gaze target that the user gazes in the eyeglass-type display device 1 of the present embodiment, and acquires and displays related information related to the gaze target from a server on the communication network. explain.

  FIG. 9 is an explanatory diagram illustrating a configuration example of the entire information providing system that displays related information on the gaze target by the glasses-type display device 1. FIG. 10 is a sequence diagram for explaining processing for displaying related information of a gaze target that the user gazes at. Here, the glasses-type display device 1 communicates with the mobile communication terminal 18 such as a mobile phone which is a mobile communication terminal by wireless or wired short-range communication as the communication unit (communication means) 25 of FIG. Short-range communication means. This short-distance communication means is configured to communicate with the mobile communication terminal 18 by, for example, wired communication using a serial communication cable, or by wireless communication such as infrared or Bluetooth.

  In the configuration example of FIG. 9, the glasses-type display device 1 can communicate with the mobile communication terminal 18 by short-range wireless communication, and the mobile communication terminal 18 transmits an image via the mobile communication network 19 and the Internet communication network 20. It is possible to communicate with the analysis server 21 and the information providing server 22. The eyeglass-type display device 1 may be provided with a network communication function so that it can be directly connected to the mobile communication network 19 or the Internet communication network 20 without using the mobile communication terminal 18.

In FIG. 10, when a user gazes at a tree at the center of the field of view as a gaze target (step 1), the gaze target distance detection unit 10 of the eyeglass-type display device 1 attached to the user's head moves the user's eyeball. Is detected to be locked on the tree (step 2), and a shooting command is output to the view tuning camera 9 (step 3). The field-of-view synchronization camera 9 that has received the imaging command captures the gaze target (step 4), and the captured image data of the gaze target is transmitted to the image analysis server 21 via the mobile communication terminal 18, the mobile communication network 19, and the Internet communication network 20. (Step 5). The image analysis server 21 that has received the image data performs image analysis, extracts features of the gaze target, and identifies the gaze target (step 6). The image analysis server 21 transmits the gaze target information specified by the image analysis to the information providing server 22 (step 7). The information providing server 22 searches the database based on the gaze target information received from the image analysis server 21, and searches for and collects related information related to the gaze target (step 8). The information providing server 22 transmits related information collected about the gaze target to the glasses-type display device 1 via the Internet communication network 20, the mobile communication network 19 and the portable communication terminal 18, and the control unit 14 of the glasses-type display device 1. Sends the relevant information of the gaze target received from the information providing server 22 to the image generation unit 11 (step 9).
In the example of FIG. 10, the gaze target information specified by the image analysis server 21 through image analysis is transmitted to the information providing server 22, but the glasses-type display device 1 receives the gaze target information from the image analysis server 21. May be received and stored, and the relevant information may be acquired by transmitting the gaze target information to the information providing server 22.
Further, whether or not the user's line of sight is locked on a tree (a gaze target) can be determined based on the time during which the focus of the user's eyeball remains on the tree. For example, when a state where the focus of the user's eyeball remains on a tree continues for a predetermined time, it can be determined that the tree is locked on. The predetermined time is set according to the type of the gaze target, and more specifically may be set to several seconds (for example, about 2 to 5 seconds).

  In parallel with the acquisition of the relevant information of the gaze target, the gaze target distance detection unit 10 of the glasses-type display device 1 calculates (detects) the gaze target distance from the eye to the gaze target (step 10). The calculated gaze target distance data is sent to the focal length adjustment unit 130 (step 11). The focal length adjustment unit 130 that has received the gaze target distance data adjusts the focal length according to the gaze target distance (step 12), and sends the adjustment completion information to the image generation unit 11 when the adjustment is completed (step 12). 13). Based on the adjustment completion information received from the focal length adjustment unit 130 and the related information of the gaze target received from the information providing server 22, the image generation unit 11 displays an image signal of related information to be displayed superimposed on the front view. An image to be generated is generated (step 14), and an image signal of the generated display image is output to the image display unit 12 (step 15). The image display unit 12 displays an image (virtual image) of related information related to the gaze target (tree) so as to be formed at a position away from the user's eyes by the gaze target distance (step 16). Thereby, the user can visually recognize the related information 902 regarding the gaze target (tree) displayed superimposed on the front view as shown in FIG. 4B (step 17). By matching the formation position of the image (virtual image) to be displayed in this way with the position of the tree that is the gaze target, the user can view the image of the related information without defocusing while gazing at the tree.

  In the eyeglass-type display device 1 according to the present embodiment, an operation menu of a gaze target device (for example, a home appliance such as a television) is displayed as the gaze target related information, and a manual operation is performed based on the operation menu. You may make it operate an apparatus, without performing.

  FIG. 11 is an explanatory diagram showing a state in which an image of an operation menu of a household electrical appliance (TV) to be watched is displayed in the field of view on the glasses-type display device 1. FIG. 12 is a sequence diagram for explaining a process of displaying an operation menu of the household electrical appliance (TV) to be watched on the glasses-type display device 1. In the example of FIG. 12, the operation flow information for displaying the image of the operation menu on the glasses-type display device 1 is transmitted from the RF tag 903T as a storage medium provided in the television 903 to be watched. Received by communication. Note that the storage medium provided in the watched television 903 is not limited to the RF tag as long as it has a function of wirelessly communicating with the glasses-type display device 1 and a function of storing information. . In addition, when the target TV 903 includes a storage medium other than the RF tag, for example, Bluetooth (registered trademark) is used for communication when the glasses-type display device 1 receives information from the storage medium included in the TV 903. ) And infrared communication such as infrared rays can be used.

In FIG. 12, when a user wearing the glasses-type display device 1 gazes at, for example, a television 903 as a device to be watched in the field of view (step 1), the gaze target distance detection unit 10 of the glasses-type display device 1 It is detected that the focus of the user's eyeball is locked on the television 903 (step 2). Then, the glasses-type display device 1 calculates (detects) the gaze target distance from the user's eye to the gaze target television (step 3), and the calculated gaze target distance information is the focal length adjustment unit 130. (Step 4). The focal length adjustment unit 130 that has received the gaze target distance information adjusts the focal length (step 5), and sends the adjustment completion information to the image generation unit 11 when the adjustment is completed (step 6). The image generation unit 11 includes an operation menu information to be displayed superimposed on the front view based on the adjustment completion information received from the focal length adjustment unit 130 and the operation flow information received from the RF tag 903T of the television 903. Image generation for generating a signal is performed (step 8). The image display unit 12 outputs an image including information on the operation menu (step 9). As a result, the image display unit 12 displays an image (virtual image) 904 including information on the operation menu in the forward field of view at the same focal position as the television 903, and the user has no defocus as shown in FIG. The operation menu image 904 can be visually recognized (steps 10 and 11).
Note that whether or not the user's line of sight has locked onto the television (gazing target) is determined on the basis of the time during which the user's eyeball stays on the television, as in the case of the tree described above. be able to.

  The user gazes and selects one of the operations on the operation menu image 904 displayed on the glasses-type display device 1, and performs a predetermined eye movement (for example, wink, vertical movement, or horizontal movement). . The glasses-type display device 1 transmits a command signal for the selected operation to the television 903 by short-range wireless communication based on the predetermined eye movement of the user. The television 903 performs operations (power on / off, channel switching, volume adjustment) corresponding to the operation selected by the user based on the command signal received from the glasses-type display device 1. Thereby, the user can operate the television 903 hands-free.

  In FIG. 11, the user wearing the glasses-type display device 1 may have other devices in view (for example, a television recording device 905, an audio device 906, speaker devices 907R and 907L, and lighting devices 908 and 909). Is displayed, an operation menu image of the device to be watched is displayed by the same processing as in FIG. Then, as in the case of the television 903, the user gazes and selects one of the operations on the operation menu image, and moves the eyes with a predetermined eye movement, so that the television recording device 905 and the audio device are moved. 906, the speaker devices 907R and 907L, and the illumination devices 908 and 909 can be operated hands-free.

  In the eyeglass-type display device 1 according to the present embodiment, the time information, environmental information such as temperature, humidity, and atmospheric pressure, the body temperature of the user, You may display health information, such as a pulse.

FIG. 13 is an explanatory diagram showing a state in which time and temperature are displayed. FIG. 14 is a sequence diagram for explaining processing for displaying time and temperature.
In FIG. 14, when a user wearing the glasses-type display device 1 gazes at, for example, a store as a gaze target (step 1), the gaze target distance detection unit 10 of the glasses-type display device 1 determines that the user's eyeball is focused. The gaze target distance detection unit 10 detects that the store is locked on (step 2). Then, in the glasses-type display device 1, the gaze target distance from the user's eye to the gaze target store is calculated (detected) (step 3), and the calculated gaze target distance information is the focal length adjustment unit 130. (Step 4). The focal length adjustment unit 130 that has received the gaze target distance information adjusts the focal length (step 5), and sends the adjustment completion information to the image generation unit 11 when the adjustment is completed (step 6). Based on the adjustment completion information received from the focal length adjustment unit 130 and the temperature information received from the temperature sensor 23, the image generation unit 11 generates an image signal of temperature and time information to be displayed superimposed on the front field of view. Image generation is performed (step 8). The image display unit 12 outputs an image including time information and temperature information (step 9). As a result, an image (virtual image) including time information and temperature information in the forward view is displayed by the image display unit 12 at the same position as the store 910, and the user is out of focus as shown in FIG. An image 911 of time and temperature without a mark can be visually recognized (steps 10 and 11). The time information can be acquired from, for example, a clock inside the image generation unit 11. Further, the temperature sensor 23 may be incorporated in the glasses-type display device 1 or may be provided separately from the glasses-type display device 1.

  Further, in the glasses-type display device 1 according to the present embodiment, the visual field tuning camera 9 is also used as the visual line direction detection unit, but the visual line direction in which the user is gazing in the forward visual field viewed by the user. A line-of-sight direction detecting means for more accurately detecting can be separately provided.

  FIG. 15 is a perspective view illustrating a configuration example of the eyeglass-type display device 1 in which a line-of-sight detection unit 24 as a line-of-sight direction detection unit is separately provided. FIG. 16A is a schematic configuration explanatory diagram showing one configuration example of the line-of-sight detection unit 24, and FIG. 16B is a schematic diagram of horizontal scanning output signals of the image sensor corresponding to each position of the eyeball. is there. FIG. 17 is a diagram showing an area obtained by dividing the front view interface viewed by the user through the right eyeglass lens 5R into a plurality of parts. The line-of-sight detection unit 24 detects the direction of the line of sight of the right eyeball in which the user who wears the glasses-type display device 1 is looking forward through the right eyeglass lens 5R.

  In FIG. 16A, the line-of-sight detection unit 24 is an image pickup composed of a light source 241 such as a light emitting diode that emits infrared light, a light projecting lens 242, a light receiving lens 243, and a solid-state image sensor such as a CCD or a CMOS. An image sensor 244 as a means and a line-of-sight calculation means 245 are provided.

  In the line-of-sight detection unit 24, the infrared light emitted from the light source 241 passes through the light projection lens 242, and then becomes substantially parallel light and irradiates the cornea of the right eye 100R. And the infrared light which passed the cornea irradiates an iris. Infrared light diffusely reflected on the surface of the cornea is guided onto the image sensor 244 via the light receiving lens 243 to form a cornea image. Similarly, infrared light diffusely reflected by the surface of the iris is guided onto the image sensor 244 via the light receiving lens 243 and forms an iris image. As shown in FIG. 16B, the output from the image sensor 244 has a remarkably higher potential at the corneal reflection image spot position where the corneal image is formed as compared with the others. The line-of-sight calculation means 245 calculates the rotation angle from the center line of the eyeball when the right eye 100R faces straight forward based on the corneal reflection image spot position. Then, based on the calculated rotation angle, the visual axis of the right eye 100R is obtained, and the relative gaze direction of the user in the front view is detected. Then, based on the relative line-of-sight direction in the forward visual field obtained at this time, the gazing point on which the user is gazing on the forward visual interface viewed through the spectacle lens 5R shown in FIG. 17, that is, the area A1 constituting the forward visual interface. The area | region where the gaze target which the user is looking at among A12 can be specified.

  By providing the line-of-sight detection unit 24, the line-of-sight direction of the user in the front field of view can be detected more accurately, and the gaze target being watched by the user can be more accurately identified. Accordingly, it is possible to prevent a malfunction in which a gaze target different from the gaze target that the user is gazing at for the gaze target is specified.

  In addition, when the gaze detection part 24 which detects more accurately the gaze direction which the user is gazing in the front view which the user is seeing as shown in FIGS. 15-17, the gaze detection is provided. You may comprise so that the imaging direction of the visual field synchronization camera 9 may face the user's gaze direction detected by the part 24. FIG. More specifically, based on the detection result of the camera driving mechanism and the line-of-sight detection unit 24 as the imaging direction variable means for changing the imaging direction of the field-of-view synchronization camera 9, the imaging direction of the field-of-view camera 9 in the user's line-of-sight direction. Imaging control means for controlling the camera driving mechanism so as to face the camera. The camera drive mechanism includes, for example, a camera holding unit that rotatably holds the field-of-view tuning camera 9 so that the imaging direction can be changed, a micromotor that can perform on / off control and forward / reverse rotation control, It can be configured using a drive transmission unit that converts the rotation into the rotation of the camera holding unit of the visual field synchronization camera 9. In this way, by configuring the imaging direction of the visual field tuning camera 9 to face the user's line-of-sight detected by the line-of-sight detection unit 24, an image of the field of view is centered on the line of sight that the user is gazing at. An image can be taken, and the gaze target is located in the center of the taken image. Further, even when the captured image is enlarged so that the gaze target can be easily specified, it is difficult for the gaze target to protrude from the enlarged image. Therefore, the gaze target can be specified with higher accuracy.

  Further, in the eyeglass-type display device 1 of the present embodiment, by providing the line-of-sight detection unit 24, an image such as an icon or the like for input or selection displayed superimposed on the front field of view viewed by the user. It is also possible to make selections.

  FIG. 18 is an explanatory diagram in which a line-of-sight cursor 920 that displays the line-of-sight direction and four icons for instructing input and selection are displayed in the front view including the store that can be seen through the eyeglass lens 5R. The line-of-sight cursor 920 moves with the movement of the user's line of sight, and is displayed at the position of the user's point of gaze. In the illustrated example, as the above icons, a mail icon 921 for confirming mail information addressed to the user, a schedule icon 922 for confirming the user's schedule information, and the latest information as distribution information distributed from the server. A news icon 923 for confirming news and a mode selection icon 924 for selecting a display mode are displayed. Here, for example, when the user moves the line of sight of the right eye and moves the line-of-sight cursor 920 to the mail icon 921 and blinks as a selection operation, the mail icon 921 is selected and is sent from the mail server on the communication network. The received new mail addressed to the user is displayed on the upper right side of the front view as shown in the figure. Similarly, when the line-of-sight cursor 920 is placed on any of the schedule icon 922, the news icon 923, and the mode selection icon 924 and blinking (wink), the icon combined with the line-of-sight cursor 920 is selected, and the upper right side of the forward view In addition, the user's schedule or the latest news image corresponding to the selected icon is displayed, or a submenu screen for selecting a display mode is displayed.

  It should be noted that the user's eye movement when selecting the icon or the like may be blinking (wink) or may be blinking (winking) a plurality of times (for example, twice). Further, it may be a movement of the eye that moves in a predetermined direction of the front view.

  The display modes that can be selected by the mode selection icon 924 include, for example, the first display mode in which the image formation position of information in the front field of view is adjusted according to the above-described gaze target distance, and the above-described gaze. This is a second display mode in which the image forming position of information in the front view is fixed and displayed regardless of the target distance.

  The display mode that can be selected by the mode selection icon 924 may be a plurality of display modes that are set according to the type of related information that is superimposed and displayed in the front field of view or the type of gaze target. For example, when the glasses-type display device 1 is used in a zoo or amusement park, an amusement navigation mode specialized in acquiring and displaying related information related to the target animal or facility to be watched, or the glasses-type display device 1 The tour guide mode specialized in the acquisition and display of related information related to the tourist facility to be watched may be used. In addition, when the glasses-type display device 1 is used in front of a store or a show window, the shopping navigation mode specialized for obtaining and displaying related information related to the product to be watched and the glasses-type display device 1 are mounted. A route navigation mode specialized in obtaining and displaying information such as route guidance to a destination and a map when walking in an urban area or traveling on a road may be used.

  In the glasses-type display device 1 according to the present embodiment, a GPS function for acquiring information on the current position of the glasses-type display device 1 and an orientation detection function for detecting the orientation of the field of view as viewed by the user. It may be provided. Information on the current position acquired by the GPS function and information on the azimuth of the field of view detected by the azimuth detection function are collated with information in a spatial database including map information, facility information, etc. provided in the server, for example. As a result, the gaze target can be estimated, and the gaze target can be more accurately specified. Therefore, the accuracy of the content of the related information transmitted to the glasses-type display device is increased. In addition, the current position information and the visual field direction information detected by the direction detection function include the above-described visual field image information, the detection result of the visual line direction, and the detection result of the posture change of the eyeglass-type display device 1. By using it for identification, it becomes possible to identify the gaze target more accurately.

  FIG. 19 is a functional block diagram of the eyeglass-type display device 1 further including a GPS receiving unit 26 and an azimuth detecting unit 27 in addition to the line-of-sight detection unit 24 described above. The GPS receiver 26 functions as a current position acquisition unit that receives a GPS signal from a GPS satellite and acquires information (latitude, longitude, altitude) of the current position where the glasses-type display device 1 is located. In addition, the direction detection unit 27 is configured by, for example, a uniaxial, biaxial, or triaxial geomagnetic sensor, and which direction (north, northeast,... It functions as an azimuth detecting means for detecting whether or not

  In the glasses-type display device 1 having the configuration of FIG. 19, the current position information acquired by the GPS receiving unit 26, the azimuth information acquired by the azimuth detecting unit 27, and the posture change and gravity direction acquired by the posture change detecting unit 28. This information may be transmitted together with the image data of the gaze target image to the information providing server. In this case, it is possible to more accurately identify the gaze target even for stores and tourist facilities that are difficult to identify with images alone. The current position information, azimuth information, and information on the direction of movement and gravity are used for navigation support when the eyeglass-type display device 1 in the route navigation mode is attached and the user walks on a city street or travels on a road. It may be sent to the server. In this case, more accurate information about route guidance to a destination, a map, and the like can be acquired. Then, based on information such as route guidance and a map received from the navigation support server, a route guidance navigation arrow is overlapped and displayed in the front field of view through the spectacle lens 5R to guide the route to the destination. Is possible.

  In the above embodiment, the gaze target distance detection unit 10 serving as the gaze target distance detection unit detects the gaze target distance using a method of detecting the thickness of the crystalline lens. However, the gaze target distance is determined based on the degree of convergence of both eyes. Can also be detected.

  FIG. 20 is an explanatory diagram of a method for detecting a distance to a gaze target based on convergence of both eyes. This method utilizes the fact that the visual axes of both eyes are parallel when looking far, but the eyes are closer to each other (convergence) and the distance between the centers of the pupils is narrower when looking closer. is there. In FIG. 20, the distance W1 between the pupil centers of both eyes 100R and 100L when looking at infinity and the distance W2 between the pupil centers when looking near the front of 50 centimeters, for example, are measured in advance. Stored in the memory 15. Since the distance to the gaze target and the distance between the pupil centers are substantially proportional, the distance to the gaze target can be calculated based on the displacement of the distance between the pupil centers by calculating the proportionality constant.

  In the above embodiment, the gaze target distance detecting means for detecting the gaze target distance irradiates the gaze target with infrared rays, ultrasonic waves, laser light, or the like, and the distance depending on the time or irradiation angle until the reflected wave returns. An active distance measuring method for detecting the image may be used, or a passive distance measuring method for performing distance measurement using an image captured by a lens may be used.

As described above, according to the present embodiment, the user's line-of-sight direction in the field of view captured by the field-of-view tuning camera 9 is detected, and the detection result of the posture change of the glasses-type display device 1 detected by the posture change detection unit 28 is obtained. Based on this, the user's line-of-sight direction can be quickly corrected. Thereby, it is possible to improve the accuracy when specifying the gaze target that the user is gazing in the visual field image captured by the visual field imaging means, and more reliably and in real time related information related to the gaze target. Can be displayed.
In addition, according to the present embodiment, unlike the eyeglass device described in Patent Document 1, the object is not limited to the face of a person, and an arbitrary object in the user's field of view is watched. With this simple operation, it is possible to identify a target object and confirm related information without being noticed by the surroundings. Further, according to the present embodiment, it is not necessary to perform pre-processing before eye-gaze detection (processing for detecting a person's face from an image taken by an imaging unit), which is essential in the eyeglass device of Patent Document 1.
Further, according to the present embodiment, it is possible to suppress an increase in eye fatigue without causing a sense of incongruity when viewing a gaze target in the front field of view and information to be displayed superimposed on the front field of view together.

  In the above embodiment, the configuration of the retinal scanning image display method and the image display method using the combiner optical system has been described. However, the present invention is not limited to this method, and other image display methods may be adopted. Good.

DESCRIPTION OF SYMBOLS 1 Glasses type display apparatus 2 Rim 8 Power switch 9 Field of view tuned camera 10 Gaze target distance detection part 11 Image generation part 12 Image display part 13 Display control part 14 Control part 15 Memory 16 Battery 18 Portable communication terminal 19 Mobile communication network 20 Internet Communication network 21 Image analysis server 22 Information providing server 24 Gaze detection unit 25 Communication unit 28 Attitude change detection unit 130 Focal length adjustment unit

JP 2010-061265 A

Claims (10)

A glasses-type display device capable of displaying information in a field of view,
A visual field imaging means for imaging the visual field;
Gaze direction detection means for detecting the gaze direction of the user in the field of view;
Attitude change detection means for detecting an attitude change of the glasses-type display device;
Based on the visual field image information captured by the visual field imaging unit, the detection result of the visual line direction detected by the visual line direction detection unit, and the detection result of the posture change detected by the posture change detection unit, Gaze target identifying means for identifying a gaze target that the user is gazing in the field of view;
Related information acquisition means for acquiring related information related to the gaze target identified by the gaze target identification means;
Display means for displaying the related information acquired by the related information acquiring means in the field of view;
An eyeglass-type display device comprising: The eyeglass-type display device according to claim 1,
The line-of-sight direction detection means includes user image pickup means for picking up an image of the user's eyes, and detects the user's line-of-sight direction based on an image of the user's eyes picked up by the user image pickup means. An eyeglass-type display device. The glasses-type display device according to claim 1 or 2,
An imaging direction variable means for changing an imaging direction of the visual field imaging means;
Based on at least one of the detection result of the line-of-sight direction detection unit and the detection result of the posture change detection unit, the imaging direction variable unit is controlled so that the imaging direction of the visual field imaging unit is directed to the line-of-sight direction of the user. Imaging control means;
An eyeglass-type display device further comprising: The glasses-type display device according to any one of claims 1 to 3,
Azimuth detection means for detecting the azimuth of the field of view, and current position information acquisition means for acquiring current position information of the glasses-type display device,
The eyeglass-type display device, wherein the gaze target specifying unit further uses at least one of the detection result of the orientation and the current position information for specifying the gaze target. The glasses-type display device according to any one of claims 1 to 4,
Based on at least one of the image information of the field of view captured by the field-of-view imaging unit, the detection result of the line-of-sight direction detection unit, and the detection result of the posture change detection unit, the position of the gaze target in the field of view and the An eyeglass-type display device, further comprising display control means for controlling the display position of the related information so that the display position of the related information has a predetermined positional relationship. The glasses-type display device according to any one of claims 1 to 4,
And further comprising display control means for controlling the display of the related information to stop when the shift between the position of the gaze target in the field of view and the display position of the related information is larger than a predetermined range. An eyeglass-type display device. The glasses-type display device according to any one of claims 1 to 4,
Gaze target distance detection means for detecting a gaze target distance between the gaze target and the user's eyes in the field of view;
An eyeglass-type display device further comprising display control means for controlling a display position of the related information based on a detection result of the gaze target distance detection means. The eyeglass-type display device according to any one of claims 1 to 7,
An eyeglass-type display device further comprising means for receiving and acquiring information stored in a storage medium provided for the gaze target by short-range wireless communication. The eyeglass-type display device according to any one of claims 1 to 8,
A communication means for communicating with the server via the communication network;
The related information acquisition means transmits at least one information of the image information of the gaze target, the feature extraction information of the gaze target, and the information acquired from the storage medium provided in the gaze target to the server, A glasses-type display device that receives and acquires related information related to the gaze target searched by a server from the server. A server capable of communicating with the glasses-type display device according to claim 9 via a communication network,
Information receiving means for receiving at least one of the image information of the gaze target, the feature extraction information of the gaze target, and the information acquired from the storage medium provided in the gaze target from the glasses-type display device;
Search means for identifying the gaze target based on information received from the glasses-type display device, and searching for related information related to the identified gaze target;
Information transmission means for transmitting related information related to the gaze target searched by the search means to the glasses-type display device;
A server characterized by comprising:

Images