JP7114564B2 - head mounted display device - Google Patents

Description

The present invention relates to a head mounted display device.

Second information (hereinafter referred to as secondary information) related to the first information (hereinafter referred to as primary information) displayed on the main display device is displayed on the head-mounted display device (head mounted display, Head A technique for displaying on a mounted display (hereafter referred to as HMD) has been devised.

As a background art of this technical field, there is Japanese Patent Laying-Open No. 2001-215920 (Patent Document 1). In Patent Document 1, when the wearer of the HMD browses the primary information displayed on the screen of the main display device, according to the direction of the wearer's head and the direction of the line of sight detected by the HMD, the primary A technique for displaying secondary information related to information on an HMD is described. This technology reduces the amount of information (primary information) displayed on the main display device, avoiding an increase in the size of the device, and allows the wearer to confirm the necessary information (secondary information) simply by moving their head or eye movement. Therefore, the operability of the device can be improved.

Further, there is Japanese Patent Laying-Open No. 2010-237522 (Patent Document 2). In Patent Document 2, when a wearer of an HMD views primary information projected on a large screen, secondary information is displayed at an appropriate position adjacent to the primary information based on the positional relationship between the wearer's seat and the screen. talking about technology. With this technology, it is possible to display subtitles (secondary information) in the native language so that only the person wearing the HMD can see them when watching a movie (primary information).

Japanese Patent Application Laid-Open No. 2001-215920 JP 2010-237522 A

Patent Literature 1 describes displaying secondary information of information present at the point of gaze of the wearer detected by the HMD. However, the gaze point detected by the HMD indicates the position in the wearer's field of view. In other words, this field of view includes the surrounding scenery in addition to the primary information displayed by the main display device, and since the wearer is not necessarily in front of the main display device, the primary view depends on the standing position of the wearer. Information comes in different forms. Therefore, in order to know where the wearer is gazing within the primary information, it is necessary to convert the point of gaze detected by the HMD into coordinates within the primary information by some means. No means are disclosed.

In Patent Document 2, coordinate conversion is performed based on the positional relationship between the seat of the wearer and the screen, and the display position of the secondary information on the HMD is calculated. It cannot be used when browsing primary information from a position.

In addition, both of Patent Documents 1 and 2 display secondary information only while primary information is being viewed, so the secondary information cannot be viewed when the wearer takes his or her eyes off the primary information.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems. Disclosed is a means for displaying a

In order to solve the above problems, the present invention provides, for example, a head-mounted display device capable of displaying secondary information related to primary information displayed on a display device,
a display unit that displays the secondary information viewable by the wearer of the head-mounted display device;
a communication unit communicable with the display device;
an imaging unit that captures a camera image in the direction in which the wearer is facing;
a point-of-regard detection unit that detects the point-of-regard of the wearer with respect to the camera image;
a gaze point calculation unit that calculates position information of the wearer's gaze point with respect to the primary information based on the detected gaze point,
The point-of-gazing detection unit determines that the positional information of the point-of-gazing of the wearer with respect to the primary information is in the direction of the primary information, and determines that it is not in the direction of the primary information. display information,
A head-mounted display device that changes and displays the time until the secondary information is erased according to each attribute of the secondary information when it is determined that the position information is not in the direction of the primary information. , configured as

According to the present invention, appropriate secondary information can be selected and displayed regardless of the position or line-of-sight direction of the HMD wearer. There is an effect that information can be browsed.

FIG. 2 is a diagram for explaining an overview of the operation of the HMD-linked display system according to the first embodiment; 1 is an overall configuration diagram of an HMD-linked display system in Example 1. FIG. 4 is a configuration diagram of a primary information database in Example 1. FIG. 4 is a configuration diagram of a secondary information database in Example 1. FIG. FIG. 10 is a diagram for explaining an operation of selecting primary information according to the first embodiment; 5 is a flowchart of primary information selection processing according to the first embodiment. 4A and 4B are diagrams for explaining a camera image and a fixation point in the camera image according to the first embodiment; FIG. 10 is a flowchart of secondary information selection processing based on a gaze point according to the first embodiment; 4 is a flow chart of calculation processing of a gaze point in primary information in Embodiment 1. FIG. 4 is a diagram for explaining an overview of projective transformation in Example 1. FIG. FIG. 10 is another diagram for explaining the overview of projective transformation in the first embodiment; 5 is a flowchart of secondary information erasing processing according to the first embodiment. FIG. 10 is a diagram for explaining an operation outline of selecting secondary information by voice in the first embodiment; 10 is a flowchart of selection processing of secondary information by voice according to the first embodiment; FIG. 11 is a diagram for explaining an overview of the operation of the HMD-linked display system according to the second embodiment; FIG. 11 is a diagram showing an overview of an HMD-linked display system in Example 2; FIG. 11 is an overall configuration diagram of an HMD-linked display system in Example 2; FIG. 10 is a diagram illustrating the configuration of a secondary information database in Example 2; FIG. 10 is a diagram for explaining a camera image and a fixation point in the camera image in Example 2; 10 is a flow chart of secondary information selection processing based on a gaze point in Example 2. FIG.

Embodiments of the present invention will be described below with reference to the drawings.

This embodiment will explain an example in which supplementary information (secondary information) related to educational content (primary information) projected onto a screen by a projector is displayed on an HMD worn by a teacher in an educational setting. According to this embodiment, the teacher can browse the supplementary information related to the educational content while teaching the students in a natural manner.

FIG. 1 is a diagram for explaining the outline of the operation of the HMD-linked display system in this embodiment. In FIG. 1, the projection device 100 projects educational content (eg, a world map) onto the screen 930 . A teacher 910 teaches while alternately viewing content projected via the HMD 300 and a student 920 . At this time, the scene that the teacher 910 sees through the HMD 300 initially consists of an educational content (world map) and buttons for operating the display of the content (“Previous”, “Next” in this example), as shown on a screen 351 . ). Next, when the teacher 910 gazes at the vicinity of Greenland on the world map, as shown in a screen 352, supplementary information (for example, country name, capital city, official language) is displayed. Next, when the teacher 910 looks in the direction of the student 920, as shown in screen 353, the buttons for manipulating the content display are erased, while the supplemental information continues to be displayed even if the world map is not being viewed. After that, after a certain period of time has passed, the supplementary information is also erased as shown in screen 354 . With such an operation, for example, the lesson can be conducted without performing an unnatural operation such as reviewing the direction of the educational content one by one to obtain supplementary information and gazing at the corresponding part.

FIG. 2 is an overall configuration diagram of the HMD-linked display system in this embodiment. In FIG. 2, this system comprises a projection device 100, a display device 200, and an HMD 300. FIG. The projection device 100 and the display device 200, and the display device 200 and the HMD 300 are connected by communication.

The projection device 100 comprises a signal input section 110 for inputting primary information to be displayed, a control section 120 for controlling display, and a display section 130 for projecting the primary information on the screen.

The display device 200 includes a recording unit 210 that stores a primary information database 510 and a secondary information database 520, a control unit 220 that performs various processes such as output of primary information and secondary information, and a signal that outputs primary information to the projection device 100. An output unit 230, a communication unit 240 that communicates with the HMD 300, and a point-of-regard detection unit that detects the position of the point-of-regard in primary information based on information acquired from the HMD 300. It is composed of a gaze point calculation unit 250 that calculates and calculates, a voice recognition unit 260 that recognizes the voice of the HMD wearer or the like, an operation unit 270 for operating the display device 200 , and a display unit 280 . Note that the point-of-regard calculation unit 250 and the speech recognition unit 260 may be configured to be realized as dedicated hardware, or may be configured to be realized as software modules executed by the control unit 220 . Also, the point-of-regard calculation unit 250 may be provided in the HMD 300 .

The HMD 300 includes an imaging unit 310 that captures an image of the scenery in the direction the wearer is looking, and a gaze detection unit that detects the gaze direction of the wearer in the captured camera image. A detection unit 320, a voice acquisition unit 330 that acquires the voice of the HMD wearer, etc., and a control unit that performs various control processes such as transmitting the acquired camera image 540, gaze point 550 in the camera image, and voice data 560 to the display device 200. 340, a communication unit 350 that communicates with the display device 200, and a display unit 360 that displays secondary information 570 acquired from the display device as an image that the wearer can view or appreciate.

In this embodiment, the projection device 100 corresponds to a projector, and the display device 200 corresponds to a PC (Personal Computer) connected to the projector. It can be applied to a general display device or to a dedicated device in which the projection device 100 and the display device 200 are integrated. Further, the HMD 300 may be divided into a device that is worn on the head and mainly displays, and a device that is worn on the waist and the like and mainly controls the HMD.

FIG. 3 is a structural diagram of the primary information database 510 in this embodiment. 3, the primary information database 510 comprises a primary information identifier 511, a primary information name 512, a file name 513 storing the primary information, and a display flag 514 indicating whether or not the primary information is being displayed. be done. It should be noted that the on-display flag 514 is '1' during display, and '0' when non-display.

The upper diagram of FIG. 4 is a structural diagram of the secondary information database 520 in this embodiment. In the upper diagram of FIG. 4, the secondary information database 520 includes a primary information identifier 521 for identifying related primary information, a gaze point range 522 for selecting secondary information based on the HMD wearer's gaze point, and a secondary information based on voice. It consists of a keyword 523 for selecting the next information, secondary information 524, and attributes 525 of the secondary information. As shown in the map shown in the lower part of FIG. 4, the coordinate system of the primary information in this embodiment is defined by (0, 0) on the upper left and (1920, 1080) on the lower right. The point-of-regard range (1680, 70) to (1880, 250) on the first line of the information database indicates the vicinity of Greenland, and the point-of-regard range (700, 720) to (1000, 870) on the second line indicates the vicinity of Australia. means The gazing point range (0, 0) to (1920, 1080) in the 3rd and 4th rows of the secondary information database is always secondary information (in the example, " "previous" and "next" buttons).

The above is the configuration of the HMD-linked display system of the present embodiment. The present embodiment will be described below along the flow of operation of this system.

FIG. 5 is a diagram for explaining selection operation of educational content (primary information) to be projected on the screen 930 displayed on the display unit 280 of the display device in this embodiment. In FIG. 5, a screen 281 is an educational content selection screen, and displays a plurality of icons including an icon 282 for displaying a world map. If the icon 282 is selected here, a world map 283 is displayed as shown in the lower part of FIG. In addition, on the world map 283, diagonally-lined markers 284 are displayed at the four corners. This is identification information for specifying the display area of the primary information, and the details will be described later.

FIG. 6 is a flowchart of processing of the control unit 220 when primary information is selected in this embodiment. In FIG. 6, the control unit 220 reads a list of primary information from the primary information database 510 and displays it on the selection screen 281 consisting of icons representing each content (step S2201). Next, in step S2202, it waits until the user selects content. Next, the file 513 corresponding to the primary information selected by the user is read and projected onto the screen 930 via the signal output unit 230 and the projection device 100 (step S2203). Finally, the display flag 514 of the selected primary information is set to 1, and the process ends (step S2204).

7 is obtained by the imaging unit 310 of the HMD 300 and the in-camera-image gaze point detection unit 320 when the teacher 910 views the screen 930 on which the educational content is projected in step S2203 of FIG. A camera image 540 and a gaze point 550 within the camera image are shown. The figure shows teacher 910 viewing primary information from a position slightly to the right of screen 930, and gazing at the vicinity of Greenland.

FIG. 8 shows secondary information selection processing that is executed in the control unit 220 of the display device 200 when the camera image 540 and the in-camera-image fixation point 550 shown in FIG. 7 are received from the HMD 300 in this embodiment. is a flow chart. In FIG. 8, the control unit 220 first refers to the primary information database 510 and determines whether there is any primary information whose display flag is set to 1 (step S2211). It should be noted that the on-display flag in the first row (world map) of the primary information database 510 is set to 1 by the operation shown in FIG. Next, it is determined whether or not the gazing point 550 in the camera image has remained the same for a certain period of time, that is, whether or not the gazing point of the teacher 910 has stayed at a specific place (step S2212). This determination can be realized, for example, by determining whether or not the state in which the distance difference from the point of interest in the camera image received last time is less than a predetermined threshold has continued for a certain period of time. Next, the gaze point calculation unit 250 is used to calculate the gaze point in the primary information (step S2213). The details of this processing will be described later.

Next, it is determined whether or not the calculation of the gazing point in the primary information has succeeded (step S2214). If the calculation of the point of regard within the primary information is successful, that is, if the point of regard of the teacher 910 is in the direction of the primary information (that is, toward the screen 930), the point of regard within the primary information calculated in step S2213 is stored ( step S2215). Next, the secondary information database 520 is referred to, and among the secondary information related to the primary information being displayed, the gazing point in the primary information stored in step S2213 is within the gazing point range 522 of the secondary information database 520. The secondary information present in the file is displayed (step S2216). On the other hand, if the calculation of the gaze point within the primary information fails in step S2214, that is, if the gaze is outside the marker range of the primary information, the secondary information corresponding to the gaze point within the primary information stored in step S2215 (ie, the currently displayed secondary information), the erasing timer is set according to the attribute 525 of the secondary information (step S2217). The erasure timer is the time until the secondary information displayed on the HMD 300 is erased. For example, if the attribute 525 of the secondary information is text, it is erased after 60 seconds, and if the attribute 525 of the secondary information is a button, it is erased after 60 seconds. set to clear after 0 seconds (ie immediately). As a result, when the teacher 910 looks in the direction of the student 920, the button disappears immediately, but the text continues to be displayed for a certain period of time. If the calculation of the gaze point in the primary information fails at the very beginning of this secondary information selection flow, there is no stored gaze point in the primary information, so the erasure timer is not set. In this way, depending on whether it is determined that the position information of the wearer's gaze point is in the primary information or not in the primary information, the display method, such as whether to continue displaying the secondary information or immediately erase it, is determined. change. In other words, the display layout or display menu may be changed.

FIG. 9 is a flow chart showing the process of calculating the gaze point in the primary information using the gaze point calculator 250, which is executed in step S2213 described above. The point-of-regard calculation unit 250 first detects the markers 284 attached to the four corners of the primary information (step S2501). Next, it is determined whether or not the marker has been detected (step S2502). If the marker cannot be detected, the calculation is terminated as a failure, and if the marker is detected, the coordinates of the gaze point in the primary information are calculated by projective transformation (step S2503), and the process is terminated.

FIG. 10 is a diagram for explaining an overview of the projective transformation executed in step S2503 described above. By the calculation described below, the coordinates of the gaze point in the coordinate system 251 in the camera image can be converted into the coordinates in the coordinate system 252 in the primary information.

In FIG. 10, let the coordinate system 251 in the camera image be a plane in which the upper left is (0, 0) and the lower right is (100, 100). On the other hand, let the coordinate system 252 in the primary information be a plane in which the upper left is (0, 0) and the lower right is (1920, 1080). Here, the primary information area 253 in the camera image specified by the four corner markers detected in step S2501 is transformed into an area in the coordinate system in the primary information. There are various calculation formulas for this transformation, but in this embodiment, a general projective transformation formula 254 is used. Here, (x, y) are the coordinates before conversion (coordinate system 251 in the camera image), and (u, v) are the coordinates after conversion (coordinate system 252 in the primary information). The projective transformation equation 254 has eight unknowns (a1, b1, c1, a2, b2, c2, a0, b0). Thus, the unknowns can be derived by substituting four points whose corresponding coordinates are known in both coordinate systems to obtain eight equations. The coordinate correspondence table 255 shows the correspondence between the coordinates (x, y) of the four markers detected in step S2501 in FIG. 9 and the converted coordinates (u, v). Top left coordinates (10,20) to (0,0), top right coordinates (70,18) to (1920,0), bottom left coordinates (12,80) to (0,1080), bottom right coordinates (65, 82) correspond to (1920, 1080), respectively. By substituting these values into the projection transformation equation 254, a simultaneous equation consisting of eight equations is obtained. The coordinate transformation result 257 of the gaze point indicates that (60, 28) is transformed into (1635, 148) when the calculation is performed according to the projection transformation equation 254 using the calculation result 256 of the unknown number. , where the calculation results are rounded to integers). Through the above calculations, the gaze point in the coordinate system of the camera image is converted into coordinates in the coordinate system of the primary information.

In the present embodiment, it is assumed that diagonally patterned markers are displayed at the four corners of the primary information as in the examples shown in FIGS. , but various other techniques are available for such markers. For example, a pattern other than oblique lines may be used, or a method of embedding a physical device for area detection on the screen 930 side instead of displaying markers in the first place may be used. Also, an invisible marker using an infrared camera or the like may be used instead of a visible pattern that can be seen by humans.

Also, in FIG. 10, an example in which projective transformation is performed using the coordinates of the four corners of the primary information area has been described, but the method of projective transformation is not limited to this. For example, FIG. 11 is a conceptual diagram showing another method of projective transformation. When the HMD wearer views the primary information, if he or she stands close to the screen 930, the four corner markers may not fit in the camera image, as in the camera image 541 shown in FIG. In such a case, instead of using the correspondence of the coordinates of the four corners of the area of the primary information in the coordinate system 258 in the camera image and the coordinate system 259 in the primary information, the coordinates of the four corners of one marker are used. The coordinate transformation of the point of gaze can be performed by obtaining the unknowns of the equation 254 of the projective transformation. In addition, instead of using the coordinates of four predetermined points, image recognition is performed on both the camera image and the primary information, and four characteristic points (feature points) are dynamically extracted. , the unknowns of the projective transformation can be derived based on these four points. Such a method is widely used, for example, in processing for recognizing a human face from various angles.

Finally, FIG. 12 is a flow chart showing processing for erasing secondary information, which is activated when the erasing timer set in step S2217 in FIG. 8 reaches a predetermined time. In FIG. 12, when the erasure timer reaches a predetermined time, the control unit 220 erases the secondary information displayed on the HMD (step S2221). By changing the value of the erase timer according to the attribute of the secondary information as described above, when the student 920 looks away from the screen 930, the button is immediately erased, but the text (supplementary information ) continues to be displayed.

The operation shown in FIG. 1 becomes possible by the contents described above. That is, the scene when the teacher 910 looks at the screen 930 through the HMD 300 is the screen 351 in FIG. 1 at first. Next, when the vicinity of Greenland on the world map is observed, the screen 352 in FIG. 1 is displayed by the processing up to step S2216 in FIG. Next, when looking in the direction of the student 920, the screen changes to the screen 353 in FIG. 1 and text (supplementary information) continues to be displayed until the time of the erasing timer set in step S2217 in FIG. 8 elapses. Finally, when the erasing timer set for the text elapses, the screen 354 shown in FIG. 1 appears.

In step S2217 of FIG. 8, in addition to setting the erasing timer, the transmittance of the secondary information displayed on the HMD may be changed according to the attribute 525 of the secondary information. For example, when the teacher 910 is looking in the direction of the screen 930, as shown in the screen 352 of FIG. 1, the supplemental information may be displayed with a transmittance of 50%, that is, semi-transparently, as exemplified on the screen 353 of FIG. As a result, it is possible to avoid losing sight of the student 920 by displaying the supplementary information.

Next, as another operation of the present embodiment, an example of the operation of displaying secondary information by voice even when primary information is not browsed will be described.

13A and 13B are diagrams for explaining an outline of the operation of displaying secondary information by voice in this embodiment. In FIG. 13, following the operation described so far, the teacher 910 is looking in the direction of the student 920, the student 920 asks "How many people live here?" ” is shown. At this time, the scene that the teacher 910 sees through the HMD 300 is, at first, only the students can see as shown on the screen 355, and then, when they say "Population is...", supplementary information (for example, the country name) is shown on the screen 356. , population, population density). Note that supplementary information may be displayed in response to the speech from the student. Also, at that time, it is possible to capture the speech, display the keyword, and display the supplementary information by paying attention to the keyword.

FIG. 14 shows how the speech acquisition unit of the HMD 300 acquires the speech of the teacher 910 and selects the secondary information executed when the control unit 220 of the display device 200 receives the speech data 560 of the teacher 910 in this embodiment. It is a flow chart of processing to do. In FIG. 14, similarly to FIG. 8, the control unit 220 first refers to the primary information database 510 and determines whether there is any primary information whose display flag is set to 1 (step S2311). Next, voice recognition processing is performed on the received voice data 560 (step S2313). Note that this speech recognition processing is not limited to the method executed inside the display device 200, and may be executed by communicating with a server that performs speech recognition via the Internet or the like. Further, the trigger of the speech recognition process may be always performed at a predetermined cycle, or may be performed when a predetermined button is pressed. Next, it is determined whether the processing of converting the voice data 560 into text by voice recognition has succeeded (step S2314). Note that this determination may be made simply based on whether voice recognition can be executed, or based on the reliability of the conversion result output by a general voice recognition technology. Also, by identifying the utterer by using a technique such as voiceprint analysis, the utterance of the student 920 may be ignored and only the utterance of the teacher 910 may be judged to be successful. Next, if the speech-to-text conversion is successful, the secondary information database 520 is referred to, and among the secondary information related to the currently displayed primary information, the primary information stored in step S2215 in FIG. Display secondary information in which the gaze point exists within the range of gaze point range 522 in the secondary information database, and the text after conversion includes the words shown in the keyword 523 of the secondary information database 520 ( step S2315). In the example of the secondary information database shown in FIG. 4, the secondary information on the 5th line in which "population" is set as the keyword is displayed.

In this example, secondary information is displayed immediately when a specific keyword is included in the utterance content. , and the secondary information may be displayed by selecting the button. Further, even if the utterance content does not completely match the keyword, secondary information may be displayed when the utterance content resembles the keyword by determining the degree of similarity of the character strings.

By the above processing, the operation shown in FIG. 13 becomes possible. That is, the scene when the teacher 910 looks at the student 920 through the HMD 300 is initially the screen 355 in FIG. Next, when the teacher 910 utters "Population is...", the screen 356 in FIG. 1 is displayed by the processing up to step S2315 in FIG.

As described above, in this embodiment, from a camera image acquired from the HMD and the gaze point of the wearer of the HMD with respect to the camera image, the position information of the gaze point of the wearer with respect to the primary information is calculated, and the position information related to the position information is calculated. The secondary information is selected and displayed, and the display method of the secondary information to be displayed on the HMD is changed depending on whether the HMD wearer is viewing the primary information or not. As a result, the teacher can give supplementary information to the educational content projected on the screen while teaching the students in a natural manner.

In other words, this embodiment is a system including a display device and a head-mounted display, wherein the display device includes a first display unit capable of displaying an image or a projection unit capable of projecting an image, and the head-mounted display The head mounted display includes a first communication unit capable of communicating with the head mounted display, a second display unit displaying an image viewable by the wearer of the head mounted display, and a line-of-sight direction of the wearer of the head mounted display. a line-of-sight detection unit, and a second communication unit communicable with the display device, based on information transmitted and received via the first communication unit and the second communication unit, the first display unit of the display device either the display device or the head mounted display is provided with a gaze point detection unit for detecting the gaze point position of the line of sight of the wearer of the head mounted display with respect to the image displayed on the display or the image projected by the projection unit, The detection unit calculates position information in the image corresponding to the position of the point of gaze when the point of gaze is on the image, and the head-mounted display generates related data related to the target data displayed at the calculated position in the image. is obtained via communication with the display device or other communication and configured to be displayed on the second display.

Further, in a head-mounted display cooperation display system including a display device and a head-mounted display, the display device includes a first display unit for displaying primary information, a projection unit for projecting primary information, or a signal for outputting an image signal. An output unit, a first communication unit capable of communicating with a head-mounted display, and a point-of-regard calculation unit that calculates position information of a point-of-regard of a wearer of the head-mounted display with respect to primary information, wherein the head-mounted display a second display unit that displays secondary information that can be viewed by the wearer; a second communication unit that can communicate with the display device; an imaging unit that captures a camera image in the direction the wearer is facing; Equipped with a point-of-regard detection unit in a camera image that detects the point-of-regard of the wearer, the display device calculates the point-of-regard calculation unit based on the detected point-of-regard transmitted and received via the first communication unit and the second communication unit. The position information of the gaze point of the wearer with respect to the primary information is calculated, and the secondary information related to the position information is selected. is configured to change the display method of the secondary information on the display section of the

As a result, appropriate secondary information can be selected and displayed regardless of the position or line-of-sight direction of the HMD wearer, increasing the freedom of action of the HMD wearer and viewing secondary information in a more natural manner. It has the effect of being able to

This embodiment will explain an example in which supplementary information (secondary information) related to broadcast content (primary information) displayed on a TV is displayed on an HMD worn by a viewer of the TV in a general home or the like. According to this embodiment, when watching TV, it is possible to obtain supplementary information that cannot be obtained only from the content of the broadcast, and to view secondary information even if the user takes his or her eyes off the TV.

FIG. 15 is a diagram for explaining the outline of the operation of the HMD-linked display system in this embodiment. In FIG. 15, the display device 400 displays the contents of the TV broadcast on the screen. It should be noted that the screen shown in FIG. 15 shows a state in which a program providing information on four products, product A, product B, product C, and product D, is being broadcast as TV broadcast contents. A viewer 911 is viewing this displayed screen via the HMD 300 . At this time, the scene seen by the viewer 911 through the HMD 300 initially consists of the TV screen and the buttons for operating the TV (for example, the button "volume +" for adjusting the volume of the TV) as shown on the screen 357. (indicating "Volume -") is visible. Next, when the viewer 911 gazes at product A displayed on the TV screen, supplementary information (for example, store selling product A, price, telephone number) is displayed as shown in screen 358 . Next, when the user looks away from the TV screen, as shown in screen 359, the buttons for operating the TV are erased, while the supplementary information continues to be displayed even if the TV screen is not viewed. By operating in this manner, the supplementary information can be confirmed when, for example, leaving the vicinity of the TV to make a call to the store displayed in the supplementary information.

FIG. 16 is a diagram illustrating an overview of the HMD-linked display system in this embodiment. In FIG. 16, this system comprises a broadcasting facility 940 that transmits a broadcast signal via a transmission antenna 950, a display device 400 that receives and displays the broadcast signal, and an HMD 300. FIG. The display device 400 can also receive communication data via a communication network 960 such as the Internet in addition to normal broadcast signals. As an apparatus that receives and displays both a broadcast signal and communication data, there is, for example, a TV compatible with Hybridcast (registered trademark). In this embodiment, by using such a device, a secondary information database related to TV broadcasts (primary information) received by broadcast signals is obtained by communication via the Internet.

FIG. 17 is an overall configuration diagram of an HMD-linked display system in this embodiment. In FIG. 17, the display device 400 of the present embodiment is the display device 200 described in the first embodiment, with some modules included in a device such as a TV capable of viewing both broadcast signals and communication data. It is. The display device 400 includes a tuner unit 420 that receives a broadcast signal, a separation unit 430 that separates the received broadcast signal into various signals such as video, audio, and data and outputs them, and a display that performs processing such as demodulation of the received video signal. It includes a control unit 440, a display unit 450 that displays video, an audio control unit 460 that performs processing such as demodulation of received audio signals, and a speaker 470 that outputs audio. These modules are the modules required in a typical TV for viewing broadcast signals. In addition to these modules, the display device 400 includes an IP (Internet Protocol) communication unit 410 that receives communication data via a communication network such as the Internet, program identification information 580 that stores the number of the currently viewed channel, etc., and a secondary module. A recording unit 210 that stores an information database 590, a control unit 220 that performs various processes such as outputting primary information and secondary information, a communication unit 240 that communicates with the HMD 300, It includes a gaze point calculation unit 250 that calculates the coordinates of the gaze point, and a voice recognition unit 260 that recognizes the voice of the HMD wearer or the like. Note that the point-of-regard calculation unit 250 and the speech recognition unit 260 may be configured to be realized as dedicated hardware, or may be configured to be realized as software modules executed by the control unit 220 . The configuration of the HMD 300 is similar to that of the first embodiment.

FIG. 18 is a diagram illustrating the configuration of the secondary information database 590 in this embodiment. In FIG. 18, the secondary information database 590 includes program identification information 591, time zone 592 indicating the period during which the secondary information is valid, gaze point range 593 for selecting secondary information based on the gaze point of the HMD wearer, It consists of secondary information 594 and attributes 595 of the secondary information. As shown in the lower screen of FIG. 18, the coordinate system of the primary information in this embodiment is defined by (0, 0) on the upper left and (1920, 1080) on the lower right. The gaze point range (300, 50) to (900, 450) is a rectangular area including the image of product A, and the gaze point range (1000, 50) to (1600, 450) on the second line includes the image of product B. Coordinates indicating the range of a rectangle.

The above is the configuration of the HMD-linked display system of the present embodiment. The present embodiment will be described below along the flow of operation of this system. It should be noted that the operation of operating the display device 400 to watch the TV broadcast is the same as the operation method of a commonly used TV, and therefore will be omitted. Also, the following description is based on the assumption that channel 1 is being viewed.

FIG. 19 shows a camera image 540 and an in-camera-image gazing point 550 obtained by the imaging unit 310 and the in-camera-image gazing-point detecting unit 320 of the HMD 300 when the viewer 911 looks at the display device 400 through the HMD 300 . is shown. In FIG. 19, a viewer 911 is viewing the primary information from a position slightly to the right of the display device 400, and is gazing at product A. In FIG.

FIG. 20 is a flow chart of processing for selecting secondary information, which is executed in the control unit 220 of the display device 400 when the camera image 540 and the camera image fixation point 550 are received from the HMD 300 in this embodiment. In FIG. 20, the control unit 220 first determines whether the program identification information 580 is recorded in the recording unit 210 (step S2411). Since channel 1 is currently being viewed, the program identification information 580 exists and channel 1 is described as the program being viewed. Next, it is determined whether or not the gazing point 550 in the camera image has remained the same for a certain period of time, that is, whether or not the gazing point of the viewer 911 has stayed at a specific place (step S2412). Next, the gaze point calculation unit 250 is used to calculate the gaze point in the primary information (step S2413). The details of this process are the same as those described with reference to FIGS. 9 to 11 of the first embodiment.

Next, it is determined whether or not the calculation of the gazing point in the primary information has succeeded (step S2414). If the calculation of the gazing point within the primary information is successful, that is, if the gazing point of the viewer 911 is in the direction of the primary information (that is, the direction of the display device 400), the gazing point within the primary information calculated in step S2413 is stored. (step S2415). Next, the secondary information database 590 is referred to, and among the secondary information related to the currently viewed program, the point of interest in the stored primary information exists within the range of the point of interest 593 in the secondary information database. and the secondary information whose current time is within the time zone 592 of the secondary information database is displayed (step S2416).

On the other hand, if the calculation of the gaze point in the primary information fails in step S2414, the secondary information (that is, the currently displayed secondary information) corresponding to the gaze point in the primary information stored in step S2415 is , an erasure timer is set according to the attribute 595 of the secondary information (step S2417). For example, if the attribute 595 of the secondary information is text, it will be erased after 60 seconds, and if the attribute 595 of the secondary information is a button, it will be erased after 0 seconds (that is, immediately). As a result, when the viewer 911 looks away from the display device 400, the button is immediately deleted, but the text continues to be displayed for a certain period of time. Subsequent processing is the same as in the first embodiment.

By the above processing, the operation shown in FIG. 15 becomes possible. That is, the scene that the viewer 911 sees through the HMD 300 is the screen 357 in FIG. 15 at first. Next, when product A is gazed at, screen 358 in FIG. 15 is displayed by the processing up to step S2416 in FIG. Next, when the user looks away from the display device 400 and moves to another place, the screen 359 shown in FIG. 15 appears.

As described above, the present embodiment is a display device connected to a head-mounted display, and includes a display unit for displaying primary information, a projection unit for projecting primary information, or a signal output unit for outputting primary information, and a head-mounted display. A communicable communication unit and a point-of-regard calculation unit that calculates the position information of the gaze point of the wearer of the head-mounted display with respect to the primary information. Calculate the position information of the gaze point of the wearer, select the secondary information related to the position information, and judge that the position information is in the direction of the primary information, and judge that the position information is not in the direction of the primary information. The secondary information to be displayed on the head-mounted display is changed depending on the case.

As a result, it becomes possible to obtain supplementary information that cannot be obtained from the content of the broadcast when watching TV, and it is also possible to view secondary information even if the user takes his or her eyes off the TV, increasing the degree of freedom of action of the viewer. can be enhanced.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add/delete/replace other configurations for a part of each embodiment.

100: projection device, 110: signal input section, 120: control section (projection device), 130: display section (projection device), 200: display device, 210: recording section, 220: control section (display device), 230: Signal output unit, 240: communication unit (display device), 250: gaze point calculation unit, 260: voice recognition unit, 270: operation unit, 280: display unit (display device), 300: head mounted display, 310: imaging unit , 320: in-camera image fixation point detection unit, 330: voice acquisition unit, 340: control unit (head-mounted display), 350: communication unit (head-mounted display), 360: display unit (head-mounted display), 400: display Device 410: IP communication unit 420: Tuner unit 430: Separation unit 440: Display control unit 450: Display unit (display device) 460: Audio control unit 470: Speaker 510: Primary information database 520 : secondary information database, 530: primary information, 540: camera image, 550: gaze point in camera image, 560: audio data, 570: secondary information, 580: program identification information, 590: secondary information database, 910: Teacher, 911: Viewer, 920: Student, 930: Screen, 940: Broadcasting equipment, 950: Transmitting antenna, 960: Internet

Claims (3)

A display device that acquires gaze point information of the wearer from a head-mounted display device comprising an imaging unit and a gaze point detection unit capable of detecting the gaze point by detecting the gaze direction of the wearer and controls display information. There is
a signal output unit that outputs the primary information to an external display device that displays the primary information;
a communication unit that inputs information related to a captured image from the head-mounted display device and the gaze point information in the captured image ;
display on the external display device for displaying the primary information output from the signal output unit based on the position coordinates of a specific target in the information related to the captured image input from the communication unit and the gaze point information; a point-of-regard calculation unit that calculates positional information of the point-of-regard of the wearer with respect to the primary information area, which is an area;
Equipped with a control unit,
When the control unit determines that the position information of the gaze point of the wearer calculated by the gaze point calculation unit is in the primary information area, the control unit outputs secondary information related to the gaze point position of the primary information to the When it is determined that the positional information of the gaze point of the wearer is not in the primary information area while the secondary information is output to the head-mounted display device by the communication unit and the secondary information is being output to the head-mounted display device 7. A display device, wherein a time until erasure of said secondary information is set according to an attribute of said secondary information. A display device that acquires gaze point information of the wearer from a head-mounted display device comprising an imaging unit and a gaze point detection unit capable of detecting the gaze point by detecting the gaze direction of the wearer and controls display information. There is
a signal output unit that outputs the primary information to an external display device that displays the primary information;
a communication unit that inputs information related to a captured image from the head-mounted display device and the gaze point information in the captured image ;
display on the external display device for displaying the primary information output from the signal output unit based on the position coordinates of a specific target in the information related to the captured image input from the communication unit and the gaze point information; a point-of-regard calculation unit that calculates positional information of the point-of-regard of the wearer with respect to the primary information area, which is an area;
Equipped with a control unit,
When the control unit determines that the position information of the gaze point of the wearer calculated by the gaze point calculation unit is in the primary information area, the control unit outputs secondary information related to the gaze point position of the primary information to the When it is determined that the positional information of the gaze point of the wearer is not in the primary information area while the secondary information is output to the head-mounted display device by the communication unit and the secondary information is being output to the head-mounted display device 2. A display device, wherein the transmittance of said secondary information is changed according to the attribute of said secondary information. The display device according to claim 1 or claim 2,
The communication unit acquires sounds around the wearer as audio data from a head-mounted display device capable of acquiring sounds around the wearer,
further comprising a speech recognition unit that converts the speech data into text;
A display device, wherein, when a specific keyword is detected in the text, the control unit outputs information related to the keyword to the head-mounted display device as the secondary information.

Images