JP7372401B2 - Head-mounted display cooperation display system, system including display device and head-mounted display, and display device thereof - Google Patents

Description

The present invention relates to a display system and a display device using a head-mounted display.

Second information (hereinafter referred to as secondary information) related to the first information (hereinafter referred to as primary information) displayed on the main display device is displayed on a head-mounted display device (Head Mounted Display, Head Mounted Display). A technology for displaying on a mounted display (hereinafter referred to as HMD) has been devised.

As background technology in this technical field, there is Japanese Patent Laid-Open No. 2001-215920 (Patent Document 1). In Patent Document 1, when a wearer of an HMD views primary information displayed on the screen of a main display device, primary information existing in the direction is detected according to the direction of the wearer's head and line of sight detected by the HMD. This article describes a technology for displaying secondary information related to information on an HMD. This technology reduces the amount of information displayed on the main display device (primary information), avoiding the need to increase the size of the device, and allows the wearer to check the necessary information (secondary information) by simply moving their head or line of sight. Therefore, the operability of the device can be improved.

Additionally, there is Japanese Patent Application Publication No. 2010-237522 (Patent Document 2). In Patent Document 2, when a wearer of an HMD views primary information projected on a large screen, secondary information is displayed at an appropriate position adjacent to the primary information based on the positional relationship between the wearer's seat and the screen. It talks about technology. With this technology, it is possible to display subtitles (secondary information) in the native language so that only the person wearing the HMD can see them when watching a movie (primary information).

Japanese Patent Application Publication No. 2001-215920 JP2010-237522A

Patent Document 1 describes that secondary information of information existing at the wearer's gaze point detected by the HMD is displayed. However, the gaze point detected by the HMD indicates the position in the wearer's field of vision. In other words, this field of view includes the surrounding scenery in addition to the primary information displayed by the main display device, and since the wearer is not necessarily in front of the main display device, the primary view changes depending on the wearer's standing position. Information comes in different forms. Therefore, in order to know where in the primary information the wearer is gazing, it is necessary to convert the gaze point detected by the HMD into coordinates in the primary information by some means. Nothing is disclosed about the means.

Furthermore, Patent Document 2 calculates the display position of secondary information on the HMD by performing coordinate transformation based on the positional relationship between the wearer's seat and the screen. It cannot be used when viewing primary information based on location.

Further, in both Patent Documents 1 and 2, the secondary information is displayed only while viewing the primary information, so if the wearer takes his eyes off the primary information, he cannot view the secondary information.

The present invention has been made in view of the above problems, and even when the HMD wearer moves or takes his/her eyes off the main display device, secondary information of primary information displayed on the main display device can be appropriately displayed on the HMD. The present invention discloses a means for displaying the information on the screen.

In order to solve the above problems, the present invention provides, for example, a system including a display device and a head-mounted display, wherein the display device has a first display section capable of displaying an image or a first display section capable of projecting an image. The head-mounted display includes a first communication section capable of communicating with the head-mounted display, and a second display section that displays an image that can be viewed by the wearer of the head-mounted display; and a line-of-sight detection unit that detects the line-of-sight direction of the person wearing the display, and a second communication unit capable of communicating with the display device, and based on the information transmitted and received via the first communication unit and the second communication unit. The point of gaze detection unit that detects the position of the gaze point of the eye of the wearer of the head mounted display with respect to the image displayed on the first display unit of the display device or the image projected by the projection unit is connected to the display device or the head. In preparation for one of the mounted displays, the gaze point detection unit calculates positional information in the image corresponding to the position of the gaze point when the gaze point is on the image, and the head mounted display calculates position information in the image corresponding to the position of the gaze point when the gaze point is on the image. The second display section is configured to acquire related data regarding the target data displayed on the second display section through communication with the display device or other communication.

According to the present invention, appropriate secondary information can be selected and displayed regardless of the position or direction of line of sight of the HMD wearer, increasing the degree of freedom of action of the HMD wearer and This has the effect of allowing you to view information.

FIG. 2 is a diagram illustrating an overview of the operation of the HMD-linked display system in Example 1. 1 is an overall configuration diagram of an HMD-linked display system in Example 1. FIG. 2 is a configuration diagram of a primary information database in Example 1. FIG. 2 is a configuration diagram of a secondary information database in Example 1. FIG. 3 is a diagram illustrating a primary information selection operation in Example 1. FIG. 7 is a flowchart of primary information selection processing in the first embodiment. 2 is a diagram illustrating a camera image and a gaze point within the camera image in Example 1. FIG. 7 is a flowchart of secondary information selection processing based on a gaze point in Example 1. FIG. 7 is a flowchart of calculation processing of a gaze point in primary information in Example 1. FIG. 2 is a diagram illustrating an overview of projective transformation in Example 1. FIG. 7 is another diagram illustrating an overview of projective transformation in Example 1. FIG. 7 is a flowchart of secondary information deletion processing in Example 1. FIG. FIG. 3 is a diagram illustrating an outline of an operation for selecting secondary information by voice in the first embodiment. 7 is a flowchart of a process for selecting secondary information using audio in Example 1. FIG. FIG. 7 is a diagram illustrating an overview of the operation of the HMD-linked display system in Example 2. FIG. FIG. 2 is a diagram showing an overall image of an HMD-linked display system in Example 2. FIG. 2 is an overall configuration diagram of an HMD-linked display system in Example 2. FIG. FIG. 7 is a diagram illustrating the configuration of a secondary information database in Example 2. FIG. FIG. 7 is a diagram illustrating a camera image and a gaze point within the camera image in Example 2. FIG. 7 is a flowchart of secondary information selection processing using a gaze point in Example 2. FIG.

Embodiments of the present invention will be described below with reference to the drawings.

This embodiment describes an example in which supplementary information (secondary information) related to educational content (primary information) projected onto a screen by a projector is displayed on an HMD worn by a teacher in an educational setting. According to this embodiment, a teacher can view supplementary information related to educational content while teaching students in a natural manner.

FIG. 1 is a diagram illustrating an overview of the operation of the HMD linked display system in this embodiment. In FIG. 1, the projection device 100 projects educational content (a world map in the example) onto a screen 930. The teacher 910 is teaching while alternately viewing the content projected via the HMD 300 and the students 920 . At this time, what the teacher 910 sees through the HMD 300 is initially the educational content (world map) and buttons for operating the content display (in the example, "previous" and "next"), as shown in the screen 351. ) only. Next, when the teacher 910 looks at the vicinity of Greenland on the world map, supplementary information (in the example, country name, capital, and official language) is displayed, as shown on screen 352. Next, when the teacher 910 looks in the direction of the student 920, as shown in screen 353, the buttons for operating the content display disappear, while the supplementary information continues to be displayed even if the teacher 910 is not looking at the world map. Thereafter, when a certain period of time has elapsed, the supplementary information is also deleted, as shown on screen 354. With such an operation, it is possible to conduct a lesson without performing unnatural movements such as reorienting the educational content one by one and gazing at the relevant part in order to obtain supplementary information, for example.

FIG. 2 is an overall configuration diagram of the HMD linked display system in this embodiment. In FIG. 2, this system includes a projection device 100, a display device 200, and an HMD 300. The projection device 100 and the display device 200, and the display device 200 and the HMD 300 are connected through communication.

The projection device 100 includes a signal input section 110 for inputting primary information to be displayed, a control section 120 for controlling display, and a display section 130 for projecting the primary information onto a screen.

The display device 200 includes a recording unit 210 that stores a primary information database 510 and a secondary information database 520, a control unit 220 that performs various processes such as outputting primary information and secondary information, and a signal that outputs primary information to the projection device 100. A communication unit 240 that communicates with the output unit 230, the HMD 300, and a gaze point detection unit that detects the position of the gaze point in the primary information based on the information acquired from the HMD 300, and detects the coordinates that are the position information of the gaze point. It is comprised of a gaze point calculation unit 250 that performs calculations, a voice recognition unit 260 that recognizes the voice of an HMD wearer, an operation unit 270 that operates the display device 200, and a display unit 280. Note that the gaze point calculation unit 250 and the speech recognition unit 260 may be implemented as dedicated hardware, or may be implemented as a software module executed by the control unit 220. Moreover, the gaze point calculation unit 250 may be provided in the HMD 300.

The HMD 300 includes an imaging unit 310 that captures an image of the scenery in the direction in which the wearer is looking, a line-of-sight detection unit that detects the direction of the wearer's gaze in the captured camera image, and a gaze point in the camera image that detects the gaze point. A detection unit 320, a voice acquisition unit 330 that acquires the voice of the HMD wearer, etc., and a control unit that performs various control processes such as transmitting the acquired camera image 540, camera image gaze point 550, and voice data 560 to the display device 200. 340, a communication unit 350 that communicates with the display device 200, and a display unit 360 that displays secondary information 570 acquired from the display device as an image that can be viewed or viewed by the wearer.

In this embodiment, the projection device 100 corresponds to a projector, and the display device 200 corresponds to a PC (Personal Computer) connected to the projector. However, the projection device 100 is not limited to a form using a projector. The present invention can be applied to a general display device or a dedicated device that integrates the projection device 100 and the display device 200. Furthermore, the HMD 300 may be configured to be divided into a device that is worn on the head and mainly performs display, and a device that is worn on the waist or the like and mainly controls the HMD.

FIG. 3 is a structural diagram of the primary information database 510 in this embodiment. In FIG. 3, the primary information database 510 includes a primary information identifier 511, a name 512 of the primary information, a name 513 of the file storing the primary information, and a display flag 514 indicating whether the primary information is being displayed. be done. Note that the on-display flag 514 will be described as being '1' during display, and '0' when not displaying.

The upper diagram of FIG. 4 is a structural diagram of the secondary information database 520 in this embodiment. In the upper diagram of FIG. 4, the secondary information database 520 includes a primary information identifier 521 that identifies related primary information, a gaze point range 522 for selecting secondary information based on the gaze point of the HMD wearer, and a It is composed of a keyword 523 for selecting next information, secondary information 524, and attributes 525 of the secondary information. As shown in the map shown in the lower part of FIG. 4, in this example, the coordinate system of the primary information is defined as (0, 0) at the upper left and (1920, 1080) at the lower right, and the coordinate system of the primary information in the upper left of FIG. The coordinates of the first line of the information database (1680, 70) to (1880, 250) indicate the vicinity of Greenland, and the second line (700, 720) to (1000,870) indicate the vicinity of Australia. It means. The gaze point range (0, 0) to (1920, 1080) in the third and fourth lines of the secondary information database always contains secondary information (in the example, " ``Previous'' button and ``Next'' button).

The above is the configuration of the HMD linked display system of this embodiment. The present embodiment will be described below along with the flow of the operation of the present system.

FIG. 5 is a diagram illustrating an operation for selecting educational content (primary information) to be displayed on the display unit 280 of the display device in this embodiment and projected onto the screen 930. In FIG. 5, a screen 281 is an educational content selection screen, and displays a plurality of icons including an icon 282 for displaying a world map. When the icon 282 is selected here, a world map 283 is displayed as shown in the lower part of FIG. Note that in the world map 283, markers 284 indicated by diagonal lines are displayed at the four corners. This is identification information for specifying the display area of the primary information, and details will be described later.

FIG. 6 is a flowchart of the process of the control unit 220 when selecting primary information in this embodiment. In FIG. 6, the control unit 220 reads a list of primary information from the primary information database 510 and displays it on the selection screen 281 consisting of icons representing each content (step S2201). Next, in step S2202, the process waits until the user selects content. Next, the file 513 corresponding to the primary information selected by the user is read and projected onto the screen 930 via the signal output unit 230 and the projection device 100 (step S2203). Finally, the displaying flag 514 of the selected primary information is set to 1, and the process ends (step S2204).

FIG. 7 shows the image obtained by the imaging unit 310 and camera image gaze point detection unit 320 of the HMD 300 when the teacher 910 views the screen 930 on which the educational content is projected in step S2203 of FIG. 6 via the HMD 300. A camera image 540 and a gaze point 550 in the camera image are shown. The figure shows that the teacher 910 is viewing primary information from a position slightly to the right of the screen 930, and is gazing at the vicinity of Greenland.

FIG. 8 shows a process for selecting secondary information, which is executed in the control unit 220 of the display device 200 when the camera image 540 and camera image gaze point 550 shown in FIG. 7 are received from the HMD 300 in this embodiment. This is a flowchart. In FIG. 8, the control unit 220 first refers to the primary information database 510 and determines whether there is any primary information whose display flag is set to 1 (step S2211). Note that the display flag in the first line (world map) of the primary information database 510 is set to 1 by the operation shown in FIG. Next, it is determined whether the gaze point 550 in the camera image has been the same for a certain period of time, that is, whether the teacher's 910 gaze point has stayed at a specific location (step S2212). This determination can be realized, for example, by a process of determining whether a state in which the distance difference from the gaze point in the camera image received last time is less than a predetermined threshold continues for a certain period of time. Next, a gaze point within the primary information is calculated using the gaze point calculation unit 250 (step S2213). Details of this processing will be described later.

Next, it is determined whether the calculation of the gaze point within the primary information has been successful (step S2214). If the calculation of the gaze point within the primary information is successful, that is, when the gaze point of the teacher 910 is in the direction of the primary information (i.e., the direction of the screen 930), the gaze point within the primary information calculated in step S2213 is stored ( Step S2215). Next, the secondary information database 520 is referred to, and among the secondary information related to the primary information being displayed, the primary information gaze point stored in step S2213 is within the gaze point range 522 of the secondary information database 520. The secondary information existing within is displayed (step S2216). On the other hand, if the calculation of the gaze point in the primary information fails in step S2214, that is, if the gaze is outside the marker range of the primary information, then the secondary information corresponding to the gaze point in the primary information stored in step S2215 (that is, the currently displayed secondary information), an erasure timer is set according to the attribute 525 of the secondary information (step S2217). The erasure timer is the time until the secondary information displayed on the HMD 300 is erased. For example, if the attribute 525 of the secondary information is text, it will be erased after 60 seconds, and if the attribute 525 of the secondary information is a button, it will be erased. is set to be erased after 0 seconds (that is, immediately). This causes the button to disappear immediately when the teacher 910 looks in the direction of the student 920, but the text remains displayed for a certain period of time. Note that if the calculation of the primary information gaze point fails at the beginning of this secondary information selection flow, the deletion timer is not set because there is no stored primary information gaze point. In this way, the display method, such as whether to continue displaying the secondary information or immediately erasing it, can be determined depending on whether the position information of the wearer's gaze point is found in the primary information or not. change. In other words, the display layout or display menu may be changed.

FIG. 9 is a flowchart illustrating a process of calculating a gaze point in primary information using the gaze point calculation unit 250, which is executed in step S2213 described above. The gaze point calculation unit 250 first detects the markers 284 attached to the four corners of the primary information (step S2501). Next, it is determined whether the marker has been detected (step S2502). If the marker cannot be detected, the calculation is deemed to have failed, and the process ends; if the marker can be detected, the coordinates of the gaze point within the primary information are calculated by projective transformation, which will be described later (step S2503), and the process ends.

FIG. 10 is a diagram illustrating an overview of the projective transformation executed in step S2503 described above. By the calculation described below, the coordinates of the gaze point in the coordinate system 251 in the camera image can be converted to the coordinates in the coordinate system 252 in the primary information.

In FIG. 10, the coordinate system 251 in the camera image is a plane in which the upper left is (0, 0) and the lower right is (100, 100). On the other hand, the coordinate system 252 in the primary information is a plane in which the upper left is (0, 0) and the lower right is (1920, 1080). Here, the primary information area 253 within the camera image specified by the four corner markers detected in step S2501 is converted into an area in the coordinate system within the primary information. There are various calculation formulas for this transformation, but in this embodiment, a general projective transformation formula 254 is used. Here, (x, y) are the coordinates before transformation (coordinate system 251 in the camera image), and (u, v) are the coordinates after transformation (coordinate system 252 in the primary information). The projective transformation equation 254 has eight unknowns (a1, b1, c1, a2, b2, c2, a0, b0). Therefore, by substituting four points whose corresponding coordinates are known in both coordinate systems and obtaining eight equations, the unknown quantity can be derived. The coordinate correspondence table 255 shows the correspondence between the coordinates (x, y) of the four markers detected in step S2501 of FIG. 9 and the coordinates (u, v) after conversion. The upper left coordinates (10, 20) become (0, 0), the upper right coordinates (70, 18) become (1920, 0), the lower left coordinates (12, 80) become (0, 1080), the lower right The coordinates (65, 82) correspond to (1920, 1080), respectively. By substituting these values into the projective transformation equation 254, a simultaneous equation consisting of eight equations is obtained, and by solving this, a calculation result 256 of the unknown is obtained. The coordinate transformation result 257 of the gaze point shows that (60, 28) is transformed into (1635, 148) when calculation is performed using the projection transformation formula 254 using the unknown quantity calculation result 256 (note that , where the calculation result is rounded to an integer). Through the above calculation, the gaze point in the coordinate system of the camera image is converted into coordinates in the coordinate system of the primary information.

In this embodiment, it is assumed that diagonal line pattern markers are displayed at the four corners of the primary information as in the examples shown in FIGS. 5 and 7, and the area of the primary information is detected by image recognition. However, various other techniques can be used for such markers. For example, a pattern other than diagonal lines may be used, or a method may be used in which a physical device for area detection is embedded on the screen 930 side, rather than displaying markers in the first place. Furthermore, instead of a visible pattern that is visible to humans, an invisible marker using an infrared camera or the like may be used.

Further, although FIG. 10 describes an example in which projective transformation is performed using the coordinates of the four corners of the primary information area, the method of projective transformation is not limited to this. For example, FIG. 11 is a conceptual diagram showing another method of projective transformation. When the HMD wearer views primary information and stands close to the screen 930, the markers at the four corners may not be included in the camera image, as in the camera image 541 shown in FIG. 11. In such a case, instead of using the correspondence between the coordinates of the four corners of the primary information area in the coordinate system 258 in the camera image and the coordinate system 259 in the primary information, the coordinates of the four corners of one marker are used. By finding the unknowns of the projective transformation equation 254, the coordinate transformation of the gaze point can be performed. In addition, instead of using the coordinates of four predetermined points, image recognition is performed on both the camera image and primary information, and four distinctive points (feature points) are dynamically extracted. , it is also possible to derive the unknowns of the projective transformation based on these four points. Such methods are widely used, for example, in processing for recognizing human faces from various angles.

Finally, FIG. 12 is a flowchart showing a process for erasing secondary information, which is activated when the erasure timer set in step S2217 of FIG. 8 reaches a predetermined time. In FIG. 12, when the erasure timer reaches a predetermined time, the control unit 220 erases the secondary information displayed on the HMD (step S2221). As described above, by changing the value of the erasure timer according to the attributes of the secondary information, if you take your eyes off the screen 930 and look at the student 920, the button will disappear immediately, but the text (supplementary information ) can continue to be displayed.

The contents described above enable the operation shown in FIG. 1. That is, when the teacher 910 looks in the direction of the screen 930 via the HMD 300, the scene initially seen is the screen 351 in FIG. Next, when looking at the vicinity of Greenland on the world map, the screen 352 in FIG. 1 is displayed through the processing up to step S2216 in FIG. Next, when looking in the direction of the student 920, the screen 353 in FIG. 1 appears and the text (supplementary information) continues to be displayed until the erasure timer set in step S2217 in FIG. 8 has elapsed. When the erasure timer time set for the last text has elapsed, the screen 354 shown in FIG. 1 appears.

Note that in step S2217 of FIG. 8, in addition to setting the erasure timer, the transmittance of the secondary information displayed on the HMD may be changed according to the attribute 525 of the secondary information. For example, when the teacher 910 is looking in the direction of the screen 930, as shown in the screen 352 of FIG. When viewing the screen, the supplementary information may be displayed with a transmittance of 50%, that is, semi-transparent, as illustrated in the screen 353 of FIG. 1. This makes it possible to avoid obscuring the appearance of the student 920 by displaying supplementary information.

Next, as another operation of this embodiment, an example of an operation of displaying secondary information by voice even when primary information is not viewed will be described.

FIG. 13 is a diagram illustrating an overview of the operation of displaying secondary information by voice in this embodiment. In FIG. 13, following the actions described so far, the teacher 910 is looking in the direction of the student 920, the student 920 asks the question, "How many people live there?", and the teacher 910 says, "The population is... '' is shown saying. At this time, the scene that the teacher 910 sees through the HMD 300 is that at first only the students are visible, as shown in the screen 355, and then when the teacher 910 utters "The population is...", supplementary information (in the example, the name of the country) is displayed as shown in the screen 356. , population, population density). Note that supplementary information may be displayed in response to the student's voice. Further, at this time, the user may capture the utterance, display a keyword, and display supplementary information by gazing at the keyword.

FIG. 14 shows the selection of secondary information that is executed when the voice acquisition unit of the HMD 300 acquires the utterance of the teacher 910 and the control unit 220 of the display device 200 receives the voice data 560 in this embodiment. FIG. In FIG. 14, similarly to FIG. 8, the control unit 220 first refers to the primary information database 510 and determines whether there is any primary information whose display flag is set to 1 (step S2311). Next, voice recognition processing is performed on the received voice data 560 (step S2313). Note that this voice recognition processing is not limited to a method executed within the display device 200, but may be executed by communicating with a server that performs voice recognition via the Internet or the like. Further, the voice recognition process may be triggered always at a predetermined cycle, or may be triggered when a predetermined button is pressed. Next, it is determined whether the conversion process from the voice data 560 to text by voice recognition has been successful (step S2314). Note that this determination may be made simply based on whether or not speech recognition can be performed, or may be made based on the reliability of the conversion result output by a general speech recognition technology. Furthermore, by identifying the speaker by using a technology such as voice print analysis, the utterances of the student 920 may be ignored and only the utterances of the teacher 910 may be judged as successful. Next, if the conversion from voice to text is successful, the secondary information database 520 is referred to, and among the secondary information related to the primary information currently displayed, the primary information stored in step S2215 of FIG. Displays secondary information in which the gaze point is within the gaze point range 522 in the secondary information database, and the converted text includes the words shown in the keywords 523 of the secondary information database 520 ( Step S2315). In the example of the secondary information database shown in FIG. 4, the secondary information in the fifth line in which "population" is set as the keyword is displayed.

In addition, here we have shown an example of displaying secondary information immediately when the utterance content contains a specific keyword, but instead of displaying the secondary information immediately after recognizing the voice, a button displaying the recognized keyword is displayed on the HMD. The secondary information may be displayed by selecting the button. Further, even if the content of the utterance does not completely match the keyword, secondary information may be displayed if the content of the utterance is similar to the keyword by determining the degree of similarity between character strings.

The above processing enables the operation shown in FIG. 13. That is, the scene when the teacher 910 looks in the direction of the student 920 via the HMD 300 is the screen 355 in FIG. 1 at first. Next, when the teacher 910 utters "The population is...", the screen 356 in FIG. 1 is displayed through the processing up to step S2315 in FIG.

As described above, this embodiment calculates the positional information of the wearer's gaze point with respect to primary information from the camera image acquired from the HMD and the gaze point of the HMD wearer with respect to the camera image, and calculates the positional information of the wearer's gaze point with respect to the primary information. The secondary information is selected and displayed, and the display method of the secondary information displayed on the HMD is changed depending on whether the wearer of the HMD is viewing the primary information or not viewing the primary information. This allows the teacher to provide supplementary information to the educational content projected on the screen while teaching students in a natural manner.

In other words, this embodiment is a system including a display device and a head-mounted display, where the display device includes a first display section capable of displaying an image or a projection section capable of projecting an image, and the display device includes a first display section capable of displaying an image or a projection section capable of projecting an image. The head-mounted display includes a first communication unit capable of communicating with a second display unit that displays an image that can be viewed by the wearer of the head-mounted display, and a second display unit that detects the line of sight direction of the wearer of the head-mounted display. and a second communication section capable of communicating with the display device, the first display section of the display device is provided with a second communication section that can communicate with the display device, Either the display device or the head-mounted display is equipped with a gaze-point detection unit that detects the position of the gaze point of the eye of the wearer of the head-mounted display with respect to the image displayed on the image or the image projected by the projection unit, and The detection unit calculates positional information in the image corresponding to the position of the gaze point when the gaze point is on the image, and the head mounted display calculates related data regarding the target data displayed at the calculated position in the image. is configured to be acquired through communication with the display device or other communication and displayed on the second display unit.

Also, a head-mounted display cooperation display system including a display device and a head-mounted display, wherein the display device includes a first display section that displays primary information, a projection section that projects the primary information, or a signal that outputs an image signal. The head mounted display includes an output unit, a first communication unit capable of communicating with the head mounted display, and a gaze point calculation unit that calculates positional information of the gaze point of the wearer of the head mounted display based on primary information. a second display unit that displays secondary information that can be viewed by the wearer; a second communication unit that can communicate with the display device; an imaging unit that captures a camera image in the direction in which the wearer is facing; The display device includes an in-camera image gaze point detection unit that detects the wearer's gaze point, and the display device uses a gaze point calculation unit to calculate the gaze point based on the detected gaze point transmitted and received via the first communication unit and the second communication unit. The position information of the wearer's gaze point relative to the primary information is calculated, the secondary information related to the position information is selected, and the second The display unit is configured to change the display method of secondary information on the display unit.

As a result, appropriate secondary information can be selected and displayed regardless of the position or line of sight of the HMD wearer, increasing the freedom of movement of the HMD wearer and viewing secondary information in a more natural manner. There is an effect that it can be done.

This embodiment describes an example in which supplementary information (secondary information) related to broadcast content (primary information) displayed on a TV is displayed on an HMD worn by a TV viewer in a general home or the like. According to this embodiment, when watching TV, it is possible to obtain supplementary information that cannot be obtained from the broadcast content alone, and it is also possible to view secondary information even if the user takes his eyes off the TV.

FIG. 15 is a diagram illustrating an overview of the operation of the HMD linked display system in this embodiment. In FIG. 15, a display device 400 displays the contents of a TV broadcast on its screen. Note that the screen shown in FIG. 15 shows that a program providing information on four products, product A, product B, product C, and product D, is being broadcast as the content of the TV broadcast. The viewer 911 is viewing this displayed screen via the HMD 300. At this time, the scene that the viewer 911 sees through the HMD 300 is initially a TV screen and buttons for operating the TV (for example, a button "Volume +" for adjusting the volume of the TV), as shown in a screen 357. (indicating "Volume -") is visible. Next, when the viewer 911 gazes at the product A displayed on the TV screen, supplementary information (in the example, the store selling the product A, the price, and the telephone number) is displayed as shown in a screen 358. Next, when the user takes his or her eyes off the TV screen, the buttons for operating the TV disappear, as shown in screen 359, while the supplementary information continues to be displayed even when the TV screen is not viewed. By operating in this manner, the user can check the supplementary information when leaving the vicinity of the TV, for example, to make a call to a store displayed in the supplementary information.

FIG. 16 is a diagram illustrating the overall image of the HMD linked display system in this embodiment. In FIG. 16, this system is comprised of broadcast equipment 940 that transmits broadcast signals via transmission antenna 950, display device 400 that receives and displays the broadcast signals, and HMD 300. In addition to normal broadcast signals, the display device 400 can also receive communication data via a communication network 960 such as the Internet. Note that such a device that receives and displays both broadcast signals and communication data includes, for example, a TV compatible with Hybridcast (registered trademark). In this embodiment, by using such a device, a secondary information database related to TV broadcasts (primary information) received by broadcast signals is acquired by communication via the Internet.

FIG. 17 is an overall configuration diagram of the HMD linked display system in this embodiment. In FIG. 17, a display device 400 according to the present embodiment has several modules added to the display device 200 described in the first embodiment, including a device such as a TV that can view both broadcast signals and communication data. It is something. The display device 400 includes a tuner section 420 that receives a broadcast signal, a separation section 430 that separates the received broadcast signal into various signals such as video, audio, and data and outputs the signals, and a display that performs processing such as demodulation of the received video signal. It includes a control section 440, a display section 450 that displays images, an audio control section 460 that performs processing such as demodulation of received audio signals, and a speaker 470 that outputs audio. These modules are required by a typical TV to view broadcast signals. In addition to these modules, the display device 400 includes an IP (Internet Protocol) communication unit 410 that receives communication data via a communication network such as the Internet, program identification information 580 that stores the channel number currently being viewed, etc. A recording unit 210 that stores an information database 590, a control unit 220 that performs various processes such as outputting primary information and secondary information, a communication unit 240 that communicates with the HMD 300, and a It includes a gaze point calculation unit 250 that calculates the coordinates of a gaze point, and a voice recognition unit 260 that recognizes the voice of an HMD wearer or the like. Note that the gaze point calculation unit 250 and the speech recognition unit 260 may be implemented as dedicated hardware, or may be implemented as a software module executed by the control unit 220. The configuration of the HMD 300 is the same as that in the first embodiment.

FIG. 18 is a diagram illustrating the configuration of the secondary information database 590 in this embodiment. In FIG. 18, the secondary information database 590 includes program identification information 591, a time period 592 indicating the period during which the secondary information is valid, a gaze point range 593 for selecting secondary information based on the gaze point of the HMD wearer, It is composed of secondary information 594 and secondary information attributes 595. As shown in the screen at the bottom of FIG. 18, in this example, the coordinate system of the primary information is defined as (0, 0) at the top left and (1920, 1080) at the bottom right, and the coordinate system in the first row of the secondary information database is The gaze point range (300, 50) to (900, 450) is a rectangular range that includes the image of product A, and the gaze point range (1000, 50) to (1600, 450) in the second row includes the image of product B. These are coordinates that indicate the range of a rectangle.

The above is the configuration of the HMD linked display system of this embodiment. The present embodiment will be described below along with the flow of the operation of the present system. Note that the operation of operating the display device 400 to watch TV broadcasts is the same as the operation method of a commonly used TV, so a description thereof will be omitted. Further, the following explanation will be based on the assumption that channel 1 is being viewed.

FIG. 19 shows a camera image 540 and a camera image gazing point 550 acquired by the imaging unit 310 and camera image gazing point detection unit 320 of the HMD 300 when a viewer 911 looks in the direction of the display device 400 via the HMD 300. This is what is shown. In FIG. 19, a viewer 911 is viewing the primary information from a position slightly to the right of the display device 400, and is gazing at product A.

FIG. 20 is a flowchart of a process for selecting secondary information, which is executed in the control unit 220 of the display device 400 when the camera image 540 and camera image intra-gazing point 550 are received from the HMD 300 in this embodiment. In FIG. 20, the control unit 220 first determines whether the program identification information 580 is recorded in the recording unit 210 (step S2411). Since channel 1 is currently being viewed, program identification information 580 exists, and channel 1 is listed as the program being viewed. Next, it is determined whether the camera image gazing point 550 has remained the same for a certain period of time, that is, whether the viewer 911's gazing point has stayed at a specific location (step S2412). Next, a gaze point within the primary information is calculated using the gaze point calculation unit 250 (step S2413). The details of this process are the same as those described using FIGS. 9 to 11 in the first embodiment.

Next, it is determined whether the calculation of the gaze point within the primary information has been successful (step S2414). If the calculation of the gaze point within the primary information is successful, that is, when the gaze point of the viewer 911 is in the direction of the primary information (i.e., the direction of the display device 400), the gaze point within the primary information calculated in step S2413 is stored. (Step S2415). Next, the secondary information database 590 is referred to, and among the secondary information related to the program currently being viewed, the stored primary information point of interest is located within the point of interest range 593 in the secondary information database. , and whose current time is within the time zone 592 of the secondary information database is displayed (step S2416).

On the other hand, if the calculation of the gaze point within the primary information fails in step S2414, the secondary information (that is, the currently displayed secondary information) corresponding to the gaze point within the primary information stored in step S2415 is , an erasure timer is set according to the attribute 595 of the secondary information (step S2417). For example, if the attribute 595 of the secondary information is text, it will be deleted after 60 seconds, and if the attribute 595 of the secondary information is a button, it will be deleted after 0 seconds (that is, immediately). As a result, when the viewer 911 takes his eyes off the display device 400, the button is immediately deleted, but the text continues to be displayed for a certain period of time. The subsequent processing is the same as in the first embodiment.

The above processing enables the operation shown in FIG. 15. That is, the scene that the viewer 911 sees via the HMD 300 is the screen 357 in FIG. 15 at first. Next, when looking at product A, the screen 358 in FIG. 15 is displayed through the processing up to step S2416 in FIG. Next, when the user takes his eyes off the display device 400 and moves to another location, the screen 359 in FIG. 15 appears.

As described above, this embodiment is a display device connected to a head-mounted display, which includes a display section that displays primary information, a projection section that projects primary information, or a signal output section that outputs primary information, and a head-mounted display. It is equipped with a communication unit that can communicate, and a gaze point calculation unit that calculates the position information of the gaze point of the wearer of the head-mounted display with respect to the primary information. Calculates the location information of the wearer's gaze point, selects secondary information related to the location information, and determines when the location information is in the direction of the primary information and when it is determined that the location information is not in the direction of the primary information. The configuration is configured to change the secondary information displayed on the head-mounted display depending on the case.

This makes it possible to obtain supplementary information that cannot be obtained from the broadcast content alone when watching TV, and it also makes it possible to view secondary information even if you take your eyes off the TV, giving viewers more freedom in their actions. can be increased.

The present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Further, it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace other configurations to a part of each embodiment.

100: Projection device, 110: Signal input section, 120: Control section (projection device), 130: Display section (projection device), 200: Display device, 210: Recording section, 220: Control section (display device), 230: Signal output section, 240: communication section (display device), 250: gaze point calculation section, 260: voice recognition section, 270: operation section, 280: display section (display device), 300: head mounted display, 310: imaging section , 320: Camera image gaze point detection unit, 330: Audio acquisition unit, 340: Control unit (head mounted display), 350: Communication unit (head mounted display), 360: Display unit (head mounted display), 400: Display device, 410: IP communication section, 420: tuner section, 430: separation section, 440: display control section, 450: display section (display device), 460: audio control section, 470: speaker, 510: primary information database, 520 : Secondary information database, 530: Primary information, 540: Camera image, 550: Gaze point in camera image, 560: Audio data, 570: Secondary information, 580: Program identification information, 590: Secondary information database, 910: Teacher, 911: Viewer, 920: Student, 930: Screen, 940: Broadcasting equipment, 950: Transmission antenna, 960: Internet

Claims (4)

A head-mounted display device in a head-mounted display cooperation display system including a display device that displays primary information and a head-mounted display device that images the primary information,
a display section that displays information viewable by the wearer ;
an imaging unit that captures a camera image in the direction in which the wearer is facing ;
a camera image gaze point detection unit that detects a camera image gaze point based on the wearer's line of sight direction;
a communication unit capable of communicating with the display device;
Equipped with a control unit,
The display device calculates position information of the wearer's gaze point with respect to the primary information based on the camera image received via the communication unit and the gaze point in the camera image, and the display device calculates position information of the wearer's gaze point with respect to the primary information. outputting secondary information related to the position of the gaze point of the primary information when it is determined that the primary information area is in the area of
When the secondary information is received by the communication unit, the control unit displays the secondary information on the display unit,
When the display device determines that the position information is not in the primary information area when the secondary information is displayed on the display section, the time set according to the attribute of the secondary information Outputting a deletion instruction to delete the secondary information after the elapse of time;
The head mounted display device is characterized in that the control unit controls the display unit to erase the secondary information when the erasure instruction is received by the communication unit . The head mounted display device according to claim 1,
Equipped with an audio acquisition unit that acquires surrounding sounds as audio data,
The display device outputs information related to the keyword as the secondary information when a specific keyword is detected in the audio data received via the communication unit,
The head-mounted display device is characterized in that the control unit displays secondary information related to the keyword on the display unit when the secondary information is received by the communication unit . A head-mounted display device in a head-mounted display cooperation display system including a display device that displays primary information and a head-mounted display device that images the primary information,
a display section that displays information viewable by the wearer;
an imaging unit that captures a camera image in the direction in which the wearer is facing;
a camera image gaze point detection unit that detects a camera image gaze point based on the wearer's line of sight direction;
a communication unit capable of communicating with the display device;
Equipped with a control unit,
The display device calculates position information of the wearer's gaze point with respect to the primary information based on the camera image received via the communication unit and the gaze point in the camera image, and the display device calculates position information of the wearer's gaze point with respect to the primary information. outputting secondary information related to the position of the gaze point of the primary information when it is determined that the primary information area is in the area of
When the secondary information is received by the communication unit, the control unit displays the secondary information on the display unit,
When the secondary information is displayed on the display section and the display device determines that the position information is not in the primary information area, the Outputs instructions to change the transmittance of secondary information,
The head mounted display device is characterized in that the control unit controls the display unit to change the transmittance of the secondary information when the communication unit receives an instruction to change the transmittance of the secondary information. The head mounted display device according to claim 3 ,
Equipped with an audio acquisition unit that acquires surrounding sounds as audio data,
The display device outputs information related to the keyword as the secondary information when a specific keyword is detected in the audio data received via the communication unit,
The head- mounted display device is characterized in that the control unit displays secondary information related to the keyword on the display unit when the secondary information is received by the communication unit .

Images