JP6670965B2 - Display system and display method - Google Patents

Description

  The present invention relates to a display system and a display method using a display device mounted on a user's head.

  In recent years, there is a VR (Virtual Reality) technology using a head-mounted display (hereinafter, referred to as an HMD) or the like. For example, a background image captured by a virtual camera and constituting a virtual world is displayed on the HMD to give the user a sense of entering the virtual world (so-called immersion).

  There is also a technique of displaying an image captured by another virtual camera different from the virtual camera, separately from the displayed background image, so as to be superimposed on the background image so as to be included in the user's field of view.

  However, in the related art, so-called VR sickness has occurred. This is because the background image and the superimposed image move in response to the movement of the HMD even though the user's body is not moving, causing a shift in the sense of the user's body.

  An object of at least one embodiment of the present invention is to provide a display system capable of reducing VR sickness of a user using an HMD.

  According to a non-limiting aspect, a display system according to the present invention is a display system including a display device, a direction specifying unit that specifies a direction of the display device, and a display screen of the display device, the display screen having the specified direction. In response to displaying a first image indicating the state of the virtual space, in a predetermined area of the display screen, display means for displaying a second image that does not change according to the specified orientation, the display means, A display system for displaying a second image at a position different from the predetermined region when a predetermined object in the first image exists at a position overlapping the predetermined region.

  According to a non-limiting aspect, a display method according to the present invention is a display method that is executed in a system including a display device, and specifies an orientation of the display device, and specifies a display screen of the display device. Displaying a first image indicating the state of the virtual space in accordance with the orientation, and displaying a second image that does not change in accordance with the specified orientation in a predetermined area of the display screen. A display method for displaying a second image at a position different from the predetermined area when a predetermined object in the image is present at a position overlapping the predetermined area.

  Each embodiment of the present invention resolves one or more deficiencies.

FIG. 1 is a block diagram illustrating a configuration of a display system corresponding to at least one of the embodiments of the present invention. 9 is a flowchart of an execution process corresponding to at least one of the embodiments of the present invention. FIG. 1 is a block diagram illustrating a configuration of a display system corresponding to at least one of the embodiments of the present invention. FIG. 1 is a block diagram illustrating a configuration of a display device corresponding to at least one of the embodiments of the present invention. FIG. 3 is a diagram illustrating an orthogonal coordinate system for specifying a user's posture when a display device is mounted on a head, corresponding to at least one of the embodiments of the present invention. FIG. 1 is a block diagram showing a configuration of a computer device corresponding to at least one of the embodiments of the present invention. FIG. 1 is a block diagram illustrating a configuration of a display system corresponding to at least one of the embodiments of the present invention. It is an example of an execution screen corresponding to at least one of the embodiments of the present invention. FIG. 2 is a diagram illustrating a concept of a virtual space corresponding to at least one of the embodiments of the present invention. 9 is a flowchart of an execution process corresponding to at least one of the embodiments of the present invention. FIG. 5 is a diagram illustrating a region displayed in a sub window, corresponding to at least one of the embodiments of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Hereinafter, the description related to the effect is one aspect of the effect of the embodiment of the present invention, and the present invention is not limited to this. In addition, the order of each process constituting the flowchart described below is not specified as long as no inconsistency or inconsistency occurs in the process contents.

[First embodiment]
An outline of the first embodiment of the present invention will be described. Hereinafter, as a first embodiment of the present invention, a display system including a display device used by being mounted on a user's head will be described as an example.

  FIG. 1 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. The display system 1 includes at least an orientation identification unit 101 and a display unit 102.

  The direction specifying unit 101 has a function of specifying the direction of the display device. The display unit 102 displays the first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation, and does not change in a predetermined area of the display screen according to the specified orientation. It has a function of displaying the second image.

  The execution process according to the first embodiment of the present invention will be described. FIG. 2 is a flowchart of an execution process corresponding to at least one of the embodiments of the present invention.

  The display system 1 uses the orientation identification unit 101 to identify the orientation of the display device that is used by being worn on the head of the user (step S1). The display system 1 displays the first image indicating the state of the virtual space on the display screen of the display device according to the specified orientation, and does not change in a predetermined area of the display screen according to the specified orientation. The second image is displayed (step S2), and the process ends.

  As one aspect of the first embodiment, by displaying a second image that does not change according to the specified orientation in a predetermined area of the display screen, it is possible to reduce VR sickness of a user using the HMD. it can.

  In the first embodiment, "wearing on the head" means, for example, putting on the head like a hat, putting on the face like glasses, or attaching to a part of the other head. It is a concept that includes a case where the head is extended from a part of the body. A "display device" is, for example, a display device that can visually recognize an image with both eyes or a single eye, and is a non-transmissive type in which the outside state cannot be seen when worn, or a transparent member such as eyeglasses. It includes a transmission type that can be seen. The “direction” is a concept including, for example, not only a direction and an angle of an object but also a direction related to a change amount per unit time such as acceleration. The “virtual space” is, for example, a space defined by programs and data, and is a virtual space generated by a computer. The “first image” refers to, for example, an image indicating the state of the virtual space. The “second image” is, for example, an image that is different from the image indicating the state of the virtual space and does not change according to the orientation specified by the display device.

[Second embodiment]
An outline of a second embodiment of the present invention will be described. Hereinafter, as a second embodiment of the present invention, a display system including a display device used by being mounted on a user's head will be described as an example.

  Regarding the configuration of the display system, the contents described in FIG. 1 can be adopted within a necessary range. Regarding the flowchart of the execution process, the contents described in FIG. 2 can be adopted within a necessary range.

  As an example of the second embodiment, it is preferable that the second image is displayed when a predetermined condition is satisfied.

  As one aspect of the second embodiment, it is possible to reduce a VR sickness of a user using an HMD by displaying a second image that does not change according to the specified orientation in a predetermined area of a display screen. it can.

  As one aspect of the second embodiment, by displaying the second image by satisfying a predetermined condition, without impairing the user's visibility to the first image, the second image can be displayed in an appropriate scene without discomfort. Can be displayed.

  In the second embodiment, “attached to the head”, “display device”, “direction”, “virtual space”, “first image”, and “second image” are the same as in the first embodiment, respectively. The contents described in the embodiment can be adopted within a necessary range.

[Third embodiment]
Next, an outline of a third embodiment of the present invention will be described. As a third embodiment of the present invention, a display system including a display device used by being mounted on a user's head will be described as an example.

  FIG. 3 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. FIG. 3A is a diagram illustrating one example of the configuration of the display system. The display system 1 includes a display device 201 equipped with a sensor for detecting movement and orientation, an irradiation device 202 that irradiates light, a computer device 203 that generates or processes an image displayed on the display device 201, And a controller 204 equipped with a sensor.

  The display device 201 is preferably a goggle-type display device that is mounted on the head so as to cover the field of view. The display may be a transmissive display having a high transmittance or a non-transmissive display having a low transmittance. Further, the display device 201 can specify the posture of the user by using a plurality of mounted sensors. The configuration of the display device 201 will be described later.

  The irradiation device 202 is a device that irradiates a laser beam over a wide range. When a laser beam is irradiated around the irradiation device 202, the laser beam is sensed by an optical sensor mounted on the display device 201 and the controller 204, and the positions of the display device 201 and the controller 204 are specified. This position identification method is called a “Lighthouse method”. It is preferable that at least two irradiation devices 202 are provided at a predetermined interval to more accurately specify the position.

  The computer device 203 can be connected to the display device 201 and the controller 204 by communication. Communication may be wired or wireless. The computer device 203 may be a device independent of the display device 201, or may be an integrated device incorporated in the display device 201.

  The controller 204 is an input device that includes a button and a trigger, and receives input by a user operation. In addition, an optical sensor is mounted, and can detect a laser beam emitted from the irradiation device 202. Furthermore, a touch sensor is mounted, and an input by a user's contact can be received. The controller 204 is used as a pair while being held by the left and right hands, but only one of them may be used.

  FIG. 3B is a diagram illustrating an example of a configuration of a display system different from the display system illustrated in FIG. The display system illustrated in FIG. 3B includes a display device 251 including a light emitting unit serving as a light source, an imaging device 252 configured to capture light emitted from the light emitting unit serving as a light source of the display device 251 and the controller 254, and a display device. The image processing apparatus includes a computer device 253 that generates or processes an image displayed at 251 and a controller 254 that includes a light emitting unit serving as a light source.

  Like the display device 201, the display device 251 is preferably a goggle-type display device mounted on the head so as to cover the field of view. The display may be a transmissive display having a high transmittance or a non-transmissive display having a low transmittance.

  The photographing device 252 is a device for specifying the positions of the display device 251 and the controller 254 by photographing light emitted from a light emitting unit serving as a light source of the display device 251 and the controller 254.

  The computer device 253 can be connected to the display device 251 and the controller 254 by communication. Communication may be wired or wireless.

  The controller 254 is an input device that includes a button and receives an input by a user operation. A light emitting unit serving as a light source is provided, and the position of the controller 254 is specified by the image capturing device 252 capturing the light emitted from the controller 254. The controllers are gripped by hand and are preferably used in pairs, but a single controller may be used.

  In the following description, in the embodiment of the present invention, the positions of the display device 201 and the controller 204 are specified by the method shown in FIG. 3A, but the present invention is not limited to this method. For example, the method shown in FIG. 3B or the method of attaching to a part of the body other than the head and specifying the positions of the display device 201 and the controller 204 may be used.

  The image displayed on the display device 201 is an image generated or processed by the computer device 203 based on a value measured by a sensor mounted on the display device 201 and a user input operation to the controller 204. .

  Here, the configuration of the display device 201 will be described. FIG. 4 is a block diagram showing a configuration of a display device corresponding to at least one of the embodiments of the present invention. The display device 201 includes a control unit 11, a RAM (Random Access Memory) 12, a storage unit 13, an imaging unit 14, a graphics processing unit 15, a display unit 16, a sensor unit 18, a communication interface 19, an interface unit 20, and a lens 21. And are connected by an internal bus.

  The control unit 11 includes a CPU (Central Processing Unit) and a ROM (Read Only Memory). The control unit 11 executes a program stored in the storage unit 13 and controls the display device 201. The RAM 12 is a work area of the control unit 11. The storage unit 13 is a storage area for storing programs and data.

  The control unit 11 reads a program and data from the RAM 12 and the storage unit 13 and performs processing. The control unit 11 outputs a drawing command to the graphics processing unit 15 by processing the program and data loaded in the RAM 12. Further, the control unit 11 performs processing based on various values measured by the sensor unit 18, or transmits data to the computer device 3 via the interface unit 20.

  The imaging unit 14 outputs an image captured through the lens 21 to the graphics processing unit 15. When the field of view is covered by attaching the display device 201 to the head, information outside the device cannot be confirmed, and thus the image is captured by the imaging unit 14 on the display screen 17 provided in the display unit 16 described below. By displaying the information, the user can recognize the real space, and security can be ensured.

  The graphics processing unit 15 executes rendering of one image in frame units. One frame time of an image is, for example, 1/30 second. The graphics processing unit 15 has a role to handle a part of the arithmetic processing related to drawing and to distribute the load of the entire system.

  The display unit 16 includes a display screen 17 and is installed inside goggles of the display device 201 that covers the field of view of the user. For example, an organic EL or an inorganic EL is used for the display screen 17, but the display screen 17 is not limited to this, and a terminal device having a display such as a smartphone may be incorporated.

  The sensor unit 18 includes at least a proximity sensor 18a, an infrared sensor 18b, a gyro sensor 18c, and an acceleration sensor 18d. The proximity sensor 18a is installed in a goggle portion of the display device 201, and is used to determine whether or not there is contact with a user's face. A plurality of infrared sensors 18b are provided in the display device 201, and are used to identify the position of the display device 201 by sensing light beams emitted from the irradiation device 202. The gyro sensor 18c and the acceleration sensor 18d are used to specify the posture of the user.

  Here, a method of specifying the posture of the user will be described with reference to the drawings. FIG. 5 is a diagram illustrating an orthogonal coordinate system for specifying the posture of the user when the display device is mounted on the head, corresponding to at least one of the embodiments of the present invention.

  As shown in the drawing, XYZ coordinates are defined around the head of the user wearing the display device 201. The vertical direction in which the user stands upright is the Y axis (yaw angle), the direction orthogonal to the Y axis, and the direction connecting the center of the display screen 17 of the display device 201 to the user's head is the Z axis (roll angle), and the Y axis And a direction orthogonal to the Z axis is defined as an X axis (pitch angle). The gyro sensor 18c detects an angle (tilt) around each axis, and the acceleration sensor 18d detects the movement of the display device 201, and displays the movement on the display screen 17 according to the detected angle and the detected movement. Change the image. The display device 201 can specify the user's posture using values measured by a plurality of sensors (hereinafter, referred to as posture information).

  Subsequently, the communication interface 19 can be connected to the communication line 5 wirelessly or by wire, and can receive data via the communication line 5. The data received via the communication interface 19 is loaded into the RAM 12, and the control unit 11 performs an arithmetic process.

  The interface unit 20 is mainly connected to the computer device 203 and can receive data such as images processed or generated in the computer device 203. Further, the measurement values of various sensors acquired by the sensor unit 18 can be transmitted to the computer device 203.

  Next, the computer device 203 will be described. FIG. 6 is a block diagram showing a configuration of a computer device corresponding to at least one of the embodiments of the present invention. The computer device 203 includes a control unit 31, a RAM 32, a storage unit 33, a sound processing unit 34, a graphics processing unit 35, a DVD / CD-ROM drive 36, a communication interface 37, an interface unit 38, and a frame memory 39. Connected by internal bus.

  The control unit 31 includes a CPU and a ROM. The control unit 31 executes a program stored in the storage unit 33 and controls the computer device 203. Further, the control unit 31 includes an internal timer that measures time. The RAM 32 is a work area of the control unit 31. The storage unit 33 is a storage area for storing programs and data.

  The control unit 31 reads a program and data from the RAM 32 and performs processing. The control unit 31 outputs a drawing command to the graphics processing unit 35 by processing the program and data loaded in the RAM 32.

  The sound processing unit 34 is connected to the sound output device 40. When the control unit 31 outputs a sound output instruction to the sound processing unit 34, the sound processing unit 34 outputs a sound signal to the sound output device 40. The sound output device 40 is, for example, headphones, and may be worn by the user on the head together with the display device 201, or a speaker may be incorporated in the display device 201.

  The graphics processing unit 35 executes drawing of one image in frame units. One frame time of an image is, for example, 1/90 second. The graphics processing unit 35 is responsible for a part of arithmetic processing related to drawing, and has a role of distributing the load of the entire system. The graphics processing unit 35 of the computer device 203 is included in the display device 201 to generate an image to be displayed on the display screen 17 based on the position information and orientation information of the display device 201 and the position information of the controller 204. It requires higher performance than the graphics processing unit 15.

  The DVD / CD-ROM drive 36 reads data from the disk 41, loads the data into the RAM 32, and performs arithmetic processing by the control unit 31. The type of the disk 41 is not limited to a DVD or CD, but may be a Blu-ray (registered trademark) disk or another standard.

  The communication interface 37 can be connected to the communication line 5 wirelessly or by wire, and can receive data via the communication line 5. The data received via the communication interface 37 is loaded into the RAM 32 in the same manner as the data read from the disk 41, and the control unit 31 performs arithmetic processing.

  The interface unit 38 is mainly connected to the display device 201 and can transmit data such as images processed or generated in the computer device 203.

  In the above description, the display device 201 and the computer device 203 are described as separate devices. However, the display device 201 may perform all the processing of the computer device 203. In this case, the computer device 203 becomes unnecessary, and the controller 204 is connected to the display device 201 by communication.

  Next, the configuration of the display system according to the embodiment of the present invention will be described. FIG. 7 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. The display system 1 includes an angle information measurement unit 301, a position and orientation identification unit 302, an information transmission unit 303, an information reception unit 304, a line-of-sight identification unit 305, an intersection determination unit 306, a clock unit 307, an elapsed time determination unit 308, and object superposition. It includes at least a determination unit 309, a sub-window position change unit 310, a sub-window image generation unit 311, a display image generation unit 312, an image transmission unit 313, an image reception unit 314, and an image display unit 315.

  The angle information measurement unit 301 has a function of measuring angle information from the attitude of the display device 201. The position and orientation specifying unit 302 has a function of specifying a position and an orientation of the display device 201 by an optical sensor mounted on the display device 201 detecting a laser beam emitted from the irradiation device 202. The information transmitting unit 303 has a function of transmitting the angle information measured by the display device 201 and the specified position and orientation information to the computer device 203.

  The information receiving unit 304 has a function of receiving the information transmitted by the information transmitting unit 303. The line-of-sight specifying unit 305 has a function of specifying the user's line of sight in the virtual space based on the information received by the information receiving unit 304. The intersection determining unit 306 has a function of determining whether or not the object arranged in the virtual space intersects with the line of sight specified by the line of sight specifying unit 305.

  The timer 307 has a function of counting the time of continuous intersection after it is determined that the object and the line of sight intersect. The elapsed time determination unit 308 has a function of determining whether the time measured by the clock unit 307 has passed a predetermined time. The object superimposition determining unit 309 has a function of determining whether or not the display area of the subwindow and the object arranged in the virtual space are superimposed.

  The subwindow position changing unit 310 has a function of changing the display position of the subwindow when it is determined that the display area of the subwindow and the object overlap. The sub-window image generation unit 311 has a function of generating an image to be displayed in the sub-window. The display image generation unit 312 has a function of generating an image to be displayed.

  The image transmission unit 313 has a function of transmitting the display image generated by the display image generation unit 312 to the display device 201. The image receiving unit 314 has a function of receiving the display image transmitted by the image transmitting unit 313. The image display unit 315 has a function of displaying the display image received by the image receiving unit 314 on the display screen 17.

  Next, an execution process according to the third embodiment of the present invention will be described. As an example of the third embodiment of the present invention, there is a system for browsing contents. More specifically, the content is composed of a plurality of sheets of paper, and is data relating to a manga that progresses in accordance with the development of the story. It is a display system that can enter the space.

  FIG. 8 is an example of an execution screen corresponding to at least one of the embodiments of the present invention. In FIG. 8A, the display screen 17 displays the state of one frame of a cartoon, and displays a character 401 and a sub window 402. In the sub window 402, an image captured by a virtual camera different from the virtual camera that captures the image displayed on the display screen 17 is displayed. In FIG. 8, the face of the character 401 is enlarged and displayed.

  FIG. 8B is an example of an execution screen in which the user moves the line of sight (hereinafter, also referred to as the visual axis) to the left to change the orientation of the display device 201 from the situation of FIG. 8A. When the user moves the head, the virtual space on the left side of FIG. 8A with respect to the user is displayed on the display screen 17. In this case, the sub-window 402 remains displayed in a predetermined area on the display screen 17, and its position does not change according to the movement of the user's head. The predetermined area may be a partial area included in the display screen 17 and is not limited to the lower right of the display screen 17.

  The sub-window 402 may be designed to be stored in association with one frame of the comic so that it is always displayed when entering the virtual space representing one frame of the comic. That is, the image displayed on the display screen 17 may have, for example, a flag for displaying the sub window 402 as the attribute information. Alternatively, when the content indicated by the image is a specific scene, the sub-window 402 may be displayed according to the content of the image such that the sub-window 402 is displayed.

  The sub-window 402 may be designed to be displayed when a predetermined condition is satisfied. The predetermined condition is, for example, when an input operation is received by the controller 204, or the direction specified from the attitude of the display device 201 is associated with the visual axis of the virtual camera, and the visual axis intersects with the object in the virtual space. And the like.

  Next, generation of an image to be displayed on the display screen according to the embodiment of the present invention will be described. When the program is executed, the control unit 31 of the computer device 203 causes the graphics processing unit 35 to output the virtual space defined by the program. Further, the position of the user wearing the display device 201 on the head and the position of the controller 204 are specified based on the position and direction of the light beam emitted from the irradiation device 202. Further, the attitude of the display device 201 is specified based on the measured values of various sensors of the sensor unit 18 of the display device 201.

  The virtual space defined by the program will be described. FIG. 9 is a diagram illustrating a concept of a virtual space corresponding to at least one of the embodiments of the present invention. When the virtual space 70 defined by the program is represented by a hemispherical space as shown in the figure, the visual field of the user 80 wearing the display device 201 on the head corresponds to the visible region 81. That is, an image corresponding to the visible region 81 is displayed on the display screen 17 of the display device 201. For example, if the user 80 turns left, the visible area 81 moves to the left according to the movement of the user 80. Similarly, in the vertical direction, for example, when the user 80 faces upward, the visible area 81 moves upward according to the movement of the user 80.

[Execution process]
FIG. 10 is a flowchart of an execution process corresponding to at least one of the embodiments of the present invention. First, the display device 201 measures angle information from the attitude of the display device 201 (step S11). The angle information is measured by various sensors included in the sensor unit 18 of the display device 201.

  Next, the display device 201 specifies the position and the orientation of the display device 201 by the optical sensor mounted on the display device 201 detecting the laser beam emitted from the irradiation device 202 (step S12). The processing in steps S11 and S12 may be performed at predetermined intervals, and may be transmitted to the computer device 203 at each measurement or at a specific time to generate a display image to be displayed on the display screen 17 of the display device 201.

  Next, the display device 201 transmits the angle information measured in step S11 and the information on the orientation and position specified in step S12 to the computer device 203 (step S13). Next, the computer device 203 receives the information (Step S14), and based on the angle information measured in Step S11 and the information on the position and orientation specified in Step S12, changes the user's gaze in the virtual space. It is specified (step S15).

  The user's line of sight in the virtual space is based on angle information (information on tilt with respect to a horizontal plane), information on position, and information on orientation (information on which direction on a plane parallel to the horizontal plane) of display device 201. This is the direction in which it is presumed that the user is looking in the virtual space in the visible area 81 specified by the user. That is, the specified direction corresponds to the visual axis of the virtual camera in the virtual space. For example, the viewpoint of the virtual camera is set at the position of the display device 201, and based on the orientation of the display device 201, from the viewpoint as a starting point, it is determined in which direction the line of sight is directed on a plane parallel to the horizontal plane in the virtual world. It is possible to specify and to determine the inclination of the line of sight with respect to the horizontal plane based on the posture information of the display device 201. Further, by performing perspective transformation from the viewpoint of the virtual camera onto the virtual screen, an image corresponding to the line of sight of the user can be generated. By doing so, the objects existing in the direction in which the user is supposed to be watching in the virtual world are displayed on the display device 201 without being conscious of the user, and the convenience of the user is improved. It becomes possible to be more immersed in a virtual world.

  By the way, the display device 201 may have a function of tracking the line of sight of the user wearing the display device 201. In this case, the line of sight can be calculated from the movement of the user's eyeball.

  Next, the computer device 203 determines whether or not the line of sight of the user specified in step S15 intersects with the object arranged in the virtual space (step S16). The object is, for example, a character, an enemy character, an item, an object displayed in the virtual space, or the like, which can be arranged in the virtual space. The intersection means that the line of sight intersects with the boundary surface forming the object, and is a concept that also includes that the line and the surface are in contact with each other. The intersection between the line segment and the object in the three-dimensional space can be determined by, for example, a method for finding the intersection between the object and a ray (ray) in the ray tracing method. Other methods are also possible, and in the embodiment of the present invention, the method of intersection determination is not limited.

  If it is determined that the user's line of sight intersects with a predetermined object in the virtual space (YES in step S16), computer device 203 measures the time using an internal timer provided in control unit 31 of computer device 203. Is started (step S17). If it is determined that the user's line of sight does not intersect with the predetermined object in the virtual space (NO in step S16), the sub window is not displayed. If the sub-window is already displayed, the sub-window may not be displayed at the timing when the line of sight does not cross the predetermined object.

  Next, the computer apparatus 203 determines whether or not the time during which the intersection between the predetermined object and the line of sight continues for a predetermined time has elapsed, based on the clocking started in step S17 (step S18). If it is determined that the predetermined time has elapsed (YES in step S18), computer device 203 determines whether or not the display area of the subwindow overlaps with a predetermined object arranged in the virtual space (step S18). Step S19). The predetermined object determined in step S19 refers to, for example, an object that has an important meaning with respect to the content, an object whose visibility becomes poor when superimposed, and the like. If the predetermined time has not elapsed (NO in step S18), the sub window is not displayed. When the sub-window is already displayed, the state where the existing sub-window remains displayed may be maintained.

  If it is determined that the display area of the sub-window and the object arranged in the virtual space overlap (YES in step S19), computer device 203 changes the display position of the sub-window (step S20). If it is determined that the display area of the subwindow and the object arranged in the virtual space do not overlap (NO in step S19), the display position of the subwindow is not changed.

  Here, the display position of the sub window will be described. The sub-window is set in advance to be displayed in a predetermined area of the display screen 17, and it is preferable that the sub-window is displayed at a predetermined position when the sub-window is displayed. FIG. 11 is a diagram illustrating a region displayed in a sub window, corresponding to at least one of the embodiments of the present invention. Assuming that the visual axis is directed to the center of the display screen 17, an area or a line having a predetermined positional relationship with the visual axis is the predetermined area.

  FIG. 11A is a diagram illustrating a case where the predetermined area 91 is determined as one of equally divided areas around the visual axis 90. The area to be divided is not limited to four as illustrated. That is, the predetermined area may be determined by designating the number of divisions and the direction of the area.

  FIG. 11B is a diagram illustrating a case where the predetermined area 91 is defined within a range of a predetermined angle α with the visual axis 90 as a center point. Here, α is an arbitrary value. That is, the predetermined area may be determined by designating the angle α.

  FIG. 11C is a diagram illustrating a case where the predetermined area 91 is defined within a range of a predetermined distance β2 around the visual axis 90 as a center point. Here, β1 and β2 are arbitrary values. That is, the predetermined area may be determined by designating the length. As the length, only β1 may be specified, or β1 and β2 may be specified, and a ring-shaped region may be specified as illustrated.

  FIG. 11D is a diagram illustrating a case where a predetermined straight line 92 is calculated by an inner product of vectors 95 and 96 starting from the visual axis 90. That is, the predetermined area may be determined at any point on the straight line. In FIG. 11D, the calculation is performed using a vector as an example, but the calculation may be performed using a trigonometric function or the like.

  Although not shown, the predetermined area may be set by designating coordinates. For example, if an XY coordinate system is set on the display screen 17 and the upper left corner of the screen is set to the coordinates (0, 0), when the start point (100, 100) and the end point (200, 200) are designated, the start point and the end point are set. A rectangular area that is a diagonal may be used as the predetermined area. That is, the predetermined area is an area determined with respect to the visual axis, and is an area that does not depend on a change in the position and orientation of the display device 201.

  Changing the position of the sub-window in step S20 means, for example, changing the position of the sub-window so that the object arranged in the virtual space does not overlap the sub-window. More specifically, this is effective, for example, when it is considered that there is a serious influence such as the user's visibility being deteriorated due to the superimposition of the sub-window and the object in the virtual space and the story cannot be understood. To change the position of the sub-window, a predetermined area may be changed to change the display position of the sub-window. For example, the sub window displayed at the lower right of the screen may be displayed at the upper right of the screen. Further, by slightly shifting the position within the predetermined area, the visibility may be prevented from being impaired.

  Next, the computer device 203 generates an image to be displayed in the sub window (Step S21). The generated image may be an arbitrary image, and may be an image related to the object determined to intersect with the line of sight in step S16.

  Next, the computer device 203 generates an image corresponding to the direction of the line of sight specified in step S15 (step S22). The image corresponding to the direction of the line of sight is an image captured by the virtual camera. Next, the computer device 203 transmits the sub-window image generated in step S21 and the image corresponding to the line of sight generated in step S22 to the display device 201 (step S23).

  The display device 201 receives the image transmitted in step S23 (step S24). The display device 201 displays the image received in step S24 (step S25), and ends.

  The execution process may be repeatedly executed at a predetermined timing. The predetermined timing can be set, for example, at each frame rate of the screen or at a timing that is a multiple of the frame rate.

  In step S25, the display device 201 superimposes and displays the received sub-window image and the image corresponding to the line of sight. Here, the sub-window image is displayed at a position determined based on the relationship with the line of sight, and does not change according to the position and orientation specified in step S12. That is, even when the user moves his / her head, the sub-window image is controlled so as not to be linked to the movement of the head but to be fixedly displayed at a predetermined position on the screen.

  If the sub-window has already been displayed in step S25, the image displayed in the sub-window may be updated.

  In the third embodiment, the light beam for specifying the positions of the display device 201 and the controller 204 is a laser beam, for example, an infrared ray, but the light beam is not limited to this, and is a visible light beam. Or other invisible light rays.

  In the third embodiment, the sub-window is displayed when the line of sight continues to intersect with the predetermined object for a predetermined time. However, without determining whether or not the predetermined time has elapsed, the sub-window is displayed. The sub-window may be displayed only by determining the intersection with the object.

  In the third embodiment, when the predetermined object in the image corresponding to the visual axis is superimposed on the predetermined area of the subwindow, the position of the subwindow is changed. May not display the subwindow.

  As one aspect of the third embodiment, VR sickness of a user using the HMD can be reduced by displaying a sub-window image that does not change according to the specified orientation in a predetermined area of the display screen. .

  As one aspect of the third embodiment, by displaying the second image by satisfying a predetermined condition, the second image is displayed in an appropriate scene without discomfort without impairing the user's visibility to the first image. Can be displayed.

  As one aspect of the third embodiment, a sub-window image is displayed when a predetermined object arranged in a virtual space and a line of sight intersect, thereby displaying the sub-window image without performing an input operation by a user. And the convenience of the user can be improved.

  As one aspect of the third embodiment, when a visual axis of a user and a predetermined object continuously intersect for a predetermined time, a sub-window image is displayed to suppress display of a sub-window not intended by the user. However, it is possible to improve the convenience by enabling only the sub-window that the user wants to display.

  As one aspect of the third embodiment, by displaying an image related to a predetermined object that intersects the visual axis of the user in the sub-window, the user can know information about the predetermined object without performing an input operation. There is also an advantage that a place where the user's visual axis is hit can be recognized.

  As one aspect of the third embodiment, displaying a sub-window image associated with an image corresponding to a visual axis by displaying a sub-window image when information about an image corresponding to the visual axis satisfies a predetermined condition. And an effect of combining images can be produced. For example, in the case of content such as cartoons, when the main character is trapped in a freezer, a background image in the freezer is displayed as an image corresponding to the visual axis, and an image of a thermometer is displayed in a sub window, and only the background image is displayed. Rather than giving the user a sense of tension, the effect of the combination of images can be produced.

  As one aspect of the third embodiment, when a predetermined object in an image corresponding to a visual axis is present at a position overlapping a predetermined area where a subwindow is displayed, the subwindow is located at a position different from the predetermined area. , The visibility of the user can be improved.

  As one aspect of the third embodiment, when a predetermined object in an image corresponding to the visual axis is present at a position overlapping a predetermined area of the subwindow, the visibility of the user is improved by not displaying the subwindow. Can be improved.

  In the third embodiment, “attached to the head”, “display device”, “orientation”, “virtual space”, “first image”, and “second image” refer to the first embodiment, respectively. The contents described in the embodiment can be adopted within a necessary range.

  In the third embodiment, “overlap” means, for example, that a line and an object intersect. “Information about the first image” refers to, for example, information of the content represented by the first image or attribute information of the first image.

[Appendix]
In the above description of the embodiments, the following invention has been described so that a person having ordinary knowledge in the field to which the invention belongs can carry out the invention.

[1] A display system including a display device used by being worn on a user's head,
Orientation specifying means for specifying the orientation of the display device;
On the display screen of the display device, a first image indicating the state of the virtual space is displayed according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen. A display system comprising:

[2] The display system according to [1], wherein the second image is displayed when a predetermined condition is satisfied.

[3] The direction specified by the specifying unit corresponds to a visual axis of the virtual camera in the virtual space,
The first image is generated by performing a perspective transformation of a virtual space on a virtual screen with a virtual camera as a viewpoint,
The predetermined condition is that the visual axis overlaps a predetermined object in a virtual space,
The display system according to [2].

[4] The display system according to [3], wherein the predetermined condition is that a state in which the visual axis overlaps a predetermined object in a virtual space continues for a predetermined time.

[5] The display system according to [3] or [4], wherein the second image is an image related to a predetermined object.

[6] The display system according to any one of [2] to [5], wherein the predetermined condition is that information on the first image satisfies a predetermined condition.

[7] The display means displays the second image at a position different from the predetermined area when the predetermined object in the first image is present at a position overlapping the predetermined area. The display system according to any one of [1] to [6].

[8] The display unit according to any one of [1] to [7], wherein the display unit does not display the second image when a predetermined object in the first image exists at a position overlapping the predetermined area. Display system.

[9] A display method in a system including a display device used by being worn on a user's head,
Determining the orientation of the display device;
On the display screen of the display device, a first image indicating the state of the virtual space is displayed according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen. How to do and display.

DESCRIPTION OF SYMBOLS 1 Display system 11 Control part 12 RAM
Reference Signs List 13 storage unit 14 imaging unit 15 graphics processing unit 16 display unit 17 display screen 18 sensor unit 18a sensor 18b infrared sensor 18c gyro sensor 18d acceleration sensor 19 communication interface 20 interface unit 201 display device 202 irradiation device 203 computer device 204 controller 21 lens 251 display device 252 photographing device 253 computer device 254 controller 31 control unit 32 RAM
33 storage section 34 sound processing section 35 graphics processing section 36 CD-ROM drive 37 communication interface 38 interface section 39 frame memory 40 sound output device 41 disk 5 communication line

Claims (8)

A display system including a display device,
Orientation specifying means for specifying the orientation of the display device;
On the display screen of the display device, a first image indicating the state of the virtual space is displayed according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen. Display means for performing
A display system, wherein the display means displays a second image at a position different from the predetermined area when a predetermined object in the first image is present at a position overlapping the predetermined area. The display system according to claim 1, wherein the second image is displayed when a predetermined condition is satisfied. The direction specified by the specifying unit corresponds to the visual axis of the virtual camera in the virtual space,
The first image is generated by performing a perspective transformation of a virtual space on a virtual screen with a virtual camera as a viewpoint,
The predetermined condition is that the visual axis overlaps a predetermined object in a virtual space,
The display system according to claim 2. The display system according to claim 3, wherein the predetermined condition is that a state in which the visual axis overlaps a predetermined object in a virtual space continues for a predetermined time. The display system according to claim 3, wherein the second image is an image related to a predetermined object. The display system according to claim 2, wherein the predetermined condition is that information on the first image satisfies a predetermined condition. The display device is equipped with sensors including a gyro sensor and an acceleration sensor,
The display system according to any one of claims 1 to 6, wherein the direction specifying unit specifies the direction of the display device using a value measured by the sensor. A display method executed in a system including a display device,
Determining the orientation of the display device;
On the display screen of the display device, a first image indicating the state of the virtual space is displayed according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen. Do and do
A display method, wherein a second image is displayed at a position different from the predetermined area when a predetermined object in the first image exists at a position overlapping the predetermined area.

Images