JP2019174811A - Display system and display method - Google Patents

Abstract

To provide a display system capable of minimizing VR sickness of a user who uses an HMD.SOLUTION: A display system having a display device identifies a direction of the display device and displays a first image which shows a state of a virtual space according to the identified direction on a display screen of the display device and a second image which does not change according to the identified direction on a predetermined region of the display screen.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display system and a display method using a display device mounted on a user's head.

  In recent years, there is a VR (Virtual Reality) technology using a head mounted display (hereinafter referred to as HMD) or the like. For example, a background image that forms a virtual world captured by a virtual camera is displayed on the HMD, and the user feels as if he / she has entered the virtual world (so-called immersion).

  There is also a technique for displaying an image captured by another virtual camera different from the virtual camera so as to be superimposed on the background image so as to be included in the user's field of view separately from the displayed background image.

  However, in the prior art, so-called VR sickness has occurred. This is because, although the user's body is not moving, the background image and the superimposed image move according to the movement of the HMD, thereby causing a shift in the sense of the user's body.

  An object of at least one embodiment of the present invention is to provide a display system capable of reducing VR sickness of a user who uses an HMD.

  According to a non-limiting viewpoint, the display system according to the present invention is a display system provided with a display device, and the orientation specifying means for specifying the orientation of the display device and the orientation specified on the display screen of the display device. A display unit that displays a first image indicating a state of the virtual space according to the display unit and displays a second image that does not change according to the specified orientation in a predetermined region of the display screen. In the display system, when a predetermined object in the first image is present at a position overlapping the predetermined area, the second image is displayed at a position different from the predetermined area.

  According to a non-limiting viewpoint, the display method according to the present invention is a display method executed in a system including a display device, and is specified on the display screen of the display device and the orientation of the display device. Displaying a first image indicating the state of the virtual space according to the orientation, and displaying a second image that does not change according to the specified orientation in a predetermined area of the display screen. In the display method, the second image is displayed at a position different from the predetermined area when a predetermined object in the image is present at a position overlapping the predetermined area.

  Each embodiment of the present invention solves one or more deficiencies.

It is a block diagram which shows the structure of the display system corresponding to at least 1 of embodiment of this invention. It is a flowchart of the execution process corresponding to at least 1 of embodiment of this invention. It is a block diagram which shows the structure of the display system corresponding to at least 1 of embodiment of this invention. It is a block diagram which shows the structure of the display apparatus corresponding to at least 1 of embodiment of this invention. It is a figure which shows the orthogonal coordinate system for specifying a user's attitude | position when the display apparatus corresponding to at least 1 of embodiment of this invention is mounted | worn with a head. It is a block diagram which shows the structure of the computer apparatus corresponding to at least 1 of embodiment of this invention. It is a block diagram which shows the structure of the display system corresponding to at least 1 of embodiment of this invention. It is an example of the execution screen corresponding to at least 1 of embodiment of this invention. It is a figure showing the concept of virtual space corresponding to at least 1 of embodiment of this invention. It is a flowchart of the execution process corresponding to at least 1 of embodiment of this invention. It is a figure explaining the area | region displayed on a subwindow corresponding to at least 1 of embodiment of this invention.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. Hereinafter, the description regarding the effect is one aspect of the effect of the embodiment of the present invention, and is not limited to what is described here. In addition, the order of the processes constituting the flowchart described below is out of order as long as no contradiction or inconsistency occurs in the process contents.

[First embodiment]
The outline of the first embodiment of the present invention will be described. Hereinafter, as a first embodiment of the present invention, a display system including a display device that is used by being worn on the user's head will be described as an example.

  FIG. 1 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. The display system 1 includes at least an orientation specifying unit 101 and a display unit 102.

  The orientation specifying unit 101 has a function of specifying the orientation of the display device. The display unit 102 displays the first image indicating the state of the virtual space according to the specified orientation on the display screen of the display device, and does not change according to the specified orientation in a predetermined area of the display screen. It has a function of displaying the second image.

  The execution process in the first embodiment of the present invention will be described. FIG. 2 is a flowchart of execution processing corresponding to at least one of the embodiments of the present invention.

  In the display system 1, the orientation identifying unit 101 identifies the orientation of the display device used by being mounted on the user's head (step S1). The display system 1 displays the first image indicating the state of the virtual space according to the specified orientation on the display screen of the display device, and does not change according to the specified orientation in a predetermined region of the display screen. The second image is displayed (step S2) and the process ends.

  As one aspect of the first embodiment, the VR sickness of the user who uses the HMD is reduced by displaying a second image that does not change according to the specified orientation in a predetermined area of the display screen. it can.

  In the first embodiment, “attaching to the head” means, for example, putting on the head like a hat, attaching to the face like glasses, or attaching to a part of the head. It is a concept that includes the case of covering the head by extending from a part of the body. “Display device” is, for example, a display device capable of visually recognizing images with both eyes or a single eye. It includes a transmission type that can be seen. “Direction” is a concept including not only the direction and angle of an object, but also a direction related to the amount of change in unit time, such as acceleration. The “virtual space” is a space defined by a program and data, for example, and is a virtual space generated by a computer. The “first image” refers to, for example, an image indicating the state of the virtual space. The “second image” is, for example, an image that is different from an image indicating the state of the virtual space and does not change according to the orientation specified by the display device.

[Second Embodiment]
The outline of the second embodiment of the present invention will be described. Hereinafter, as a second embodiment of the present invention, a display system including a display device used by being mounted on the user's head will be described as an example.

  With regard to the configuration of the display system, the contents described in FIG. 1 can be adopted within a necessary range. Regarding the flowchart of the execution process, the contents described in FIG. 2 can be adopted within a necessary range.

  As an example of the second embodiment, the second image is preferably displayed by satisfying a predetermined condition.

  As one aspect of the second embodiment, by displaying a second image that does not change according to the specified orientation in a predetermined region of the display screen, it is possible to reduce VR sickness of a user who uses the HMD. it can.

  As one aspect of the second embodiment, by displaying the second image by satisfying a predetermined condition, the second image can be displayed without an uncomfortable feeling in an appropriate scene without impairing the visibility of the user with respect to the first image. Can be displayed.

  In the second embodiment, “mounted on the head”, “display device”, “orientation”, “virtual space”, “first image”, and “second image” are the same as those in the first embodiment. The contents described in the form can be adopted within a necessary range.

[Third embodiment]
Next, an outline of the third embodiment of the present invention will be described. As a third embodiment of the present invention, a display system including a display device used by being worn on the user's head will be described as an example.

  FIG. 3 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. FIG. 3A is a diagram illustrating an example of the configuration of the display system. The display system 1 includes a display device 201 equipped with a sensor for detecting movement and orientation, an irradiation device 202 that emits light, a computer device 203 that generates or processes an image displayed on the display device 201, It is comprised from the controller 204 carrying a sensor.

  The display device 201 is preferably a goggle type display device that is attached to the head so as to cover the field of view. The display may be a transmissive display having a high transmittance or a non-transmissive display having a low transmittance. In addition, the display device 201 can specify the posture of the user using a plurality of mounted sensors. The configuration of the display device 201 will be described later.

  The irradiation device 202 is a device that irradiates a laser beam over a wide range. When a laser beam is irradiated around the irradiation device 202, the laser beam is detected by an optical sensor mounted on the display device 201 and the controller 204, and the positions of the display device 201 and the controller 204 are specified. This position specifying method is called “Lighthouse method”. In order to specify the position more accurately, it is preferable that at least two irradiation devices 202 are installed at a predetermined interval.

  The computer device 203 can be connected to the display device 201 and the controller 204 by communication. Communication may be wired or wireless. The computer device 203 may be a device independent of the display device 201 or may be an integrated device incorporated in the display device 201.

  The controller 204 is an input device that includes a button and a trigger and receives an input by a user operation. In addition, an optical sensor is mounted, and the laser beam emitted from the irradiation device 202 can be sensed. Furthermore, the touch sensor is mounted and the input by a user's contact can be received. The controller 204 is used by being gripped by a pair of left and right hands, but may use only one of them.

  FIG. 3B is a diagram illustrating an example regarding the configuration of a display system different from the display system illustrated in FIG. The display system illustrated in FIG. 3B includes a display device 251 including a light emitting unit serving as a light source, a photographing device 252 that captures light emitted from the light emitting unit serving as a light source of the display device 251 and the controller 254, and a display device. The computer apparatus 253 which produces | generates or processes the image displayed on 251 and the controller 254 provided with the light emission part used as a light source are comprised.

  The display device 251 is preferably a goggle-type display device that is attached to the head so as to cover the field of view, like the display device 201. The display may be a transmissive display having a high transmittance or a non-transmissive display having a low transmittance.

  The photographing device 252 is a device for identifying the positions of the display device 251 and the controller 254 by photographing light emitted from a light emitting unit serving as a light source of the display device 251 and the controller 254.

  The computer device 253 can be connected to the display device 251 and the controller 254 by communication. Communication may be wired or wireless.

  The controller 254 is an input device that includes buttons and receives an input by a user operation. A light emitting unit serving as a light source is provided, and the position of the controller 254 is specified by the photographing device 252 photographing the light emitted from the controller 254. The controller is gripped by the hand and preferably used in pairs, but one controller may be used.

  In the following description, in the embodiment of the present invention, the positions of the display device 201 and the controller 204 are specified by the method shown in FIG. 3A, but the present invention is not limited to this method. For example, a method shown in FIG. 3B or a method of specifying the positions of the display device 201 and the controller 204 by wearing on a part of the body other than the head may be used.

  The image displayed on the display device 201 is an image generated or processed by the computer device 203 based on a value measured by a sensor mounted on the display device 201 and a user input operation to the controller 204. .

  Here, the configuration of the display device 201 will be described. FIG. 4 is a block diagram showing a configuration of a display device corresponding to at least one of the embodiments of the present invention. The display device 201 includes a control unit 11, a RAM (Random Access Memory) 12, a storage unit 13, an imaging unit 14, a graphics processing unit 15, a display unit 16, a sensor unit 18, a communication interface 19, an interface unit 20, and a lens 21. Each connected by an internal bus.

  The control unit 11 includes a CPU (Central Processing Unit) and a ROM (Read Only Memory). The control unit 11 executes a program stored in the storage unit 13 and controls the display device 201. The RAM 12 is a work area for the control unit 11. The storage unit 13 is a storage area for storing programs and data.

  The control unit 11 reads the program and data from the RAM 12 and the storage unit 13 and performs processing. The control unit 11 processes the program and data loaded in the RAM 12 to output a drawing command to the graphics processing unit 15. Further, the control unit 11 performs processing based on various values measured by the sensor unit 18 or transmits data to the computer apparatus 3 via the interface unit 20.

  The imaging unit 14 outputs an image captured through the lens 21 to the graphics processing unit 15. When the field of view is covered by mounting the display device 201 on the head, information outside the device cannot be confirmed, and thus the image is captured by the imaging unit 14 on the display screen 17 provided in the display unit 16 described later. By displaying the information, the user can recognize the real space and ensure safety.

  The graphics processing unit 15 performs drawing of one image for each frame. One frame time of an image is, for example, 1/30 second. The graphics processing unit 15 is responsible for a part of arithmetic processing related to drawing and has a role of distributing the load of the entire system.

  The display unit 16 includes a display screen 17 and is installed inside the goggles of the display device 201 that covers the user's field of view. For example, organic EL or inorganic EL is used for the display screen 17, but the display screen 17 is not limited to this, and a terminal device including a display such as a smartphone may be incorporated.

  The sensor unit 18 includes at least a proximity sensor 18a, an infrared sensor 18b, a gyro sensor 18c, and an acceleration sensor 18d. The proximity sensor 18a is installed in the goggle portion of the display device 201, and is used to determine whether or not the user touches the face. A plurality of infrared sensors 18b are installed in the display device 201, and are used to identify the position of the display device 201 by sensing the light emitted from the irradiation device 202. The gyro sensor 18c and the acceleration sensor 18d are used for specifying the posture of the user.

  Here, a method for specifying the posture of the user will be described with reference to the drawings. FIG. 5 is a diagram illustrating an orthogonal coordinate system for specifying the posture of the user when the display device is mounted on the head, corresponding to at least one of the embodiments of the present invention.

  As shown in the figure, XYZ coordinates are defined around the head of the user wearing the display device 201. The vertical direction in which the user stands upright is the Y axis (yaw angle), the direction orthogonal to the Y axis, and the direction connecting the center of the display screen 17 of the display device 201 and the user's head is the Z axis (roll angle), Y axis A direction orthogonal to the Z axis is defined as an X axis (pitch angle). The gyro sensor 18c detects an angle (inclination) around each axis, and the acceleration sensor 18d detects the movement of the display device 201 and displays it on the display screen 17 according to the detected angle and the detected movement. Change the image. The display device 201 can identify the user's posture using values measured by a plurality of sensors (hereinafter referred to as posture information).

  Subsequently, the communication interface 19 can be connected to the communication line 5 wirelessly or by wire, and can receive data via the communication line 5. Data received via the communication interface 19 is loaded into the RAM 12 and arithmetic processing is performed by the control unit 11.

  The interface unit 20 is mainly connected to the computer device 203 and can receive data such as images processed or generated by the computer device 203. In addition, the measurement values of various sensors acquired by the sensor unit 18 can be transmitted to the computer device 203.

  Next, the computer device 203 will be described. FIG. 6 is a block diagram showing a configuration of a computer device corresponding to at least one of the embodiments of the present invention. The computer device 203 includes a control unit 31, a RAM 32, a storage unit 33, a sound processing unit 34, a graphics processing unit 35, a DVD / CD-ROM drive 36, a communication interface 37, an interface unit 38, and a frame memory 39. Connected by internal bus.

  The control unit 31 includes a CPU and a ROM. The control unit 31 executes a program stored in the storage unit 33 and controls the computer device 203. In addition, the control unit 31 includes an internal timer that measures time. The RAM 32 is a work area for the control unit 31. The storage unit 33 is a storage area for storing programs and data.

  The control unit 31 reads the program and data from the RAM 32 and performs processing. The control unit 31 outputs a drawing command to the graphics processing unit 35 by processing the program and data loaded in the RAM 32.

  The sound processing unit 34 is connected to the sound output device 40. When the control unit 31 outputs a sound output instruction to the sound processing unit 34, the sound processing unit 34 outputs a sound signal to the sound output device 40. The sound output device 40 is, for example, a headphone, and a user may wear it on the head together with the display device 201, or a speaker may be incorporated in the display device 201.

  The graphics processing unit 35 draws one image for each frame. One frame time of an image is, for example, 1/90 second. The graphics processing unit 35 is responsible for a part of calculation processing related to drawing and has a role of distributing the load of the entire system. The graphics processing unit 35 of the computer device 203 is included in the display device 201 in order to generate an image to be displayed on the display screen 17 based on the position information and orientation information of the display device 201 and the position information of the controller 204. It requires higher performance than the graphics processing unit 15.

  The DVD / CD-ROM drive 36 reads data from the disk 41, loads it into the RAM 32, and performs arithmetic processing by the control unit 31. The type of the disc 41 is not limited to a DVD or a CD, but may be a Blu-ray (registered trademark) disc or other standards.

  The communication interface 37 can be connected to the communication line 5 wirelessly or by wire, and can receive data via the communication line 5. Data received via the communication interface 37 is loaded into the RAM 32 in the same manner as data read from the disk 41, and arithmetic processing is performed by the control unit 31.

  The interface unit 38 is mainly connected to the display device 201 and can transmit data such as an image processed or generated by the computer device 203.

  In the above description, the display device 201 and the computer device 203 are described as separate devices. However, the display device 201 may perform all processing of the computer device 203. In this case, the computer device 203 is unnecessary, and the controller 204 is connected to the display device 201 by communication.

  Next, the configuration of the display system in the embodiment of the present invention will be described. FIG. 7 is a block diagram showing a configuration of a display system corresponding to at least one of the embodiments of the present invention. The display system 1 includes an angle information measuring unit 301, a position and orientation specifying unit 302, an information transmitting unit 303, an information receiving unit 304, a line-of-sight specifying unit 305, an intersection determining unit 306, a time measuring unit 307, an elapsed time determining unit 308, and an object superposition. At least a determination unit 309, a sub window position changing unit 310, a sub window image generation unit 311, a display image generation unit 312, an image transmission unit 313, an image reception unit 314, and an image display unit 315 are provided.

  The angle information measurement unit 301 has a function of measuring angle information from the attitude of the display device 201. The position / orientation specifying unit 302 has a function in which an optical sensor mounted on the display device 201 senses a laser beam emitted from the irradiation device 202 and specifies the position and orientation of the display device 201. The information transmission unit 303 has a function of transmitting the angle information measured in the display device 201 and the specified position and orientation information to the computer device 203.

  The information receiving unit 304 has a function of receiving information transmitted from the information transmitting unit 303. The line-of-sight specifying unit 305 has a function of specifying the user's line of sight in the virtual space based on the information received by the information receiving unit 304. The intersection determination unit 306 has a function of determining whether or not the object arranged in the virtual space and the line of sight specified by the line-of-sight specifying unit 305 intersect.

  The time measuring unit 307 has a function of measuring the time during which the object and the line of sight have crossed continuously after it is determined that the object and the line of sight intersect. The elapsed time determination unit 308 has a function of determining whether or not the time measured by the time measuring unit 307 has passed a predetermined time. The object superimposition determination unit 309 has a function of determining whether or not the display area of the sub window and the object arranged in the virtual space are superimposed.

  The sub window position changing unit 310 has a function of changing the display position of the sub window when it is determined that the display area of the sub window and the object overlap. The sub window image generation unit 311 has a function of generating an image to be displayed on the sub window. The display image generation unit 312 has a function of generating an image to be displayed.

  The image transmission unit 313 has a function of transmitting the display image generated by the display image generation unit 312 to the display device 201. The image receiving unit 314 has a function of receiving the display image transmitted by the image transmitting unit 313. The image display unit 315 has a function of displaying the display image received by the image receiving unit 314 on the display screen 17.

  Next, execution processing in the third embodiment of the present invention will be described. An example of the third embodiment of the present invention is a system for browsing content. More specifically, the content is composed of a plurality of sheets of paper, and is data related to a comic that progresses according to the development of the story, and is a virtual that represents a single comic by a user operation for each comic. It is a display system that can enter the space.

  FIG. 8 is an example of an execution screen corresponding to at least one of the embodiments of the present invention. In FIG. 8A, the display screen 17 displays the status of one frame of a cartoon, and a character 401 and a sub window 402 are displayed. The sub window 402 displays an image captured by a virtual camera that is different from the virtual camera that captures the image displayed on the display screen 17. In FIG. 8, the face of the character 401 is enlarged and displayed.

  FIG. 8B is an example of an execution screen in which the user changes the direction of the display device 201 by moving the line of sight (hereinafter also referred to as the visual axis) to the left in the situation of FIG. As the user moves his / her head, the virtual space on the left side of FIG. 8A is displayed on the display screen 17 for the user. In this case, the sub window 402 remains displayed in a predetermined area on the display screen 17, and the position does not change according to the movement of the user's head. The predetermined area may be a partial area included in the display screen 17 and is not limited to the lower right of the display screen 17.

  The sub-window 402 may be designed so that it is displayed in association with a single frame of a comic so that the sub-window 402 is displayed whenever a virtual space representing the single frame of the comic is entered. That is, the image displayed on the display screen 17 may have, for example, a flag for displaying the sub window 402 as attribute information. Or you may display according to the content of an image so that the subwindow 402 may be displayed when the content which an image shows is a specific scene.

  The sub window 402 may be designed to be displayed when a predetermined condition is satisfied. The predetermined condition is, for example, when an input operation is accepted by the controller 204, or by matching the direction specified from the orientation of the display device 201 with the visual axis of the virtual camera, and the visual axis intersects with an object in the virtual space. For example, it may be determined that it has been performed.

  Next, generation of an image to be displayed on the display screen in the embodiment of the present invention will be described. When the program is executed, the control unit 31 of the computer device 203 causes the graphics processing unit 35 to output a virtual space defined by the program. Further, the position of the user wearing the display device 201 on the head and the position of the controller 204 are specified based on the position and direction in which the light beam emitted from the irradiation device 202 is received. Further, the attitude of the display device 201 is specified by the measurement values of various sensors of the sensor unit 18 of the display device 201.

  The virtual space defined by the program will be described. FIG. 9 is a diagram showing the concept of the virtual space corresponding to at least one of the embodiments of the present invention. When the virtual space 70 defined by the program is represented by a hemispherical space as illustrated, the visual field of the user 80 wearing the display device 201 on the head corresponds to the visible region 81. That is, an image corresponding to the visible region 81 is displayed on the display screen 17 of the display device 201. For example, when the user 80 turns to the left, the visible region 81 moves to the left according to the movement of the user 80. Similarly, in the vertical direction, for example, when the user 80 faces upward, the visible region 81 moves upward in accordance with the movement of the user 80.

[Execution process]
FIG. 10 is a flowchart of execution processing corresponding to at least one of the embodiments of the present invention. First, the display apparatus 201 measures angle information from the attitude of the display apparatus 201 (step S11). The angle information is measured by various sensors included in the sensor unit 18 of the display device 201.

  Next, when the optical sensor mounted on the display device 201 senses the laser beam emitted from the irradiation device 202, the display device 201 specifies the position and orientation of the display device 201 (step S12). The processing of step S11 and step S12 may be performed at predetermined intervals, and may be transmitted to the computer device 203 at every measurement or specific time to generate a display image to be displayed on the display screen 17 of the display device 201.

  Next, the display apparatus 201 transmits the angle information measured in step S11 and the information related to the orientation and position specified in step S12 to the computer apparatus 203 (step S13). Next, the computer device 203 receives the information (step S14), and based on the angle information measured in step S11 and the information on the position and orientation specified in step S12, the computer device 203 displays the user's line of sight in the virtual space. Specify (step S15).

  The user's line of sight in the virtual space is based on the angle information (information about the tilt with respect to the horizontal plane), the information about the position, and the information about the direction (information about which direction on a plane parallel to the horizontal plane). This is the direction in which it is estimated that the user is looking in the virtual space in the visible region 81 specified in the above. That is, the specified direction corresponds to the visual axis of the virtual camera in the virtual space. For example, the viewpoint of the virtual camera is set at the position of the display device 201, and based on the orientation of the display device 201, the direction of the line of sight on the plane parallel to the horizontal plane in the virtual world is determined from the viewpoint. The inclination of the line of sight with respect to the horizontal plane can be specified based on the orientation information of the display device 201. Furthermore, by performing perspective transformation from the viewpoint of the virtual camera onto the virtual screen, an image corresponding to the user's line of sight can be generated. In this way, without causing the user to be conscious, in the virtual world, an object that exists in the direction that the user is assumed to be viewing is displayed on the display device 201, and the convenience of the user is improved. It becomes possible to immerse in the virtual world more.

  Incidentally, the display device 201 may have a function of tracking the line of sight of the user wearing the display device 201. In this case, the line of sight can be calculated from the movement of the user's eyeball.

  Next, the computer device 203 determines whether or not the user's line of sight identified in step S15 intersects with the object arranged in the virtual space (step S16). The object is, for example, a character, an enemy character, an item, or an object displayed in the virtual space, which can be arranged in the virtual space. Crossing means that a line of sight intersects with a boundary surface forming an object, and is a concept including contact between a line and a surface. For the intersection determination between the line segment and the object in the three-dimensional space, for example, a method for obtaining the intersection of the object and the ray (ray) in the ray tracing method can be used. Other methods are also possible, and in the embodiment of the present invention, the method of intersection determination is not limited.

  If it is determined that the user's line of sight intersects with a predetermined object in the virtual space (YES in step S16), the computer device 203 counts time with an internal timer provided in the control unit 31 of the computer device 203. Is started (step S17). When it is determined that the user's line of sight does not intersect with a predetermined object in the virtual space (NO in step S16), the sub window is not displayed. When the sub window has already been displayed, the sub window may not be displayed at the timing when the line of sight and the predetermined object no longer intersect.

  Next, the computer device 203 determines whether or not a predetermined time has elapsed after the predetermined object and the line of sight have continued to intersect based on the timing started in step S17 (step S18). If it is determined that the predetermined time has elapsed (YES in step S18), the computer apparatus 203 determines whether or not the display area of the sub window overlaps with a predetermined object arranged in the virtual space ( Step S19). The predetermined object determined in step S19 refers to, for example, an object having an important meaning with respect to content, an object whose visibility is deteriorated by being superimposed, and the like. If the predetermined time has not elapsed (NO in step S18), the sub window is not displayed. When the subwindow is already displayed, the existing subwindow may be kept displayed.

  If it is determined that the display area of the sub window and the object arranged in the virtual space are overlapped (YES in step S19), computer apparatus 203 changes the display position of the sub window (step S20). When it is determined that the display area of the sub window and the object arranged in the virtual space do not overlap (NO in step S19), the display position of the sub window is not changed.

  Here, the display position of the subwindow will be described. The subwindow is set in advance to be displayed in a predetermined area of the display screen 17, and is preferably displayed at a predetermined position when the subwindow is displayed. FIG. 11 is a diagram for explaining an area displayed in the sub-window corresponding to at least one of the embodiments of the invention. When it is assumed that the visual axis is directed to the center of the display screen 17, a region or a line having a predetermined positional relationship with respect to the visual axis is a predetermined region.

  FIG. 11A is a diagram for explaining a case where the predetermined area 91 is defined as one of the equally divided areas around the visual axis 90. As illustrated, the area to be divided is not limited to four divisions. That is, the predetermined area may be determined by specifying the number of divisions and the direction of the area.

  FIG. 11B is a diagram illustrating a case where the predetermined region 91 is defined within a range of a predetermined angle α with the visual axis 90 as a center point. Here, α is an arbitrary value. That is, the predetermined area may be determined by designating the angle α.

  FIG. 11C is a diagram illustrating a case where the predetermined area 91 is defined within a predetermined distance β2 with the visual axis 90 as the center point. Here, β1 and β2 are arbitrary values. That is, the predetermined area may be determined by specifying the length. As for the length, only β1 may be designated, or β1 and β2 may be designated, and a ring-shaped region may be designated as illustrated.

  FIG. 11D is a diagram for explaining a case where the predetermined straight line 92 is calculated by the inner product of the vectors 95 and 96 starting from the visual axis 90. That is, the predetermined area may be determined by any point on the straight line. In FIG. 11D, the calculation is performed using a vector as an example, but the calculation may be performed using a trigonometric function or the like.

  Although not shown, the predetermined area may be set by designating coordinates. For example, when an XY coordinate system is set on the display screen 17 and the upper left corner of the screen is set to coordinates (0, 0), if the end point (200, 200) is specified from the start point (100, 100), the start point and end point are set. A rectangular range as a diagonal line may be set as the predetermined region. In other words, the predetermined area is an area defined with respect to the visual axis and is not dependent on a change in the position and orientation of the display device 201.

  The change of the position of the sub window in step S20 means, for example, changing the position of the sub window so that the object arranged in the virtual space and the sub window do not overlap. More specifically, it is effective when, for example, it is considered that there is a serious influence such that the visibility of the user deteriorates due to superimposition of the sub-window and the object in the virtual space, and the story cannot be understood. The sub window position may be changed by changing a predetermined area and changing the display position of the sub window. For example, the sub window displayed at the lower right of the screen may be displayed at the upper right of the screen. Moreover, you may avoid impairing visibility by shifting a position a little within a predetermined area | region.

  Next, the computer apparatus 203 generates an image to be displayed on the subwindow (step S21). The generated image may be an arbitrary image, and may be an image related to the object determined to intersect the line of sight in step S16.

  Next, the computer device 203 generates an image corresponding to the direction of the line of sight identified in step S15 (step S22). The image corresponding to the direction of the line of sight is an image captured by a virtual camera. Next, the computer apparatus 203 transmits the sub window image generated in step S21 and the image corresponding to the line-of-sight direction generated in step S22 to the display apparatus 201 (step S23).

  The display device 201 receives the image transmitted in step S23 (step S24). The display device 201 displays the image received in step S24 (step S25) and ends.

  The execution process may be repeatedly executed at a predetermined timing. The predetermined timing can be performed, for example, at every frame rate of the screen or at a multiple of the frame rate.

  In step S25, the display apparatus 201 displays the received subwindow image and an image corresponding to the direction of the line of sight in a superimposed manner. Here, the sub-window image is displayed at a position determined by the relationship with the line of sight, and does not change according to the position and orientation specified in step S12. That is, even when the user moves the head, the subwindow image is controlled so as to be fixedly displayed at a predetermined position on the screen without being interlocked with the movement of the head.

  In step S25, when the sub window is already displayed, the image displayed in the sub window may be updated.

  In the third embodiment, the light beam for specifying the positions of the display device 201 and the controller 204 is a laser beam, for example, an infrared ray. However, the light beam is not limited to this, and is a visible ray. Alternatively, other invisible light may be used.

  In the third embodiment, the sub window is displayed when the line of sight continues to cross a predetermined object for a predetermined time. However, the line of sight and the predetermined line are not determined without determining whether the predetermined time has passed. The sub window may be displayed only by the determination regarding the intersection with the object.

  In the third embodiment, when the predetermined object in the image corresponding to the visual axis is superimposed on the predetermined area of the sub window, the position of the sub window is changed. May not display a subwindow.

  As one aspect of the third embodiment, VR sickness of the user who uses the HMD can be reduced by displaying a sub-window image that does not change according to the specified orientation in a predetermined area of the display screen. .

  As one aspect of the third embodiment, by displaying the second image by satisfying a predetermined condition, the second image can be displayed without an uncomfortable feeling in an appropriate scene without impairing the visibility of the user with respect to the first image. Can be displayed.

  As one aspect of the third embodiment, a subwindow image is displayed without a user performing an input operation by displaying a subwindow image when a predetermined object arranged in the virtual space intersects the line of sight. And user convenience can be improved.

  As one aspect of the third embodiment, the sub-window image is displayed when the user's visual axis and the predetermined object continuously intersect for a predetermined time, thereby suppressing the display of the sub-window not intended by the user. In addition, it is possible to display only the subwindow that the user wants to display, thereby improving convenience.

  As one aspect of the third embodiment, by displaying an image related to a predetermined object intersecting with the user's visual axis in a sub-window, the user can know information related to the predetermined object without performing an input operation. In addition, there is an advantage that a place where the user's visual axis is hit can be recognized.

  As one aspect of the third embodiment, displaying a subwindow image associated with an image corresponding to the visual axis by displaying the subwindow image when information about the image corresponding to the visual axis satisfies a predetermined condition. And the effect of combining images can be produced. For example, in the case of content such as comics, when the main character is locked in a freezer, the background image in the freezer is displayed as an image corresponding to the visual axis, and when the thermometer image is displayed in the subwindow, only the background image is displayed. It is possible to give the user a feeling of tension and to produce an effect by a combination of images.

  As one aspect of the third embodiment, when a predetermined object in an image corresponding to the visual axis exists at a position overlapping a predetermined area where the sub window is displayed, the sub window is positioned different from the predetermined area. By displaying on the screen, the visibility of the user can be improved.

  As one aspect of the third embodiment, when a predetermined object in an image corresponding to the visual axis exists at a position overlapping a predetermined area of the sub window, the sub window is not displayed, thereby improving the visibility of the user. Can be improved.

  In the third embodiment, “mounted on the head”, “display device”, “orientation”, “virtual space”, “first image”, and “second image” are the same as those in the first embodiment. The contents described in the form can be adopted within a necessary range.

  In the third embodiment, “overlap” means, for example, that a line and an object intersect. “Information relating to the first image” refers to, for example, information on the content represented by the first image and attribute information of the first image.

[Appendix]
The above description of the embodiments described the following invention so that a person having ordinary knowledge in the field to which the invention belongs can carry out the invention.

[1] A display system including a display device used by being worn on a user's head,
Orientation specifying means for specifying the orientation of the display device;
A first image indicating the state of the virtual space is displayed on the display screen of the display device according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen A display system.

[2] The display system according to [1], wherein the second image is displayed by satisfying a predetermined condition.

[3] The direction specified by the specifying means corresponds to the visual axis of the virtual camera in the virtual space,
The first image is generated by perspective-transforming a virtual space on a virtual screen with a virtual camera as a viewpoint,
The predetermined condition is that the visual axis overlaps with a predetermined object in a virtual space.
The display system according to [2].

[4] The display system according to [3], wherein the predetermined condition is that a state in which the visual axis overlaps with a predetermined object in a virtual space continues for a predetermined time.

[5] The display system according to [3] or [4], wherein the second image is an image related to a predetermined object.

[6] The display system according to any one of [2] to [5], wherein the predetermined condition is that information regarding the first image satisfies a predetermined condition.

[7] The display means is configured to display the second image at a position different from the predetermined area when a predetermined object in the first image exists at a position overlapping the predetermined area. The display system according to any one of [1] to [6].

[8] The display unit according to any one of [1] to [7], wherein the display unit does not display the second image when a predetermined object in the first image exists at a position overlapping the predetermined region. Display system.

[9] A display method in a system including a display device used by being worn on a user's head,
Identifying the orientation of the display device;
A first image indicating the state of the virtual space is displayed on the display screen of the display device according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen Display method to do and perform.

1 Display System 11 Control Unit 12 RAM
DESCRIPTION OF SYMBOLS 13 Storage part 14 Imaging part 15 Graphics processing part 16 Display part 17 Display screen 18 Sensor part 18a Sensor 18b Infrared sensor 18c Gyro sensor 18d Acceleration sensor 19 Communication interface 20 Interface part 201 Display apparatus 202 Irradiation apparatus 203 Computer apparatus 204 Controller 21 Lens 251 Display device 252 Imaging device 253 Computer device 254 Controller 31 Control unit 32 RAM
33 Storage Unit 34 Sound Processing Unit 35 Graphics Processing Unit 36 CD-ROM Drive 37 Communication Interface 38 Interface Unit 39 Frame Memory 40 Sound Output Device 41 Disc 5 Communication Line

Claims (8)

A display system comprising a display device,
Orientation specifying means for specifying the orientation of the display device;
A first image indicating the state of the virtual space is displayed on the display screen of the display device according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen Display means for
The display unit is configured to display a second image at a position different from the predetermined area when a predetermined object in the first image is present at a position overlapping the predetermined area. The display system according to claim 1, wherein the second image is displayed by satisfying a predetermined condition. The direction specified by the specifying means corresponds to the visual axis of the virtual camera in the virtual space,
The first image is generated by perspective-transforming a virtual space on a virtual screen with a virtual camera as a viewpoint,
The predetermined condition is that the visual axis overlaps with a predetermined object in a virtual space.
The display system according to claim 2. The display system according to claim 3, wherein the predetermined condition is that a state in which the visual axis overlaps with a predetermined object in a virtual space continues for a predetermined time. The display system according to claim 3 or 4, wherein the second image is an image related to a predetermined object. The display system according to claim 2, wherein the predetermined condition is that information regarding the first image satisfies a predetermined condition. The display device is equipped with sensors including a gyro sensor and an acceleration sensor,
The display system according to any one of claims 1 to 6, wherein the orientation specifying unit specifies the orientation of the display device using a value measured by the sensor. A display method executed in a system including a display device,
Identifying the orientation of the display device;
A first image indicating the state of the virtual space is displayed on the display screen of the display device according to the specified orientation, and a second image that does not change according to the specified orientation is displayed in a predetermined area of the display screen Do and do
A display method of displaying a second image at a position different from the predetermined area when a predetermined object in the first image is present at a position overlapping the predetermined area.

Images