WO2024150615A1

WO2024150615A1 - Image generation device, image generation method, and image generation program

Info

Publication number: WO2024150615A1
Application number: PCT/JP2023/045397
Authority: WO
Inventors: 貴一池田; 武中川; 崇天田; 和城中村
Original assignee: 株式会社ソニー・インタラクティブエンタテインメント
Priority date: 2023-01-10
Filing date: 2023-12-19
Publication date: 2024-07-18
Also published as: JP2024098257A

Abstract

This image generation device 200 according to the present invention is equipped with: a reference determination unit 202 for determining a reference position and/or a reference direction, which are the position and direction in which a user is positioned in a virtual space, on the basis of an instruction input which differs from the instruction input of the position and/or direction of a head-mounted display 100; a viewpoint determination unit 203 for determining the viewpoint position and/or viewpoint direction of the user in the virtual space on the basis of the reference position and/or reference direction; a rendering unit 204 for generating a three-dimensional image by rendering an object in virtual space on the basis of the viewpoint position and/or viewpoint direction; and a re-projection unit 205b for executing re-projection processing for transforming the three-dimensional image so as to match a new reference position and/or reference direction.

Description

Image generating device, image generating method, and image generating program

The present invention relates to image generation technology.

Players wear a head-mounted display connected to a game console on their head and operate a controller while looking at the screen displayed on the head-mounted display. When wearing a head-mounted display, the user can see nothing but the image displayed on the head-mounted display, which increases the sense of immersion in the world of the images and has the effect of further enhancing the entertainment value of the game. In addition, when virtual reality (VR) images are displayed on the head-mounted display and the user wearing the head-mounted display rotates their head, a 360-degree virtual space that can be seen all around is displayed. This further increases the sense of immersion in the images and improves the operability of applications such as games.

When a head-mounted display is equipped with a head tracking function in this way and VR images are generated by changing the viewpoint position and direction in conjunction with the position and direction of the user's head, there is a delay between the generation and display of the VR images. This can result in a discrepancy between the position and direction of the user's head assumed when the image is generated and the position and direction of the user's head at the time the VR image is displayed on the head-mounted display. As a result, the user may experience a feeling of sickness (known as "VR sickness (Virtual Reality Sickness)"). For this reason, a reprojection process is commonly used to correct the rendered image to match the position and direction of the head-mounted display when the image is displayed.

The user's viewpoint position and viewpoint direction in the virtual space change not only in response to the position and direction of the head mounted display, but also in response to changes in the position and direction in which the user is positioned in the virtual space. This position and direction in which the user is positioned may change between the time an image is generated and the time the next image is generated. In this case, the change in the user's position and direction during this time may not be reflected, and the three-dimensional image may differ from what the user expected, causing the user to feel uncomfortable.

In view of the above, the object of the present invention is to provide a technology that prevents a user from feeling uncomfortable about a three-dimensional image due to a misalignment of the user's position in a virtual space.

In order to solve the above problem, an image generating device according to one aspect of the present invention includes a reference determination unit that determines at least one of a reference position and a reference direction, which are a position and a direction in which a user is to be placed in a virtual space, based on an instruction input different from at least one of an instruction input of a position and a direction of a head-mounted display; a viewpoint determination unit that determines at least one of a viewpoint position and a viewpoint direction of the user in the virtual space, based on at least one of the reference position and the reference direction; a rendering unit that renders an object in the virtual space, based on at least one of the viewpoint position and the viewpoint direction, to generate a three-dimensional image; and a reprojection unit that executes a reprojection process that converts the three-dimensional image to match at least one of the new reference position and the reference direction.

An image generating method according to another aspect of the present invention includes the steps of: determining at least one of a user's reference position and reference direction in a virtual space based on an instruction input different from at least one of an instruction input of a position and an orientation of a head-mounted display; determining at least one of a viewpoint position and a viewpoint direction of the user in the virtual space based on the at least one of the reference position and the reference direction; rendering an object in the virtual space based on the at least one of the viewpoint position and the viewpoint direction to generate a three-dimensional image; and executing a reprojection process to convert the three-dimensional image to match the new at least one of the reference position and the reference direction.

Another aspect of the image generation program of the present invention is an image generation program for causing a computer to execute the steps of: determining at least one of a user's reference position and reference direction in a virtual space based on an instruction input different from at least one of an instruction input of a position and direction of a head-mounted display; determining at least one of a viewpoint position and viewpoint direction of the user in the virtual space based on at least one of the reference position and reference direction; rendering an object in the virtual space based on at least one of the viewpoint position and viewpoint direction to generate a three-dimensional image; and executing a reprojection process to convert the three-dimensional image to match at least one of the new reference position and reference direction.

In addition, any combination of the above, or mutual substitution of the components or expressions of the present invention among methods, devices, programs, temporary or non-temporary storage media recording programs, systems, etc. are also valid aspects of the present invention.

The present invention makes it possible to provide a technology that prevents users from feeling uncomfortable about three-dimensional images due to misalignment of the user in a virtual space.

FIG. 1 is an external view of a head mounted display. FIG. 1 is a configuration diagram of an image generating system. FIG. 3 is a configuration diagram of the image generating device of FIG. 2. 10A to 10C are diagrams for explaining the relationship between a reference position and a reference direction, an HMD position and an HMD direction, and a viewpoint position and a viewpoint direction. 4 is a flowchart showing a flow of processing relating to image generation in the image generating device of the first embodiment. 11A to 11C are diagrams for explaining the relationship between the amount of change in the reference position and reference direction, the amount of change in the HMD position and HMD direction, and a new viewpoint position and viewpoint direction. FIG. 4 is a diagram for explaining the flow of a reprojection process according to the first embodiment. 10 is a flowchart showing a flow of processing relating to image generation in an image generating device according to a second embodiment. FIG. 11 is a diagram for explaining the flow of a reprojection process according to the second embodiment.

1 is an external view of a head mounted display (HMD) 100. The HMD 100 is a display device that is worn on a user's head to view still images and videos displayed on a display and to listen to audio and music outputted from headphones.

The position of the head of a user wearing the HMD 100 and the direction of the head's rotation angle and tilt can be measured using a gyro sensor or acceleration sensor built into or attached externally to the HMD 100. In the HMD 100, for example, the position and direction of the head are detected based on the position and direction of the head at the time the HMD 100 is turned on. The HMD 100 may further be provided with a camera that photographs the user's eyes. The camera mounted on the HMD 100 can detect the user's gaze direction, pupil movement, blinking, etc.

FIG. 2 is a configuration diagram of an image generation system according to this embodiment. The image generation system 1 includes an HMD 100, an image generation device 200, and an input device 300.

The image generating device 200 of this embodiment is a game machine. The image generating device 200 may be connected to a server via a network. In this case, the server may provide the image generating device 200 with an online application such as a game in which multiple users can participate via the network.

The image generating device 200 basically processes the content program, generates three-dimensional images, and transmits them to the HMD 100. The content program and data are read by a media drive (not shown) from a ROM medium (not shown) on which the content's application software, such as a game, and license information are recorded. This ROM medium is a read-only recording medium such as an optical disk, a magneto-optical disk, or a Blu-ray (registered trademark) disk. The image generating device 200 in one embodiment determines the viewpoint position and viewpoint direction based on the position and direction of the head of the user wearing the HMD 100 (hereinafter referred to as the HMD position and HMD direction) and the reference position and reference direction described below, generates three-dimensional images of the content at a predetermined rate so that the field of view is accordingly obtained, and transmits them to the HMD 100.

The HMD 100 receives and displays the three-dimensional image generated by the image generating device 200. The three-dimensional image displayed on the HMD 100 may be an image captured in advance by a camera, an image generated by computer graphics such as a game image, or a live image from a remote location delivered via a network. The three-dimensional image displayed on the HMD 100 may also be a VR image, an AR (augmented reality) image, an MR (mixed reality) image, etc.

The input device 300 supplies various inputs received from the user to the image generating device 200. For example, the input device 300 supplies the image generating device 200 with instruction inputs for changing the user's position and direction in the virtual space. The input device 300 is realized by any of the typical input devices, such as a game controller, a keyboard, a mouse, a joystick, a video camera that captures the user's gestures, a touchpad provided on the display screen of a flat panel display, or a combination of these. The input device 300 of this embodiment is an example of a user input device.

FIG. 3 is a functional configuration diagram of the image generating device 200 of FIG. 2. The diagram is a block diagram focusing on functions, and these functional blocks can be realized in various forms by hardware only, software only, or a combination of both. At least a part of the functions of the image generating device 200 may be implemented by the HMD 100. Alternatively, at least a part of the functions of the image generating device 200 may be implemented by a server connected to the image generating device 200 via a network. The image generating device 200 comprises a game control unit 201, a reference determination unit 202, a viewpoint determination unit 203, a rendering unit 204, an image processing unit 205, a transmission unit 206, and a storage unit 207.

The game control unit 201 executes the game program read from the storage unit 207 to progress the game. The game control unit 201 transmits instruction inputs to the reference determination unit 202 for progressing through a predetermined scene in the virtual space, and changes the reference position and reference direction of the user in the virtual space, which will be described later. The "predetermined scene in the virtual space" refers to, for example, a scene that represents an action of an object on the user in the virtual space, or an event scene that occurs in the virtual space. The "action of an object on the user in the virtual space" refers to, for example, an action that moves the user in a direction that is predetermined according to the positional relationship between the user and the object, such as when the user collides with an object of a physical object, or when the user is pushed or pulled by an enemy character object, or an action that moves the user according to a predetermined progression path of a vehicle, such as when the user rides on a vehicle object. The "event scene that occurs in the virtual space" refers to a scene in which the user is forced to move in a predetermined pattern, such as a movie scene.

The reference determination unit 202 determines a reference position and a reference direction, which are the position and direction in which the user is placed in the virtual space, based on an instruction input different from the instruction input of the position and direction of the HMD 100 (hereinafter, HMD position and HMD direction). The reference position and the reference direction change regardless of the change in the HMD position and the HMD direction. For example, the reference position and the reference direction change based on at least one of the instruction input via the input device 300 and the instruction input from the game control unit 201 for progressing a predetermined scene in the virtual space. For example, the reference position is set to the center of the user's body in the virtual space (e.g., the center of the torso, etc.), and the reference direction is set to the direction of the user's body in the virtual space (e.g., the direction of the torso, etc.). Note that setting the reference position to the center of the user's body is merely an example, and the reference position can be set to any position based at least on the user's position.

For example, the reference determination unit 202 reads out a reference position and a reference direction in the virtual space from the storage unit 207, and determines a reference position and a reference direction based on an instruction input received from at least one of the input device 300 and the game control unit 201. The reference determination unit 202 updates the reference position and reference direction in the storage unit 207 with the determined reference position and reference direction.

The viewpoint determination unit 203 acquires the HMD position and HMD direction as well as the reference position and reference direction, and determines the user's viewpoint position and viewpoint direction in the virtual space based on these. For example, the viewpoint determination unit 203 determines the viewpoint position and viewpoint direction based on the reference position and reference direction so as to reflect the relative position and relative direction of the HMD 100 with respect to the reference position and reference direction.

The relationship between the reference position and reference direction, the HMD position and HMD direction, and the viewpoint position and viewpoint direction will be explained using FIG. 4. FIG. 4 is a diagram of a three-dimensional virtual space consisting of mutually orthogonal x-, y-, and z-axes, viewed from the z direction. The rotation angle α is about the x-axis, the rotation angle β is about the y-axis, and the rotation angle γ is about the z-axis. As shown in FIG. 4, when the reference position is determined to be (x1, y1, z1) and the HMD position is detected to be (x2, y2, z2), the viewpoint position is determined to be (x1+x2, y1+y2, z1+z2). When the reference direction is determined to be (α1, β1, γ1) and the HMD direction is detected to be (α2, β2, γ2), the viewpoint direction is determined to be (α1+α2, β1+β2, γ1+γ2).

The rendering unit 204 reads image data required for generating an image from the storage unit 207, and generates a three-dimensional image by rendering objects in a virtual space. For example, the rendering unit 204 generates a three-dimensional image by rendering objects in a virtual space that are visible from the viewpoint position and in the viewpoint direction of a user wearing the HMD 100, based on the user's viewpoint position and viewpoint direction determined by the viewpoint determination unit 203.

The image processing unit 205 processes the rendered image as necessary to generate a three-dimensional image (hereinafter referred to as an HMD image) to be displayed on the HMD 100, and provides it to the transmission unit 206. The image processing unit 205 includes a post-processing unit 205a, a reprojection unit 205b, and a distortion processing unit 205c.

The post-processing unit 205a performs post-processing such as depth of field adjustment, tone mapping, and anti-aliasing on the images supplied from the rendering unit 204, so that the three-dimensional images appear natural and smooth.

The reprojection unit 205b executes the reprojection process. The reprojection process includes a first reprojection process and a second reprojection. The first reprojection process and the second reprojection process will be described later. The reprojection unit 205b supplies the three-dimensional image that has been subjected to the reprojection process to the distortion processing unit 205c.

The distortion processing unit 205c performs processing to distort the three-dimensional image by modifying the image to match the distortion caused by the optical system of the HMD 100.

The transmission unit 206 transmits the HMD image generated in the image processing unit 205 to the HMD 100. In this embodiment, the transmission unit 206 transmits the HMD image that has been subjected to the first reprojection process, and then transmits the HMD image that has been subjected to the second reprojection process. As a result, the transmission unit 206 in this embodiment causes the HMD 100 to display the HMD image that has been subjected to the first reprojection process, and then causes the HMD 100 to display the HMD image that has been subjected to the second reprojection process. The transmission unit in this embodiment is an example of a display control unit.

The memory unit 207 stores the reference position and reference direction, data required for generating images, and various programs for executing various processes.

The process S100 for image generation in the image generating device of this embodiment will be explained using FIG. 5.

In step S101, the reference determination unit 202 determines a reference position and a reference direction. The reference determination unit 202 of the present embodiment determines a reference position and a reference direction based on an instruction input received from at least one of the input device 300 and the game control unit 201 so as to be a position and a direction at the timing of rendering a three-dimensional image (hereinafter, referred to as drawing timing).
The reference determination unit 202 supplies the determined reference position and reference direction to the viewpoint determination unit 203. Furthermore, when the reference determination unit 202 determines a new reference position and reference direction, it supplies the newly determined reference position and reference direction to the reprojection unit 205b.

In step S102, the viewpoint determination unit 203 sets the viewpoint position and viewpoint direction based on the HMD position and HMD direction and the reference position and reference direction. The viewpoint determination unit 203 supplies the determined viewpoint position and viewpoint direction to the rendering unit 204.

In step S103, the rendering unit 204 renders the object in the virtual space based on the determined viewpoint position and viewpoint direction to generate a three-dimensional image. The rendering unit 204 supplies the generated three-dimensional image to the post-processing unit 205a of the image processing unit 205.

In step S104, the post-processing unit 205a performs post-processing on the rendered three-dimensional image. The post-processing unit 205a supplies the three-dimensional image that has been subjected to the post-processing to the reprojection unit 205b.

In step S105, the reprojection unit 205b acquires a new first HMD position and first HMD direction. The new first HMD position and first HMD direction here are the HMD position and HMD direction obtained at the time of drawing the three-dimensional image.

In step S106, the reprojection unit 205b executes a first reprojection process on the post-processed three-dimensional image based on the amount of change from the HMD position and HMD orientation to the new first HMD position and first HMD orientation. The first reprojection process is a process of converting the post-processed three-dimensional image to match the new HMD position and HMD orientation based on the HMD position and HMD orientation and the new HMD position and HMD orientation. The first reprojection process can reduce delays due to deviations between the HMD position and HMD orientation in step S102 and the new first HMD position and first HMD orientation obtained at the drawing timing. The reprojection unit 205b supplies the post-processed and first reprojection processed three-dimensional image to the distortion processing unit 205c.

In step S107, the distortion processing unit 205c performs distortion processing on the three-dimensional image that has been subjected to the post-processing and the first reprojection processing. The distortion processing unit 205c supplies the three-dimensional image that has been subjected to the post-processing, the first reprojection processing, and the distortion processing to the transmission unit 206.

In step S108, the transmission unit 206 transmits the three-dimensional image that has been subjected to the post-processing, the first reprojection processing, and the distortion processing to the HMD 100 as an HMD image. As a result, the HMD image that has been subjected to the first reprojection processing is displayed on the HMD 100.

In step S109, the reprojection unit 205b acquires a new reference position and reference direction as well as a new second HMD position and second HMD direction. The new reference position and reference direction as well as the new second HMD position and second HMD direction are, for example, the reference position and reference direction as well as the HMD position and HMD direction after a predetermined time has elapsed from the timing when the HMD image is displayed on the HMD 100. For example, when rendering is performed at a frame rate of 60 fps (frames per second) and the frame interval is 16.6 milliseconds, the reference position and reference direction as well as the HMD position and HMD direction after a predetermined time of 8.3 milliseconds has elapsed from the timing when the HMD image is displayed on the HMD 100 are set as the new reference position and reference direction as well as the new second HMD position and second HMD direction. The timing when the HMD image is displayed on the HMD 100 is determined taking into consideration the transmission time, etc., expected when the image rendered by the image generating device 200 is transmitted to the HMD 100.

In step S110, the reprojection unit 205b executes a second reprojection process based on the amount of change from the reference position and reference direction to the new reference position and reference direction (hereinafter referred to as the amount of change in the reference position and reference direction) and the amount of change from the HMD position and HMD direction to the new second HMD position and second HMD direction (hereinafter referred to as the amount of change in the HMD position and HMD direction). The second reprojection process is a process that converts the post-processed three-dimensional image to match the new viewpoint position and viewpoint direction based on the reference position and reference direction, the new reference position and reference direction, the HMD position and HMD direction, and the new HMD position and HMD direction. The second reprojection process converts the post-processed three-dimensional image into a three-dimensional image that can be seen from the new viewpoint position and viewpoint direction.

Here, the relationship between the amount of change in the reference position and reference direction, the amount of change in the HMD position and HMD direction, and the new viewpoint position and viewpoint direction will be described with reference to FIG. 6. As shown in FIG. 6, if the viewpoint position determined in step S102 is (x, y, z), the amount of change in the reference position is (Δxr, Δyr, Δzr), and the amount of change in the HMD position is (Δxh, Δyh, Δzh), the new viewpoint position is expressed as (x+Δxr+Δxh, y+Δyr+Δyh, z+Δzr+Δzh). Also, if the viewpoint direction determined in step S102 is (α, β, γ), the amount of change in the reference direction is (Δαr, Δβr, Δγr), and the amount of change in the HMD direction is (Δαh, Δβh, Δγh), the new viewpoint direction is expressed as (α+Δαr+Δαh, β+Δβr+Δβh, γ+Δγr+Δγh).

The reprojection unit 205b executes the reprojection process, for example, by linearly complementing the viewpoint position based on the new viewpoint position expressed as above, and spherically linearly complementing the viewpoint direction based on the new viewpoint direction expressed as above. This converts the three-dimensional image to match the new viewpoint position and viewpoint direction. The reprojection unit 205b supplies the three-dimensional image that has been subjected to the second reprojection process to the distortion processing unit 205c.

In step S111, the distortion processing unit 205c performs distortion processing on the three-dimensional image that has been subjected to the second reprojection processing. The distortion processing unit 205c supplies the three-dimensional image that has been subjected to the post-processing, the second reprojection processing, and the distortion processing to the transmission unit 206.

In step S112, the transmission unit 206 transmits the three-dimensional image that has been subjected to the post-processing, second reprojection processing, and distortion processing to the HMD 100 as an HMD image. As a result, the HMD image that has been subjected to the second reprojection processing is displayed on the HMD 100.

After step S112, process S100 ends.

The flow of the reprojection process of the first embodiment will be described with reference to FIG. 7. In FIG. 7, the synchronization timing Vsync indicates the vertical synchronization timing of the display panel of the HMD 100. When the drawing timing of the three-dimensional image is reached, a rendering process 1 is executed based on the reference position and reference direction and the HMD position and HMD direction at that drawing timing. Next, the first reprojection process is executed on the drawn three-dimensional image based on the new first HMD position and first HMD direction obtained at the drawing timing of the rendering process 1, and the three-dimensional image after the first reprojection process is transmitted to the HMD 100. As a result, the three-dimensional image 1-1 to which the first reprojection process is applied at timestamp t0 is displayed on the HMD 100 (corresponding to steps S101 to S108 above). The reason why the reference position and reference direction are not taken into consideration in the first reprojection process is that, as described above, the reference position and reference direction at the drawing timing of the three-dimensional image 1-1 are used, and it can be considered that the reference position and reference direction at the timestamp t0 are roughly reflected. Next, a second reprojection process is performed on the three-dimensional image rendered in the rendering process 1 based on a new reference position and reference direction after a predetermined time has elapsed since the timestamp t0 (for example, immediately before the timestamp t1) and a new second HMD position and second HMD direction, and the three-dimensional image that has undergone the second reprojection process is transmitted to the HMD 100. The predetermined time here is, for example, a time that is less than the frame interval time from the time when the three-dimensional image is displayed on the HMD 100 to the time when the three-dimensional image of the next frame is displayed on the HMD 100, and can be set to, for example, a time equivalent to half the frame interval time. For example, when the frame interval time is 16.6 ms, the predetermined time is set to 8.3 ms. After that, the three-dimensional image 1-2 to which the second reprojection process has been applied at the timestamp t1 is displayed on the HMD 100 (corresponding to steps S109 to S112 above).

When the next drawing timing is reached, rendering process 2 is executed based on the reference position and reference direction as well as the HMD position and HMD direction at that drawing timing. Next, a first reprojection process is executed on the drawn three-dimensional image based on the new first HMD position and first HMD direction obtained at the drawing timing of rendering process 2, and a three-dimensional image 2-1 to which the first reprojection process has been applied at timestamp t2 is displayed on the HMD 100. Next, a second reprojection process is executed on the three-dimensional image drawn in rendering process 2 based on the new reference position and reference direction as well as the new second HMD position and second HMD direction after a predetermined time has elapsed since timestamp t2 (for example, immediately before timestamp t3), and a three-dimensional image 2-2 to which the second reprojection process has been applied at timestamp t3 is displayed on the HMD 100.

Here, between the time when an image is generated and the time when the next image is generated, for example, an instruction is input from the input device 300 or the game control unit 201, and during this time the user's reference position and reference direction in the virtual space may change. If the drawing is performed at a frame rate of, for example, 60 fps (frames per second), even if the CPU is fast enough, the reference position and reference direction that change in about 16.67 milliseconds will not be reflected in the HMD image. As a result, the HMD image may differ from what the user expected, causing the user to feel uncomfortable.

In this embodiment, a reprojection process is performed to convert the three-dimensional image to match a new reference position and reference direction. With this configuration, the change in the reference position and reference direction between the time point at which an image is generated and the time point at which the next image is generated is reflected in the HMD image, so that it is possible to prevent the user from feeling uncomfortable about the HMD image due to this change in the reference position and reference direction.

In this embodiment, the second reprojection process converts the three-dimensional image to match the new viewpoint position and viewpoint direction based on the amount of change from the HMD position and HMD direction to the new second HMD position and second HMD direction, and the amount of change from the reference position and reference direction to the new reference position and reference direction. With this configuration, the reprojection process is executed taking into account changes in the HMD position and HMD direction in addition to changes in the reference position and reference direction, so that it is possible to more effectively prevent the user from feeling uncomfortable about the HMD image.

In this embodiment, the transmission unit 206 displays a three-dimensional image on the HMD 100, and then displays a three-dimensional image that has been subjected to the second reprojection process on the HMD 100. The second reprojection process converts the three-dimensional image to match a new reference position and reference direction after a predetermined time has elapsed since the time the three-dimensional image was displayed. With this configuration, after the three-dimensional image is displayed, it is possible to realize display at a higher frame rate by interpolating the image that has been converted to match the new reference position and reference direction.

　Below, we will explain some modified examples of the embodiment.

In this embodiment, the viewpoint position and viewpoint direction are determined based on the HMD position and HMD direction and the reference position and reference direction, but this is not limited to the above, and the viewpoint position and viewpoint direction may be determined based on the reference position and reference direction. In this case, the reprojection process may be performed based on the new reference position and reference direction.

In this embodiment, the reprojection process is performed on the position and the direction, but the reprojection process may be performed on at least one of the position and the direction.

In this embodiment, as shown in FIG. 7, one three-dimensional image to which the second reprojection process has been applied is generated and displayed between the drawing timing of one three-dimensional image and the drawing timing of the next, but this is not limited to this, and two or more three-dimensional images to which the second reprojection process has been applied may be generated and displayed at different timestamps.

In this embodiment, an HMD image that has been subjected to the first reprojection process is transmitted, and then an HMD image that has been subjected to the second reprojection process is transmitted. However, this is not limited to this, and an HMD image that has not been subjected to the reprojection process (i.e., a three-dimensional image generated by the rendering process in step S103) may be transmitted, and then an HMD image that has been subjected to the second reprojection process may be transmitted.

Second embodiment Hereinafter, a second embodiment of the present invention will be described. In the drawings and description of the second embodiment, the same or equivalent components and members as those of the first embodiment are denoted by the same reference numerals. Descriptions that overlap with the first embodiment will be omitted as appropriate, and the description will focus on the configurations that differ from the first embodiment.

The process S200 for image generation in the image generating device of this embodiment will be described with reference to FIG. 8. Steps S201 to S208 are basically the same as steps S101 to S104 and S109 to S112 in FIG. 5. That is, the process S200 of this embodiment does not execute a first reprojection process based only on a new HMD position and HMD direction as in step S108 of FIG. 5, but executes a second reprojection process based on a new reference position and reference direction at the timing when the three-dimensional image is displayed on the HMD 100, as well as a new HMD position and HMD direction.

The flow of the reprojection process of the second embodiment will be described with reference to FIG. 9. When the drawing timing of the three-dimensional image is reached, a rendering process 1 is executed based on the reference position and reference direction at that drawing timing and the HMD position and HMD direction. Next, a second reprojection process is executed for the three-dimensional image drawn in the rendering process 1 based on the new reference position and reference direction at the timing when the three-dimensional image is displayed on the HMD 100 and the new HMD position and HMD direction, and the three-dimensional image after the second reprojection process is transmitted to the HMD 100. Here, for example, the reference position etc. obtained at the timing when the three-dimensional image is transmitted to the HMD 100 (timestamp t0) may be regarded as the "new reference position etc. at the timing when the three-dimensional image is displayed on the HMD 100", or a "new reference position etc. at the timing when the three-dimensional image is displayed on the HMD 100" predicted by a known method based on the reference position etc. newly obtained after the drawing timing may be used. After that, the three-dimensional image to which reprojection processing has been applied based on the new reference position and reference direction at the time when the three-dimensional image is displayed on the HMD 100 and the new HMD position and HMD direction is applied is displayed on the HMD 100.

Here, because there is a slight delay between the generation and display of the three-dimensional image, the reference position and reference direction may shift during this time. In contrast, in this embodiment, the reprojection process converts the three-dimensional image so that it matches the new reference position and reference direction at the time the three-dimensional image is displayed on the HMD 100. With this configuration, it becomes difficult for the user to sense the shift in the reference position and reference direction, so it is possible to more effectively prevent the user from feeling uncomfortable about the HMD image.

In the second embodiment, as in the modified example of the first embodiment, between the drawing timing of a three-dimensional image and the next drawing timing, one or more three-dimensional images to which a new reprojection process has been applied may be displayed so as to match a new reference position and reference direction after a predetermined time has elapsed since the timing when the three-dimensional image in that rendering process was first displayed. That is, although the embodiment in FIG. 9 shows that the second reprojection process is performed only once for one rendering process, it is also possible to use an embodiment in which the second reprojection process is performed multiple times for one rendering process.

The present invention has been described above based on an embodiment. The embodiment is merely an example, and those skilled in the art will understand that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention. Such modifications will now be described.

The present invention relates to image generation technology.

100 Head-mounted display, 200 Image generation device, 201 Game control unit, 202 Reference determination unit, 203 Viewpoint determination unit, 204 Rendering unit, 205 Image processing unit, 205a Post-processing unit, 205b Reprojection unit, 205c Distortion processing unit, 206 Transmission unit, 207 Memory unit, 300 Input device.

Claims

a reference determination unit that determines at least one of a reference position and a reference direction, which are a position and a direction in which a user is to be placed in a virtual space, based on an instruction input that is different from at least one of an instruction input of a position and a direction of the head mounted display;
a viewpoint determination unit that determines at least one of a viewpoint position and a viewpoint direction of the user in the virtual space based on at least one of the reference position and the reference direction;
a rendering unit that generates a three-dimensional image by rendering an object in the virtual space based on at least one of the viewpoint position and the viewpoint direction;
a reprojection unit that performs a reprojection process to convert the three-dimensional image to match at least one of the new reference position and the new reference direction;
An image generating device comprising:
the viewpoint determination unit determines at least one of the viewpoint position and the viewpoint direction based on at least one instruction input of the position and the direction of the head mounted display;
The reprojection process converts the three-dimensional image to match at least one of the new viewpoint position and viewpoint direction based on an amount of change from at least one of the position and orientation of the head mounted display to at least one of the new position and orientation of the head mounted display and an amount of change from at least one of the reference position and reference orientation to at least one of the new reference position and reference orientation.
2. The image generating device of claim 1.
a display control unit that causes the head mounted display to display the three-dimensional image after the three-dimensional image has been subjected to the reprojection processing;
The reprojection process converts the three-dimensional image so as to conform to at least one of the new reference position and the new reference direction after a predetermined time has elapsed since the timing when the three-dimensional image was displayed.
2. The image generating device of claim 1.
The reprojection process converts the three-dimensional image so as to match at least one of the new reference position and the new reference direction when the three-dimensional image is displayed on the head mounted display.
2. The image generating device of claim 1.
The different instruction input includes at least one of an instruction input via a user input device different from the head mounted display and an instruction input for progressing a predetermined scene in the virtual space.
2. The image generating device of claim 1.
determining at least one of a reference position and a reference direction of the user in the virtual space based on an instruction input different from at least one of an instruction input of a position and an orientation of the head mounted display;
determining at least one of a viewpoint position and a viewpoint direction of the user in the virtual space based on at least one of the reference position and the reference direction;
generating a three-dimensional image by rendering an object in the virtual space based on at least one of the viewpoint position and the viewpoint direction;
performing a reprojection process to convert the three-dimensional image to match at least one of the new reference position and reference orientation;
An image generating method comprising:
On the computer,
determining at least one of a reference position and a reference direction of the user in the virtual space based on an instruction input different from at least one of an instruction input of a position and an orientation of the head mounted display;
determining at least one of a viewpoint position and a viewpoint direction of the user in the virtual space based on at least one of the reference position and the reference direction;
generating a three-dimensional image by rendering an object in the virtual space based on at least one of the viewpoint position and the viewpoint direction;
performing a reprojection process to convert the three-dimensional image to match at least one of the new reference position and reference orientation;
An image generation program for executing the above.