JP2022137945A - Image generation device, program, image generation method, and image display system - Google Patents

Abstract

To provide a mirroring technique for providing high quality mirroring images in a new system for providing images using a head-mounted display.SOLUTION: An image generation device 200 includes: a source image generation unit 203 that generates a source image to which distortion is not given; an HMD image generation unit 204a that generates an HMD image displayed on a head-mounted display based on the source image; and a mirroring image generation unit 204b that generates a mirroring image for mirroring the HMD image in a flat-panel display, based on the source image.SELECTED DRAWING: Figure 3

Description

The present invention relates to image generation technology.

A head-mounted display connected to a game machine is worn on the head, and a game is played by operating a controller or the like while viewing a screen displayed on the head-mounted display. When the user wears the head-mounted display, the user does not see anything other than the image displayed on the head-mounted display, so that the sense of immersion in the image world is enhanced, which has the effect of further enhancing the entertainment of the game. In addition, if a virtual reality image is displayed on a head-mounted display, and the user wearing the head-mounted display rotates his or her head, a 360-degree panoramic view of the virtual space can be displayed. The sense of immersion in images is enhanced, and the operability of applications such as games is also improved.

On the other hand, in recent years, mirroring has been proposed for sharing a screen displayed by a head-mounted display with another display device. Mirroring is a technology for sharing the displayed screen between the transmitting device and the receiving device by transmitting image data as information on the screen displayed by the transmitting device to the receiving device via the network. .

New systems are being developed for providing images using head-mounted displays. Accordingly, there is a need for mirroring technology to provide high-quality mirrored images in this new system.

SUMMARY OF THE INVENTION In view of the above, it is an object of the present invention to provide mirroring techniques for providing high quality mirrored images in new systems for providing images using head-mounted displays.

In order to solve the above problems, an image generation device according to one aspect of the present invention includes a source image generation unit that generates a source image that is not distorted, and a head-mounted display based on the source image. An HMD image generation unit that generates an HMD image, and a mirroring image generation unit that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image.

A program according to an aspect of the present invention comprises a processor, a source image generation unit that generates a source image that is not distorted, and an HMD image that generates an HMD image to be displayed on a head-mounted display based on the source image. and a mirroring image generating unit that generates a mirroring image for mirroring the HMD image on the flat panel display based on the source image.

An image generation method according to one aspect of the present invention includes the steps of generating a source image to which distortion is not applied, an HMD image displayed on a head-mounted display based on the source image, and displaying the HMD image on a flat panel display. and generating a mirrored image for mirroring in.

An image display system according to one aspect of the present invention includes a source image generation unit that generates a source image that is not distorted, and an HMD image generation unit that generates an HMD image to be displayed on a head-mounted display based on the source image. a mirroring image generating unit for generating a mirroring image for mirroring the HMD image on a flat panel display based on the source image; an HMD image supplying unit for supplying the HMD image to the head mounted display; an image generation device comprising: a mirrored image supply unit that supplies the mirrored image to the flat panel display; the head mounted display that displays the supplied HMD image; and the flat panel that displays the supplied mirrored image. and a mold display.

In addition, any combination of the above, and the mutual replacement of the components and expressions of the present invention between methods, devices, programs, temporary or non-temporary storage media recording programs, systems, etc. It is effective as an aspect of the present invention.

ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide a mirroring technique for providing a high-quality mirrored image in a new system for providing images using a head-mounted display.

1 is an external view of a head mounted display; FIG. 1 is a configuration diagram of an image generation system; FIG. It is a figure for demonstrating the mirroring system of this embodiment. 3 is a configuration diagram of the rendering device of FIG. 2; FIG. FIGS. 5(a) and 5(b) are examples of a head mounted display image and a mirroring image, respectively, in three-dimensional display. FIGS. 6A and 6B are examples of a head mounted display image and a mirroring image, respectively, in two-dimensional display. 4 is a flow chart showing the flow of processing in the rendering device; FIG. 10 is a diagram for explaining a mirroring system of a comparative example; FIG. 11 is a diagram for explaining a mirroring system of a modified example;

(embodiment)
FIG. 1 is an external view of the head mounted display 100. FIG. The head-mounted display 100 is a display device worn on the user's head for viewing still images and moving images displayed on the display and for listening to sounds and music output from headphones.

A gyro sensor or an acceleration sensor built in or external to the head mounted display 100 measures the position information of the head of the user wearing the head mounted display 100 and the orientation information such as the rotation angle and inclination of the head. be able to.

Head mounted display 100 may further include a camera that captures the user's eyes. A camera mounted on the head-mounted display 100 can detect the user's gaze direction, pupil movement, blinking, and the like.

Head-mounted display 100 is an example of a "wearable display." Here, a method for generating an image displayed on the head mounted display 100 will be described. Note that the image generation method of the present embodiment is not limited to the head-mounted display 100 in a narrow sense. It can also be applied when wearing a hat, a hat with a camera, a hair band, or the like.

FIG. 2 is a configuration diagram of an image transfer system according to this embodiment. For example, the head-mounted display 100 has an interface such as HDMI (registered trademark) (High-Definition Multimedia Interface), which is a communication interface standard for transmitting video and audio as digital signals, and DisplayPort, which is a video output interface standard. It is connected to the rendering device 200 via. A flat panel display 302 is connected to the rendering device 200 . The rendering device 200 of this embodiment is an example of an image generation device.

In this embodiment, the data transmission line 300 between the head mounted display 100 and the rendering device 200 is an HDMI transmission line or a DisplayPort transmission line. In the HDMI standard or DisplayPort standard, a secondary data packet can be transmitted in association with an image frame, and the secondary data packet can contain metadata related to the frame. The HDMI 2.1 standard has a function called dynamic HDR (High Dynamic Range), which refers to the dynamic metadata of the video and optimally adjusts the brightness and color depth for each frame according to the scene. It is possible to generate a video with In the HDMI 2.1 standard, dynamic metadata can transmit information necessary for dynamic HDR, such as the maximum luminance, average luminance, and minimum luminance of a scene, in synchronization with video. The communication interface between the head mounted display 100 and the rendering device 200 is not limited to HDMI or DisplayPort as long as the metadata can be transmitted in synchronization with the video.

The rendering device 200 of this embodiment is a game machine. The rendering device 200 may also be connected to a server via a network. In that case, the server may provide the rendering device 200 with an online application such as a game in which multiple users can participate via a network.

The rendering device 200 basically processes a content program, generates a display image, and transmits it to the head mounted display 100 or the flat panel display 302 . Content programs and data are read by a media drive (not shown) from a ROM medium (not shown) that records application software for content such as games and license information. This ROM medium is a read-only recording medium such as an optical disc, magneto-optical disc, Blu-ray disc, or the like. In one aspect, the rendering device 200 specifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and displays the display image of the content in accordance with the position and posture of the user's head. generated at the rate of

The head mounted display 100 receives the display image data and displays it as a content image. The video displayed on the head-mounted display 100 may be a video captured by a camera in advance, a computer graphics video such as a game video, or a remote live video delivered via a network. Also, the images displayed on the head-mounted display 100 may be VR images, AR (augmented reality) images, MR images, or the like.

The flat-panel display 302 may be a television having a display for outputting images and a speaker for outputting audio. Specifically, it may be a liquid crystal television, an organic EL television, a plasma television, a PC display, or the like. Alternatively, the flat display 302 may be the display and speaker of a tablet terminal or a mobile terminal.

Here, using the comparative example of FIG. 9, a possible method of mirroring technology for sharing an image displayed on the head-mounted display 100 by displaying it on the flat-panel display 302 will be described. The mirroring system of FIG. 9 includes a rendering device 400, a head-mounted display 100, a flat panel display 302, and an output control device 500. FIG. The output control device 500 is provided separately from the rendering device 400 and connected to the rendering device 400 . The output control device 500 processes data output by the rendering device 400 . The output control device 500 relays the processed data to the head mounted display 100 and the flat panel display 302 via the head mounted display output port 500a and the mirrored image output port 500b.

In the mirroring system of FIG. 9, the rendering device 400 first generates a left-eye image L and a right-eye image R as source images for content. The rendering device 400 generates a binocular image C for the head-mounted display 100 by applying distortion correction to the left-eye image L and the right-eye image R in accordance with the distortion caused by the eyepiece lens of the head-mounted display 100 . The binocular image C is intended for stereoscopic viewing of the image, and a pair of parallax images L and R for the left eye and an image R for the right eye are provided in the left and right regions obtained by dividing the image plane corresponding to the display panel into two. It has a configuration in which images are arranged respectively. Here, the head-mounted display 100 employs an optical lens with a high curvature in order to display an image with a wide viewing angle in front of and around the user's eyes, and the user looks into the display panel through the lens. If a lens with a high curvature is used, the image will be distorted due to lens distortion. Therefore, in order to make it look correct when viewed through a lens with a high curvature, distortion processing is applied to the rendered image in advance, and the image after distortion processing is transmitted to the head-mounted display and displayed on the display panel, where the user can view the image. Make it look normal when viewed through a lens with high curvature. The rendering device 400 supplies this binocular image C to the output control device 500 .

The output control device 500 supplies the binocular image C as it is to the head mounted display 100 via the HMD image output port 500a as a head mounted display image (hereinafter referred to as an HMD image) C. Further, the output control device 500 performs correction opposite to the above-described distortion correction to either the image for the left eye or the portion for the right eye in the binocular image C, and cuts out the central portion of the corrected image. A mirrored image M for the flat panel display 302 is generated. The output control device 500 supplies the mirrored image M to the flat panel display 302 via the mirrored image output port 500b. Thus, the HMD image C is displayed on the head-mounted display 100, the mirroring image M is displayed on the flat panel display 302, and mirroring is realized in the mirroring system of FIG. However, in the mirroring system of FIG. 9, there is a problem that the mirroring image M is degraded because it is necessary to perform correction opposite to the distortion correction.

Meanwhile, a new mirroring system of FIG. 3 has been developed. The mirroring system of FIG. 3 includes a rendering device 200, a head-mounted display 100, and a flat panel display 302. FIG. Unlike the system of FIG. 9, the system of FIG. 3 does not have the output control device 500 between the rendering device 400 and the head-mounted display 100 and flat-panel display 302 . In the system of FIG. 3, the functions of generating a source image, performing necessary processing on the source image to generate an HMD image and a mirrored image, and supplying them to the head-mounted display 100 and the flat panel display 302 are the same. is realized in the device of

An outline of the mirroring technique in the system of FIG. 3 will be described. The rendering device 200 generates a left-eye image L and a right-eye image R as source images for content. Based on the source image, the rendering device 200 performs distortion correction according to the distortion caused by the eyepiece lens of the head mounted display 100, reprojection processing based on the latest position information and orientation information of the head mounted display 100, and the like. A binocular image C is generated by applying it to the image R for the right eye. The rendering device 200 supplies the generated binocular image C as an HMD image to the head mounted display 100 via the HMD image output port 200a.

In addition, the rendering device 200 generates a mirroring image M by performing a later-described process on one of the left-eye image L and the right-eye image R (the right-eye image R in this embodiment) based on the source image. do. The rendering device 200 supplies the generated mirrored image M to the flat panel display 302 via the mirrored image output port 200b. In this manner, the HMD image C is displayed on the head-mounted display 100, the mirroring image M is displayed on the flat-panel display 302, and mirroring is realized in the mirroring system of FIG.

According to the present embodiment, the mirrored image M is generated by correcting the distortion of the source image, and since the distortion correction and the reverse correction are not performed as in the mirroring system of FIG. 9, deterioration of the mirrored image M is suppressed. It becomes possible to As a result, it is possible to provide a mirrored image of high quality. The mirroring method of this embodiment will be described in detail below.

FIG. 4 is a configuration diagram of the rendering device 200 of FIG. The rendering device 200 includes an input data acquisition unit 201, an information processing unit 202, a source image generation unit 203, a source image processing unit 204, first and second communication units 205a and 205b, and an image generation unit. and a data storage unit 206 for storing data. FIG. 4 depicts a functionally focused block diagram, and these functional blocks can be implemented in various forms by hardware only, software only, or a combination thereof.

The first and second communication units 205a and 205b establish communication using a predetermined protocol such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus). The first communication unit 205a is configured to be able to communicate with the head mounted display 100 via the HMD image output port 200a. The first communication unit 205a receives position information and attitude information of the head mounted display 100 from the head mounted display 100 . The first communication unit 205 a is configured to be able to supply the HMD image C to the head mounted display 100 . The first communication unit 205a of this embodiment is an example of an HMD image supply unit. The second communication unit 205b is configured to be able to communicate with the flat panel display 302 via the mirroring image output port 500b. The second communication unit 205 b is configured to be able to supply the mirroring image M to the flat panel display 302 . The second communication unit 205b of this embodiment is an example of a mirroring image supply unit.

The input data acquisition unit 201 acquires input data. The input data includes operation information indicating details of user operation input via an input device (not shown), position information and posture information of the user's head obtained via the first communication unit 205a, and the like. Here, the user operation may be performed by general information processing such as selection of an application to be executed, start/end of processing, command input, and the like. The input data acquisition unit 201 supplies the acquired input data to the information processing unit 202 .

The information processing unit 202 performs information processing according to an application to be executed such as a game. The information processing unit 202 advances the game or executes corresponding processing based on, for example, input data. The information processing unit 202 supplies the result of information processing to the source image generation unit 203 .

The source image generation unit 203 generates a source image such as a game image to be output by reading necessary data from the data storage unit 206 based on the result of information processing. For example, the source image generation unit 203 generates, as source images, a left-eye image L and a right-eye image R regarding the virtual world viewed from a viewpoint corresponding to the position and posture of the user's head. The source image generation unit 203 thus generates a source image that is not distorted. The source image generation unit 203 supplies the source image to the image processing unit 204 .

The image processing unit 204 generates the HMD image C and the mirroring image M by processing the source image as necessary. The source image processing unit 204 includes an HMD image generation unit 204a that generates an HMD image C and a mirrored image generation unit 204b that generates a mirrored image M.

The HMD image generation unit 204a draws the left-eye image L and the right-eye image R as source images as left and right parallax images, performs necessary processing, and generates binocular images C arranged on the left and right. The necessary processing includes distortion correction considering the lens, reprojection processing based on the latest position information and orientation information of the head-mounted display 100, superimposition processing such as a menu screen, etc., and each general method is applied. be able to. The HMD image generation unit 204a supplies the generated binocular image C as the HMD image C to the head mounted display 100 via the first communication unit 205a.

The mirroring image generation unit 204b generates a mirroring image M using a portion (for example, the central portion) of one of the left-eye image L and the right-eye image R (right-eye image R in this embodiment) as the source image. do. The mirroring image generation unit 204b supplies the mirroring image M to the flat panel display 302 via the second communication unit 205b. A method of generating a mirroring image in the mirroring image generation unit 204b will be described below.

First, the case of generating the mirroring image M in the three-dimensional display mode will be described. The mirroring image generation unit 204b obtains the viewing angle (UV coordinate value) of the head mounted display 100 for the HMD image C reflecting the reprojection processing in the HMD image generation unit 204a. UV coordinate values are parameter coordinates for referencing texture attributes. The mirroring image generation unit 204b generates a plurality of texture images corresponding to a portion of the image R for the right eye by referring to the texture corresponding to the viewing angle using the obtained UV coordinate values. The mirrored image generation unit 204b generates a mirrored image M in a three-dimensional display mode by superimposing the plurality of texture images. In this way, by using the same viewing angle as the viewing angle of the binocular image C after processing in the HMD image generation unit 204a, the mirroring image M can be generated so as to match the viewing angle of the HMD image C after the reprojection processing. It becomes possible. Therefore, it is possible to provide a high-quality mirrored image M in a three-dimensional display mode in which the influence of tracking of the head-mounted display 100 is suppressed. Also, the width of the display of the flat panel display 302 can be adjusted using the viewing angle of the head-mounted display 100 in the binocular image C after processing, and the texture can be selected. Furthermore, it is possible to make the image quality depend on the texture of the source image.

Next, a case of generating a mirrored image M in a two-dimensional display mode will be described. The mirrored image generation unit 204b creates a mirrored image M in a two-dimensional display mode by pasting a portion of the right-eye image R as a texture onto a 16:9 panel (virtual screen) corresponding to the screen size of the flat panel display 302. Generate. As a result, a portion of the right-eye image R is displayed as it is regardless of the tracking (movement of the head) of the head-mounted display 100 . Therefore, it is possible to provide a high-quality mirrored image M in a two-dimensional display mode in which the influence of tracking of the head-mounted display 100 is suppressed. Here, the mirrored image M in the two-dimensional display mode is used when projecting a two-dimensional plane image in the virtual space. A mirrored image M is used.

By supplying the HMD image and the mirrored image to the head-mounted display 100 and the flat-panel display 302 in this manner, these images are displayed on the head-mounted display 100 and the flat-panel display 302, respectively. FIGS. 5A and 5B are examples of HMD image C and mirroring image M in three-dimensional display. FIGS. 6A and 6B are examples of the HMD image C and the mirroring image M in two-dimensional display.

Processing S10 for displaying the HMD image C and the mirrored image M of the present embodiment on the head-mounted display 100 and the flat-panel display 302 will be described below with reference to FIG. In step S11, the source image generation unit 203 generates a source image that is not distorted. At step S12, the image processing unit 204 generates the HMD image C and the mirrored image M based on the source image. In step S<b>13 , the first communication unit 205 a supplies the HMD image C to the head mounted display 100 and the second communication unit 205 b supplies the mirrored image M to the flat panel display 302 . After S13, the process S10 ends.

(Modification)
Modifications of the embodiment will be described below.

In the embodiment, mirrored image M was generated using a portion of right-eye image R, but is not limited to this. For example, the mirrored image M may be equivalent to the binocular image C generated by the HMD image generator 204a (see FIG. 8). In this case, the mirroring image generation unit 204b may generate the binocular image C by performing the same processing as the HMD image generation unit 204a, and supply this to the flat panel display 302 as the mirroring image M. According to this configuration, the HMD image C (binocular image C) supplied to the head-mounted display 100 can be confirmed on the flat-panel display 302, which facilitates debugging for the developer.

In this embodiment, the rendering device 200 is a game machine, but is not limited to this. The rendering device 200 may also be a server or the like connected to the head mounted display 100 and flat panel display 302 via a network.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the embodiments are examples, and that various modifications can be made to combinations of each component and each treatment process, and that such modifications are within the scope of the present invention. . Such modifications will be described.

100 head mounted display, 200 rendering device, 200a HMD image output port, 200b mirrored image output port 200b, 201 input data acquisition unit, 202 information processing unit, 203 source image generation unit, 204 source image processing unit, 204a HMD image Generation unit 204b Mirroring image generation unit 205 Communication unit 206 Data storage unit 300 Data transmission line 302 Flat display.

Claims (7)

a source image generator that generates an undistorted source image;
an HMD image generator that generates an HMD image to be displayed on a head-mounted display based on the source image;
a mirroring image generator that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image;
An image generating device comprising: the source image includes a right-eye image and a left-eye image of the head-mounted display;
2. The image generating apparatus according to claim 1, wherein said mirroring image generator generates said mirroring image using a portion of one of said right-eye image and said left-eye image. The HMD image generation unit generates the HMD image by performing reprojection processing on a portion of the image,
3. The mirrored image generation unit generates the mirrored image in a three-dimensional display mode by referring to a texture corresponding to a viewing angle of the head mounted display for the HMD image to which the reprojection process is applied. 3. The image generation device according to 2. 3. The mirroring image generation unit according to claim 2, wherein the mirroring image in a two-dimensional display mode is generated by pasting the part of the one image onto a panel corresponding to the screen size of the flat panel display. Image production device. the processor,
a source image generator that generates an undistorted source image;
an HMD image generator that generates an HMD image to be displayed on a head-mounted display based on the source image;
a mirroring image generator that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image;
A program characterized by functioning as generating an undistorted source image;
generating an HMD image to be displayed on a head-mounted display and a mirroring image for mirroring the HMD image on a flat panel display based on the source image;
An image generation method, comprising: a source image generator that generates an undistorted source image;
an HMD image generator that generates an HMD image to be displayed on a head-mounted display based on the source image;
a mirroring image generator that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image;
an HMD image supply unit that supplies the HMD image to the head mounted display;
a mirrored image supply unit that supplies the mirrored image to the flat panel display;
an image generation device comprising
the head mounted display that displays the supplied HMD image;
the flat panel display for displaying the supplied mirrored image;
An image display system comprising:

Images