JP7234021B2 - Image generation device, image generation system, image generation method, and program - Google Patents

Description

The present invention relates to apparatus, systems and methods for generating images.

A head-mounted display connected to a game machine is worn on the head, and a game is played by operating a controller or the like while viewing a screen displayed on the head-mounted display. When the user wears the head-mounted display, the user does not see anything other than the image displayed on the head-mounted display, so that the sense of immersion in the image world is enhanced, which has the effect of further enhancing the entertainment of the game. In addition, a virtual reality (VR) image is displayed on a head-mounted display, and when the user wearing the head-mounted display rotates his or her head, a 360-degree panoramic view of the virtual space is displayed. As a result, the feeling of being immersed in the video is further enhanced, and the operability of applications such as games is also improved.

In addition, although the user wearing the non-transmissive head-mounted display cannot directly see the outside world, the camera mounted on the head-mounted display can capture images of the outside world and display them on the display panel. There is also a video see-through) type head-mounted display. In a video transmissive head-mounted display, augmented reality (AR) is realized by superimposing virtual world objects generated by computer graphics (CG) on images of the outside world captured by a camera. You can also generate and display images. Unlike virtual reality, which is separated from the real world, augmented reality video is an extension of the real world with virtual objects, allowing users to experience the virtual world while being aware of the connection with the real world. can be done.

When an augmented reality video is generated by superimposing a virtual object generated by CG on a camera image and displayed on a head-mounted display, aliasing occurs at the boundary of the virtual object due to the influence of post-processing on the image. The boundaries of the real world are conspicuous, and the AR image may not have a sense of unity. Also, unless the shadows cast by the virtual object on the real space and the reflection of the virtual object in the real space are reflected in the AR image, a sense of unity between the virtual world and the real world cannot be obtained, and the virtual object cannot be seen in the real world. It looks like it's floating.

The present invention has been made in view of these problems, and its object is to provide an image generation device, an image generation system, and an image generation method that can improve the quality of augmented reality video.

In order to solve the above-described problems, an image generating apparatus according to one aspect of the present invention renders an object in a virtual space and an object in a real space, and renders a representation of light in the virtual space with respect to the real space to generate a computer graphic image. a rendering unit that generates a physical image; a superimposing unit that generates a temporary superimposed image by superimposing the computer graphics image on the captured image of the real space; and the computer graphics based on depth information of the captured image of the real space. and a chromakey generation unit that generates a chromakey image by performing chromakey processing on the space image, and a chromakey generation unit that generates an augmented reality image by superimposing the virtual superimposed image on the captured image of the real space by masking the temporary superimposed image with the chromakey image. a compositing unit that generates a composite chromakey image that is used to The chromakey generation unit treats a real space area in which the object in the virtual space is not rendered as a chromakey area, but does not treat a real space area in which an expression related to light in the virtual space exists as a chromakey area.

Another aspect of the invention is an image generation system. This image generation system includes a head-mounted display and an image generation device. a rendering unit that renders an expression related to light to generate a computer graphics image; a chromakey generator for generating a chromakey image by chromakey processing the computer graphics image based on depth information of the captured image of the real space transmitted from the head-mounted display; and the temporary superimposed image. a synthesizing unit that generates a synthesized chromakey image that is superimposed on the captured image of the physical space and used to generate an augmented reality image by masking with the chromakey image. The head-mounted display includes a second superimposing unit that generates an augmented reality image by synthesizing the captured image of the physical space with the synthesized chromakey image transmitted from the image generating device. The chromakey generation unit treats a real space area in which the object in the virtual space is not rendered as a chromakey area, but does not treat a real space area in which an expression related to light in the virtual space exists as a chromakey area.

Yet another aspect of the invention is an image generation method. This method comprises a rendering step of rendering an object in a virtual space and an object in a real space, and rendering a representation of light in the virtual space with respect to the real space to generate a computer graphics image; A superimposing step of superimposing the computer graphics image to generate a temporary superimposed image, and a chromakey generating step of generating a chromakey image by performing chromakey processing on the computer graphics image based on depth information of the captured image of the physical space. and a synthesizing step of generating a synthesized chromakey image that is superimposed on the captured image of the physical space and used to generate an augmented reality image by masking the temporary superimposed image with the chromakey image. including. In the chromakey generating step, a real space area in which an object in the virtual space is not rendered is treated as a chromakey area, and a real space area in which light-related representation exists in the virtual space is not treated as a chromakey area.

Any combination of the above constituent elements, and expressions of the present invention converted between methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to improve the quality of augmented reality video.

1 is an external view of a head mounted display; FIG. 1 is a configuration diagram of an image generation system according to an embodiment; FIG. 2 is a diagram illustrating an example of a camera image captured by a camera mounted on the head mounted display of FIG. 1; FIG. 4 is a diagram for explaining an augmented reality image in which a CG virtual object is superimposed on the camera image of FIG. 3; FIG. FIG. 5 is a diagram showing a state in which a user is reaching out to a virtual object with respect to the augmented reality image of FIG. 4; FIG. 4 is a diagram for explaining a CG image used for chromakey synthesis; 1 is a functional configuration diagram of a head-mounted display according to a base technology; FIG. 1 is a functional configuration diagram of an image generation device according to a base technology; FIG. 1 is a diagram illustrating the configuration of an image generation system according to a base technology for superimposing a CG image on a camera image to generate an augmented reality image; FIG. 1 is a functional configuration diagram of an image generation device according to a first embodiment; FIG. 1 is a diagram illustrating the configuration of an image generation system according to a first embodiment for superimposing a CG image on a camera image to generate an augmented reality image; FIG. FIG. 4 is a diagram illustrating an augmented reality image in which a CG image is superimposed on a camera image by the image generation system according to the first embodiment; FIG. 4 is a diagram for explaining a synthesized CG chromakey image used by the image generation system according to the first embodiment; FIG. FIG. 10 is a diagram illustrating an example of deforming a polygon mesh in a real space to make a hole in a wall; FIG. 10 is a diagram illustrating an example in which a virtual object is drawn in a hole in a wall in real space; FIG. 10 is a functional configuration diagram of an image generating device according to a second embodiment; FIG. FIG. 10 is a diagram illustrating the configuration of an image generation system according to a second embodiment for superimposing a CG image on a camera image to generate an augmented reality image;

FIG. 1 is an external view of the head mounted display 100. FIG. The head-mounted display 100 is a display device worn on the user's head for viewing still images and moving images displayed on the display and for listening to sounds and music output from headphones.

A gyro sensor or an acceleration sensor built in or external to the head mounted display 100 measures the position information of the head of the user wearing the head mounted display 100 and the orientation information such as the rotation angle and inclination of the head. be able to.

A camera unit is mounted on the head mounted display 100, and the user can photograph the outside world while wearing the head mounted display 100. FIG.

Head-mounted display 100 is an example of a "wearable display." Here, a method of generating an image to be displayed on the head mounted display 100 will be described, but the image generating method of the present embodiment is not limited to the head mounted display 100 in a narrow sense. It can also be applied when wearing headphones, headsets (headphones with a microphone), earphones, earrings, ear cameras, hats, hats with cameras, hair bands, and the like.

FIG. 2 is a configuration diagram of an image generation system according to this embodiment. The head mounted display 100 is connected to the image generation device 200 via an interface 300 such as HDMI (registered trademark) (High-Definition Multimedia Interface), which is a communication interface standard for transmitting video and audio as digital signals. .

The image generation device 200 predicts the position/orientation information of the head-mounted display 100 from the current position/orientation information of the head-mounted display 100 in consideration of the delay from image generation to display. Based on the position/orientation information, an image to be displayed on the head mounted display 100 is drawn and transmitted to the head mounted display 100 .

An example of the image generation device 200 is a game machine. The image generation device 200 may also be connected to a server via a network. In that case, the server may provide the image generating device 200 with an online application such as a game in which multiple users can participate via a network. The head mounted display 100 may be connected to a computer or mobile terminal instead of the image generation device 200. FIG.

An augmented reality image in which a CG virtual object is superimposed on a camera image will be described with reference to FIGS. 3 to 6. FIG.

FIG. 3 is a diagram illustrating an example of a camera image taken by a camera mounted on head mounted display 100. As shown in FIG. This camera image captures the table and the basket 400 on it with the room as the background. The surface of the table has a floral pattern. The background hardly changes in the camera image, but the basket 400 on the table may be reached and moved by the user.

FIG. 4 is a diagram for explaining an augmented reality image in which a CG virtual object is superimposed on the camera image of FIG. A basket 400, which is a real object on the table, is replaced by a teapot 410, which is a virtual object generated by CG, and the teapot 410 is superimposed on the camera image. Thereby, the user can view an augmented reality image in which a virtual object is drawn in the real space on the head-mounted display 100 .

FIG. 5 is a diagram showing a state in which a user is reaching out to a virtual object with respect to the augmented reality image of FIG. When the user viewing the augmented reality image on the head-mounted display 100 tries to touch the virtual object teapot 410 , the user's hand is captured by the camera mounted on the head-mounted display 100 . is reflected. A teapot 410, which is a virtual object, is superimposed on the camera image in which the hand 420 is captured. At this time, it is necessary to correctly determine the positional relationship between the teapot 410 and the hand 420 using depth information so that the teapot 410 is superimposed on the hand 420 and the hand 420 is not visible, resulting in an unnatural augmented reality image. be.

Therefore, the depth information of the camera image is used to determine the positional relationship between the object captured in the camera image and the virtual object, and the drawing that accurately reflects the depth is performed. Since the depth of the background of the room and the basket 400 whose existence is already known is known in advance, the positional relationship with the virtual object can be determined in advance. When a moving object other than the user (for example, another person, a dog, a cat, etc.) comes into the field of view, the depth is not known in advance, so the depth needs to be determined each time from the depth information of the camera image.

In general, when a CG image is superimposed on a camera image, a chromakey image is created by painting an area such as the background of the CG image that is not drawn with a specific color, and is used for chromakey synthesis. A region of a color specified as chromakey (referred to as a “chromakey region”) becomes transparent, so when the chromakey image is superimposed on the camera image, the camera image is displayed in the chromakey region.

FIG. 6 is a diagram for explaining a CG image used for chromakey synthesis. In the state of FIG. 5, the background is filled with a specific chromakey color (eg, red). Further, since the hand 420 captured in the camera image is located in front of the teapot 410, the area of the teapot 410 hidden by the hand 420 is also filled with the specific chromakey color. When this chromakey image is superimposed on the camera image, the camera image remains because the specific color portion of the chromakey is transparent, and the augmented reality image shown in FIG. 5 is obtained.

FIG. 7 is a functional configuration diagram of the head mounted display 100 according to the base technology.

The control unit 10 is a main processor that processes and outputs signals such as image signals and sensor signals, commands and data. The input interface 20 receives operation signals and setting signals from the user and supplies them to the control unit 10 . The output interface 30 receives image signals from the control unit 10 and displays them on the display panel 32 .

The communication control unit 40 transmits data input from the control unit 10 to the outside by wired or wireless communication via the network adapter 42 or the antenna 44 . The communication control unit 40 also receives data from the outside by wired or wireless communication via the network adapter 42 or the antenna 44 and outputs the data to the control unit 10 .

The storage unit 50 temporarily stores data, parameters, operation signals, and the like processed by the control unit 10 .

The orientation sensor 64 detects position information of the head mounted display 100 and orientation information such as the rotation angle and inclination of the head mounted display 100 . The attitude sensor 64 is realized by appropriately combining a gyro sensor, an acceleration sensor, an angular acceleration sensor, and the like. A motion sensor that combines at least one of a 3-axis geomagnetic sensor, a 3-axis acceleration sensor, and a 3-axis gyro (angular velocity) sensor may be used to detect forward/backward, left/right, and up/down movements of the user's head.

The external input/output terminal interface 70 is an interface for connecting peripheral devices such as a USB (Universal Serial Bus) controller. The external memory 72 is an external memory such as flash memory.

The camera unit 80 includes components necessary for photographing, such as a lens, an image sensor, and a distance measuring sensor, and supplies the photographed image of the outside world and depth information to the control unit 10 . The control unit 10 controls focus, zoom, etc. of the camera unit 80 .

The image signal processing unit 82 performs image signal processing (ISP (Image Signal Processing)) such as RGB conversion (demosaicing), white balance, color correction, and noise reduction on the Raw image captured by the camera unit 80. Furthermore, distortion correction processing is performed to remove distortion due to the optical system of the camera unit 80. FIG. The image signal processing unit 82 supplies the camera image that has undergone image signal processing and distortion correction processing to the control unit 10 .

The reprojection unit 84 performs reprojection processing on the camera image based on the latest position/orientation information of the head mounted display 100 detected by the orientation sensor 64, and performs reprojection processing on the camera image from the latest viewpoint position/line-of-sight direction of the head mounted display 100. Convert to visible image.

The distortion processing unit 86 distorts the reprojection-processed camera image in accordance with the distortion generated in the optical system of the head mounted display 100 to distort the image. The captured camera image is supplied to the control unit 10 .

The AR superimposing unit 88 generates an augmented reality image by superimposing a CG image generated by the image generating device 200 on the camera image subjected to distortion processing, and supplies the augmented reality image to the control unit 10 .

The HDMI transmitting/receiving unit 90 transmits/receives video/audio digital signals to/from the image generation device 200 according to HDMI. The HDMI transmission/reception unit 90 receives the RGB image and the depth information subjected to image signal processing and distortion correction processing by the image signal processing unit 82 from the control unit 10, and transmits them to the image generation device 200 through the HDMI transmission line. The HDMI transmission/reception unit 90 receives the image generated by the image generation device 200 from the image generation device 200 via the HDMI transmission line, and supplies the image to the control unit 10 .

The control unit 10 can supply image and text data to the output interface 30 for display on the display panel 32, and can supply the data to the communication control unit 40 for external transmission.

The current position/orientation information of the head mounted display 100 detected by the orientation sensor 64 is notified to the image generation device 200 via the communication control section 40 or the external input/output terminal interface 70 . Alternatively, the HDMI transmitting/receiving section 90 may transmit the current position/orientation information of the head mounted display 100 to the image generation device 200 .

FIG. 8 is a functional configuration diagram of the image generation device 200 according to the base technology. The figure is a block diagram focusing on functions, and these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

At least part of the functions of the image generation device 200 may be implemented in the head mounted display 100 . Alternatively, at least part of the functions of image generation device 200 may be implemented in a server connected to image generation device 200 via a network.

The position/posture acquisition unit 210 acquires the current position/posture information of the head mounted display 100 from the head mounted display 100 .

The viewpoint/line-of-sight setting unit 220 uses the position/orientation information of the head mounted display 100 acquired by the position/orientation acquisition unit 210 to set the user's viewpoint position and line-of-sight direction.

The HDMI transmitting/receiving unit 280 receives the depth information of the video of the real space captured by the camera unit 80 from the head mounted display 100 and supplies the depth information to the depth acquiring unit 250 .

The image generation unit 230 reads out data necessary for generating computer graphics from the image storage unit 260, renders the objects in the virtual space to generate a CG image, and converts the camera image in the real space provided by the depth acquisition unit 250 into a CG image. A chromakey image is generated from the CG image based on the depth information, and is output to the image storage unit 260 .

The image generation unit 230 includes a rendering unit 232 , a chromakey generation unit 235 , a post-processing unit 236 , a reprojection unit 240 and a distortion processing unit 242 .

The rendering unit 232 renders an object in the virtual space seen in the line-of-sight direction from the viewpoint position of the user wearing the head-mounted display 100 according to the user's line-of-sight position and line-of-sight direction set by the viewpoint/line-of-sight setting unit 220, and performs CG rendering. An image is generated and supplied to the chromakey generation unit 235 .

The chromakey generation unit 235 generates a chromakey image from the CG image based on the depth information of the camera image given from the depth acquisition unit 250 . Specifically, the positional relationship between the object in the real space and the object in the virtual space is determined, and the background of the virtual object and the part of the object in the real space that is in front of the virtual object in the CG image are colored in a specific color (for example, red). ) to generate a chromakey image (referred to as a “CG chromakey image”).

A post-processing unit 236 performs post-processing such as depth-of-field adjustment, tone mapping, and anti-aliasing on the CG chroma-key image, and performs post-processing so that the CG chroma-key image looks natural and smooth.

The reprojection unit 240 receives the latest position/orientation information of the head mounted display 100 from the position/orientation acquisition unit 210 , performs reprojection processing on the post-processed CG chromakey image, and performs reprojection processing on the head mounted display 100 . Convert to an image that can be seen from the latest viewpoint position and line-of-sight direction.

Here, reprojection will be described. When the head-mounted display 100 is provided with a head tracking function and a virtual reality image is generated by changing the viewpoint and line-of-sight direction in conjunction with the movement of the user's head, there is a delay from the generation of the virtual reality image to its display. Therefore, a deviation occurs between the orientation of the user's head assumed at the time of video generation and the orientation of the user's head when the video is displayed on the head-mounted display 100, and the user feels drunk. A sensation (called “Virtual Reality Sickness” or the like) may occur.

In this way, the movement of the head mounted display 100 is detected, the CPU issues a drawing command, the GPU (Graphics Processing Unit) executes rendering, and the drawn image is output to the head mounted display 100. time consuming. It is assumed that drawing is performed at a frame rate of 60 fps (frames per second), for example, and that there is a delay of one frame between detection of movement of head-mounted display 100 and output of an image. At a frame rate of 60 fps, this is about 16.67 milliseconds, which is enough time for humans to perceive the shift.

Therefore, a process called "time warp" or "reprojection" is performed to correct the rendered image according to the latest position and orientation of the head mounted display 100, thereby making it difficult for humans to perceive the deviation.

The distortion processing unit 242 distorts the reprojection-processed CG chromakey image in accordance with the distortion occurring in the optical system of the head-mounted display 100 . memorize to

The HDMI transmission/reception unit 280 reads the frame data of the CG chromakey image generated by the image generation unit 230 from the image storage unit 260, and transmits it to the head mounted display 100 according to HDMI.

FIG. 9 is a diagram illustrating the configuration of an image generation system according to the base technology for superimposing a CG image on a camera image to generate an augmented reality image. Here, in order to simplify the explanation, the main configurations of the head-mounted display 100 and the image generation device 200 for generating an augmented reality image will be illustrated and explained.

A camera image of the outside world captured by the camera unit 80 of the head mounted display 100 and the depth information are supplied to the image signal processing section 82 . The image signal processing unit 82 performs image signal processing and distortion correction processing on the camera image, and supplies the processed image to the reprojection unit 84 . The image signal processing unit 82 transmits the depth information to the image generation device 200 and supplies it to the chromakey generation unit 235 .

The rendering unit 232 of the image generation device 200 generates a virtual object viewed from the viewpoint position/line-of-sight direction of the user wearing the head mounted display 100 , and provides it to the chromakey generation unit 235 .

The chromakey generator 235 generates a CG chromakey image from the CG image based on the depth information. A post-processing unit 236 applies post-processing to the CG chromakey image. The reprojection unit 240 converts the post-processed CG chromakey image so as to match the latest viewpoint position and line-of-sight direction. A distortion processing unit 242 applies distortion processing to the CG chromakey image after reprojection. The final RGB image after distortion processing is transmitted to the head mounted display 100 and supplied to the AR superimposing section 88 . This RGB image is an image in which the area on which the camera image is to be superimposed is filled with one color (for example, red) designated by chromakey synthesis. Since one color specified as chroma key cannot be used as a CG image, the one color specified as chroma key is avoided in the CG image and another color is used. For example, if you want to use the same color as the chromakey color in the CG image, you can use it by changing 1 bit of the chromakey color.

The reprojection unit 84 of the head-mounted display 100 converts the camera image, which has been subjected to image signal processing and distortion correction processing, so as to match the latest viewpoint position and line-of-sight direction, and supplies it to the distortion processing unit 86 . A distortion processing unit 86 performs distortion processing on the reprojected camera image. The AR superimposing unit 88 generates an augmented reality image by superimposing the CG chromakey image supplied from the image generating device 200 on the camera image after distortion processing. The generated augmented reality image is displayed on the display panel 32 .

In the image generation system according to the base technology described above, the CG chromakey image generated by the chromakey generation unit 235 undergoes post-processing by the post-processing unit 236, re-projection processing by the re-projection unit 240, and distortion processing by the distortion processing unit 242. , aliasing occurs at the boundary of the virtual object, and false colors that do not actually exist occur, and when superimposed on the camera image by the AR superimposing unit 88, the unnaturalness becomes conspicuous. In addition, although general image transmission interfaces support RGB, they do not support RGBA including alpha values. . In order to represent the shadow cast by the virtual object on the real space and the reflection of the virtual object in the real space, it is necessary to synthesize a translucent CG image with the camera image.

In the following, image generation systems according to several embodiments that overcome the problems in the image generation system according to the base technology will be described, but explanations overlapping with the base technology will be omitted as appropriate, and configurations improved from the base technology will be described. .

A first embodiment will be described. The configuration of the head mounted display 100 is the same as that shown in FIG.

FIG. 10 is a functional configuration diagram of the image generation device 200 according to the first embodiment.

The HDMI transmission/reception unit 280 receives the camera image captured by the camera unit 80 and the depth information from the head mounted display 100 , and supplies the camera image to the camera image acquisition unit 252 and the depth information to the depth acquisition unit 250 . This depth information is depth information of the camera image in the physical space, and is called "camera depth information".

The image generation unit 230 reads data necessary for generating computer graphics from the image storage unit 260, renders objects in the virtual space to generate a CG image, and applies CG to the camera image provided from the camera image acquisition unit 252. Images are superimposed to generate a temporary superimposed image, and a chromakey image is generated from the CG image based on the camera depth information provided from the depth acquisition unit 250 . Post-processing, reprojection processing, and distortion processing are performed on the temporary superimposed image, and reprojection processing and distortion processing are performed on the chromakey image. Note that the chromakey image is not post-processed. Finally, by masking the temporary superimposed image with a chromakey image, a final synthesized CG chromakey image is generated and output to the image storage unit 260 .

The image generating unit 230 includes a rendering unit 232, a superimposing unit 234, a chromakey generating unit 235, a post-processing unit 236, reprojection units 240a and 240b, distortion processing units 242a and 242b, and a synthesizing unit 244. .

The rendering unit 232 renders an object in the virtual space seen in the line-of-sight direction from the viewpoint position of the user wearing the head-mounted display 100 according to the user's line-of-sight position and line-of-sight direction set by the viewpoint/line-of-sight setting unit 220, and performs CG rendering. Generate an image. When the rendering unit 232 renders an object in the virtual space, depth information of the virtual object (referred to as “scene depth information”) is written to a depth buffer for virtual space rendering (referred to as “scene depth buffer”), and determine the context of A specific depth value is not written in the scene depth buffer for pixels where no virtual object is drawn, and the scene depth value is infinite (undefined).

Furthermore, the rendering section 232 renders the physical object in the physical space captured by the camera unit 80 . Shape information and depth information of objects in the real world can be obtained by 3D scanning the space in the real world and recognizing the space. For example, the depth information of the real space can be obtained using depth sensors such as infrared pattern, structured light, and TOF (Time Of Flight), or the depth information of the real space can be obtained from the parallax information of a stereo camera. In this way, the real space is 3D scanned in advance and modeled with a polygon mesh structure. The rendering unit 232 renders walls, floors, ceilings, stationary objects, and the like in the real space, but renders them in white without providing color information. When the rendering unit 232 renders an object in the physical space, depth information of the physical object (referred to as “real space depth information”) is written to a depth buffer for physical space rendering (referred to as “real space depth buffer”), Determine context between objects.

The reason why the real space depth buffer is provided separately from the scene depth buffer is that if the depth value is written to the scene depth buffer when rendering the real space, the scene depth value and the real space depth value cannot be distinguished, and the chroma key area and the This is because it becomes impossible to determine whether or not to

The rendering unit 232 expresses light in the virtual space with respect to the real space. Visible expressions, lighting expressions by virtual light sources in virtual space, etc. are drawn as translucent CG images. For example, shadow mapping can draw shadows and reflections by using methods such as a method of projecting a depth map from a light source onto a plane and methods such as ray tracing. By superimposing a translucent CG image of the shadow or reflection of the virtual object on the low-resolution camera image of the real space, the shadow or reflection of the virtual object in the real space can be expressed. Since objects in the real space are rendered in solid white, they can be distinguished from areas where shadows and reflections are drawn. A rendering area in the real space is specified as a chromakey area, but an area where shadows and reflections are drawn is not specified as a chromakey area.

The rendering unit 232 provides the superimposing unit 234 and the chromakey generating unit 235 with a CG image in which expressions related to light in the virtual space, such as the virtual object and the shadow or reflection of the virtual object, are drawn.

The superimposing unit 234 superimposes the CG image on the low-resolution camera image with delay provided from the camera image acquiring unit 252 to generate a temporary superimposed image, and supplies the temporary superimposed image to the post-processing unit 236 . The superimposition unit 234 superimposes a low-resolution camera image with a delay on the area where the scene depth value is infinite (that is, the area where the virtual object is not drawn) and where the real space depth value is written. . As a result, the camera image with low resolution and delay is superimposed on the CG image in which the shadows and reflections are rendered translucent while the color information of the shadows and reflections remains. That is, the color information of shadows and reflections and the color information of a low-resolution camera image with delay are alpha synthesized.

Here, it should be noted that the real space depth value is not written and the camera image is not superimposed on the area where the scene depth value is written. As described later, when empty areas such as holes are created by transforming the mesh structure of the real space, the real space depth value is not written in the empty area, but the scene depth value is written, so the color will not be chroma key. Therefore, the camera image is not superimposed. Instead, it is possible to render the virtual space in the empty area and give the effect that the other side can be seen through the empty area.

A post-processing unit 236 performs post-processing on the temporary superimposed image, and performs post-processing so that the temporary superimposed image looks natural and smooth.

The first reprojection unit 240a receives the latest position/orientation information of the head mounted display 100 from the position/orientation acquisition unit 210, performs reprojection processing on the post-processed provisional superimposed image, The image is converted into an image that can be seen from the latest viewpoint position/line-of-sight direction of the mount display 100 .

The first distortion processing unit 242 a performs distortion processing on the temporary superimposed image that has been subjected to the reprojection processing, and supplies the result to the synthesizing unit 244 .

The chromakey generation unit 235 generates a chromakey image based on the camera depth information provided from the depth acquisition unit 250, the scene depth information provided from the rendering unit 232, and the real space depth information. Specifically, the area where the scene depth value is infinite or the real space depth value is written, and the area other than the shadow and reflection drawing area is painted with chromakey color, and the camera depth information and A chromakey image is generated by referring to the scene depth information and painting the portion of the real object in front of the virtual object with the chromakey color.

Chroma key images are not post-processed.

The second reprojection unit 240 b receives the latest position/orientation information of the head mounted display 100 from the position/orientation acquisition unit 210 , performs reprojection processing on the chromakey image, and obtains the latest viewpoint position of the head mounted display 100 .・Convert to an image that can be seen from the line of sight.

The second distortion processing unit 242b performs distortion processing on the reprojection-processed chromakey image, and supplies it to the synthesizing unit 244. FIG.

The synthesizing unit 244 synthesizes the temporary superimposed image using the chromakey image as a mask, generates a synthesized CG chromakey image, and stores it in the image storage unit 260 .

Here, the first reprojection unit 240a performs reprojection processing on the temporary superimposed image, and the first distortion processing unit 242a performs distortion processing on the temporary superimposed image after reprojection processing. In this case, interpolation processing such as bilinear interpolation is generally performed when sampling pixels. On the other hand, when the second reprojection unit 240b performs reprojection processing on the chromakey image, and when the second distortion processing unit 242b performs distortion processing on the chromakey image after reprojection processing, Point sampling occurs when pixels are sampled. This is because if interpolation processing such as bilinear interpolation is performed on the chromakey image, the chromakey color will be mixed with the non-chromakey color and become a different color, and the chromakey image will lose its meaning.

In reprojection processing and distortion processing for a chromakey image, the following two methods may be used in addition to the point sampling described above. In the first method, bilinear interpolation is performed between chromakey colors and non-chromakey colors at the time of sampling to calculate intermediate color pixel values. sets the pixel value to the chromakey color, and sets the pixel value to the original non-chromakey color if the interpolated pixel value exceeds the threshold. According to this method, intermediate colors caused by interpolation are discarded, and depending on whether or not they exceed the threshold, chromakey colors are mixed with non-chromakey colors, because chromakey colors or the original non-chromakey colors are determined. no.

The second method is to sample 4 pixels near the point referred to by (u, v) when generating the chromakey image, and set the chromakey color if all of the 4 pixels near the point meet the chromakey conditions. , 4 pixels in the neighborhood do not meet the chromakey conditions, the chromakey color is not set. Nine pixels may be selected instead of four pixels as a method of selecting neighboring pixels. According to this method, since the edge boundary portion is not painted with the chromakey color, the CG image and the camera image appear to merge more naturally near the boundary.

The HDMI transmission/reception unit 280 reads the frame data of the composite CG chromakey image generated by the image generation unit 230 from the image storage unit 260, and transmits it to the head mounted display 100 according to HDMI.

FIG. 11 is a diagram illustrating the configuration of an image generation system according to the first embodiment for superimposing a CG image on a camera image to generate an augmented reality image.

A camera image of the outside world captured by the camera unit 80 of the head mounted display 100 and the camera depth information are supplied to the image signal processing section 82 . The image signal processing unit 82 performs image signal processing and distortion correction processing on the low-delay and high-resolution camera image, and supplies the processed image to the reprojection unit 84 . Further, the image signal processing unit 82 transmits the camera image subjected to the image signal processing and the distortion correction processing and the camera depth information to the image generation device 200. The camera image is transmitted to the superimposition unit 234, and the camera depth information is transmitted to the chromakey generation unit. 235. The camera images sent to the image generation device 200 are delayed and of low resolution.

The rendering unit 232 of the image generation device 200 renders a real object in the real space, generates a virtual object seen from the viewpoint position/line-of-sight direction of the user wearing the head mounted display 100, and converts the light in the virtual space to the real space. It renders expressions related to objects, specifically shadows cast by virtual objects on real objects, reflections of virtual objects, translucency of virtual objects, and lighting expressions using virtual light sources. The rendering unit 232 provides the generated CG image to the superimposing unit 234 and provides the scene depth information and real space depth information to the chromakey generating unit 235 .

The superimposing unit 234 superimposes the camera image on the CG image to generate a temporary superimposed image, and supplies it to the post-processing unit 236 . Here, the camera image provided from the head mounted display 100 may have a low resolution and a delay. This is because the part of the camera image in the temporary superimposed image is masked with the chromakey image and finally erased. The superimposing unit 234 superimposes the camera image on the area in which the scene depth value is not written but the real space depth value is written. Therefore, the camera image is superimposed on an area where no virtual object is drawn and an area where expressions related to light in the virtual space, such as shadows and reflections, are drawn. In a region where an expression related to the light in the virtual space, such as a shadow or reflection, is drawn, the color information of the expression related to the light in the virtual space, such as a shadow or reflection, is combined with the low-resolution camera image as a translucent CG image.

A post-processing unit 236 applies post-processing to the temporary superimposed image. The reprojection unit 240a converts the post-processed provisional superimposed image so as to match the latest viewpoint position and line-of-sight direction. The distortion processing unit 242 a applies distortion processing to the temporary superimposed image after reprojection, and supplies it to the synthesizing unit 244 .

The chromakey generation unit 235 generates a chromakey image based on the camera depth information, scene depth information, and real space depth information. Areas in which scene depth values are not written are filled with the chromakey color, but translucent CG areas where representations related to light in the virtual space, such as shadows and reflections, are drawn are not painted with the chromakey color. In addition, based on the camera depth information and scene depth information, the positional relationship between the objects in the real space and the objects in the virtual space is determined in real time. Filled with chromakey color. The reprojection unit 240b converts the chromakey image so as to match the latest viewpoint position and line-of-sight direction. The distortion processing unit 242b applies distortion processing to the chromakey image after reprojection, and supplies it to the synthesizing unit 244. FIG.

The combining unit 244 generates a combined CG chromakey image by masking the temporary superimposed image with the chromakey image. Here, since the temporary superimposed image has undergone post-processing, it is a smooth image, and the boundary between the camera image and the CG image is inconspicuous. On the other hand, chromakey images are not post-processed, so there is no aliasing or false colors at the borders of virtual objects. Therefore, by masking the post-processed superimposed image with a non-post-processed chromakey image, a natural and smooth composite CG chroma-key image is synthesized without aliasing or false colors at the boundary of the virtual object. .

Even if the scene depth value is infinite, semi-transparent CG areas with shadows and reflections are not filled with chromakey colors, so the semi-transparent CG areas with shadows and reflections are rendered as low-resolution camera images. are left in the synthesized CG chromakey image in a superimposed state.

The synthesized CG chromakey image is transmitted to the head-mounted display 100 as an RGB image in which one specific color is designated as chromakey, and supplied to the AR superimposing unit 88 .

The reprojection unit 84 of the head-mounted display 100 converts the low-delay, high-resolution camera image that has undergone image signal processing and distortion correction processing so as to match the latest viewpoint position and line-of-sight direction, and sends it to the distortion processing unit 86. supply. A distortion processing unit 86 performs distortion processing on the low-delay and high-resolution camera image after reprojection. The AR superimposing unit 88 generates an augmented reality image by superimposing the synthesized CG chromakey image supplied from the image generating device 200 on the low-delay and high-resolution camera image after distortion processing. The generated augmented reality image is displayed on the display panel 32 . A low-delay and high-resolution camera image is superimposed on the area where the chromakey color is specified on the head-mounted display 100 side.

FIG. 12 is a diagram for explaining an augmented reality image in which a CG image is superimposed on a camera image by the image generation system according to the first embodiment. As in FIG. 5 , the user is about to touch the virtual object teapot 410 , and the virtual object teapot 410 is superimposed on the camera image in which the hand 420 is captured. A shadow 412 cast by a teapot 410, which is a virtual object, on a real table is drawn as semi-transparent CG so as to be superimposed on the camera image. Therefore, the flower pattern on the actual table surface can be seen through the drawing area of the shadow 412 .

FIG. 13 is a diagram for explaining a composite CG chromakey image used by the image generation system according to the first embodiment. In the synthesized CG chromakey image, the drawing area of the virtual object (here, the teapot) and the drawing area of the shadow and reflection of the virtual object (here, the shadow cast by the teapot on the floral table surface) are CG, and the rest is chromakey. filled with color. Also, if there is a real moving object (here, the user's hand) in front of the virtual object, the portion of the moving object in front is also painted with the chromakey color.

The augmented reality image shown in FIG. 12 is generated by superimposing the synthesized CG chromakey image shown in FIG. 13 on the high-resolution camera image on the head-mounted display 100 side. Note that the shaded portion of the teapot 410 is composited with the low resolution camera image, but the rest of the camera image is high resolution, including the area of the user's hand.

By using the image generation system of the first embodiment, it is possible to virtually make a hole in a wall, ceiling, table, or the like in the real space and superimpose a CG image. This will be described with reference to FIGS. 14 and 15. FIG.

FIG. 14 is a diagram illustrating an example of deforming a polygon mesh in the real space to make a hole in a wall. A polygon mesh in the real space is obtained by 3D scanning, and by deforming it, a hole 430 can be provided in the wall as shown in FIG. Another representation of the hole 430 may utilize a texture with an alpha mask.

Since no physical space object exists in the hole 430 , depth information is not written to the area corresponding to the hole 430 in the real space depth buffer when the rendering unit 232 renders the physical space. On the other hand, the rendering unit 232 can render a virtual object in the hole 430, and depth information is written in the area corresponding to the hole 430 in the scene depth buffer. Therefore, when the superimposing unit 234 superimposes the camera image on the CG image, the camera image is not superimposed on the hole 430 for which the real space depth value is not set, and the hole 430 for which the scene depth value is set. A CG image is displayed.

FIG. 15 is a diagram illustrating an example in which a virtual object is drawn in a hole in a wall in real space. A virtual hole 440 is drawn by CG in FIG. 15 corresponding to the hole 430 in FIG. 14 , and a virtual car is drawn by CG on the other side of the virtual hole 440 .

According to the image generation system of the first embodiment, since post-processing alleviates discomfort at the boundary between the camera image and the CG image, and post-processing is not applied to the chromakey image, the chromakey image is free from aliasing and false colors. can be generated to generate a high-quality synthetic CG chromakey image. Since an augmented reality image is generated by superimposing this synthesized CG chromakey image on a camera image, an augmented reality image without unnaturalness can be generated. Also, even if the CG image has a semi-transparent part, it can be superimposed on the camera image and subjected to post-processing. Also, since the synthesized CG chromakey image is an RGB image with one specific color as the chromakey, it can be transmitted through a general communication interface such as HDMI capable of transmitting RGB images.

In the above description, the synthesized CG chromakey image was an RGB image with one specific color as the chromakey, but the chromakey color of the chromakey image may be stored in the RGBA alpha component. That is, until the intermediate processing of the post-process, the chromakey color part of the image is processed with the RGBA alpha component transparent, and in the final stage of the post-process, the color of the part where the alpha component is transparent is replaced with the chromakey color. may This eliminates the need to use two separate frame buffers for storing the temporary superimposed image and the chromakey image, and the temporary superimposed image and the chromakey image can be processed in one frame buffer.

A second embodiment will be described. The configuration of the head-mounted display 100 is basically the same as that shown in FIG. It has a reprojection part 84b.

FIG. 16 is a functional configuration diagram of an image generation device 200 according to the second embodiment.

The HDMI transmission/reception unit 280 receives the camera image captured by the camera unit 80 and the depth information from the head mounted display 100 , and supplies the camera image to the camera image acquisition unit 252 and the depth information to the depth acquisition unit 250 .

The image generation unit 230 reads data necessary for generating computer graphics from the image storage unit 260, renders objects in the virtual space to generate a CG image, and applies CG to the camera image provided from the camera image acquisition unit 252. Images are superimposed to generate a temporary superimposed image, and a chromakey image is generated from the CG image based on the camera depth information provided from the depth acquisition unit 250 . A post-process is applied to the temporary superimposed image, but a post-process is not applied to the chromakey image. Finally, by masking the temporary superimposed image with a chromakey image, a final synthesized CG chromakey image is generated and output to the image storage unit 260 .

The image generating section 230 includes a rendering section 232 , a superimposing section 234 , a chromakey generating section 235 , a post-processing section 236 and a synthesizing section 244 .

The rendering unit 232 renders an object in the virtual space seen in the line-of-sight direction from the viewpoint position of the user wearing the head-mounted display 100 according to the user's line-of-sight position and line-of-sight direction set by the viewpoint/line-of-sight setting unit 220, and performs CG rendering. Generate an image. When rendering an object in the virtual space, the rendering unit 232 writes scene depth information to the scene depth buffer.

Furthermore, the rendering section 232 renders the physical object in the physical space captured by the camera unit 80 . When the rendering unit 232 renders the physical space object, it writes the physical space depth information of the physical object to the physical space depth buffer.

The rendering unit 232 provides the superimposing unit 234 and the chromakey generating unit 235 with a CG image in which expressions related to light in the virtual space, such as the virtual object and the shadow or reflection of the virtual object, are drawn.

The superimposing unit 234 superimposes the CG image on the low-resolution camera image with delay provided from the camera image acquiring unit 252 to generate a temporary superimposed image, and supplies the temporary superimposed image to the post-processing unit 236 . The superimposing unit 234 superimposes a low-resolution camera image with a delay on an area where the scene depth value is infinite and where the real space depth value is written.

A post-processing unit 236 performs post-processing on the temporary superimposed image, and performs post-processing so that the temporary superimposed image looks natural and smooth.

The chromakey generation unit 235 generates a chromakey image based on the camera depth information provided from the depth acquisition unit 250, the scene depth information provided from the rendering unit 232, and the real space depth information.

The synthesizing unit 244 synthesizes the temporary superimposed image using the chromakey image as a mask, generates a synthesized CG chromakey image, and stores it in the image storage unit 260 .

The HDMI transmission/reception unit 280 reads the frame data of the composite CG chromakey image generated by the image generation unit 230 from the image storage unit 260, and transmits it to the head mounted display 100 according to HDMI.

FIG. 17 is a diagram illustrating the configuration of an image generation system according to the second embodiment for superimposing a CG image on a camera image to generate an augmented reality image.

A camera image of the outside world captured by the camera unit 80 of the head mounted display 100 and the camera depth information are supplied to the image signal processing section 82 . The image signal processing unit 82 performs image signal processing and distortion correction processing on the low-delay and high-resolution camera image, and supplies the processed image to the reprojection unit 84 . Further, the image signal processing unit 82 transmits the camera image subjected to the image signal processing and the distortion correction processing and the camera depth information to the image generation device 200 , and transmits the camera image to the superimposition unit 234 and the depth information to the chromakey generation unit 235 . supplied to The camera images sent to the image generation device 200 are delayed and of low resolution.

The rendering unit 232 of the image generating device 200 renders a real object in the real space, generates a virtual object seen from the viewpoint position/line-of-sight direction of the user wearing the head mounted display 100, and renders the virtual object onto the real object. Render lighting expressions such as shadows and reflections of virtual objects. The rendering unit 232 provides the generated CG image to the superimposing unit 234 and provides the scene depth information and real space depth information to the chromakey generating unit 235 .

The superimposing unit 234 superimposes the camera image on the CG image to generate a temporary superimposed image, and supplies it to the post-processing unit 236 . As in the first embodiment, the camera image provided from the head-mounted display 100 may be of low resolution and delayed.

The post-processing unit 236 applies post-processing to the temporary superimposed image and supplies it to the synthesizing unit 244 .

The chromakey generation unit 235 generates a chromakey image based on the camera depth information, the scene depth information, and the real space depth information, and provides the chromakey image to the synthesis unit 244 .

The combining unit 244 generates a combined CG chromakey image by masking the superimposed image with the chromakey image. As in the first embodiment, by masking the post-processed superimposed image with a non-post-processed chromakey image, there is no aliasing or false color at the boundary of the virtual object, and natural and smooth synthesis is achieved. A CG chromakey image is synthesized.

The synthesized CG chromakey image is transmitted to the head-mounted display 100 as an RGB image in which one specific color is designated as chromakey, and supplied to the reprojection unit 84b.

The first reprojection unit 84a of the head-mounted display 100 converts the low-delay, high-resolution camera image that has been subjected to image signal processing and distortion correction processing so as to match the latest viewpoint position and line-of-sight direction, and superimposes the AR image. 88.

The second reprojection unit 84 b of the head-mounted display 100 converts the synthesized CG chromakey image so as to match the latest viewpoint position/line-of-sight direction, and supplies it to the AR superimposition unit 88 .

Here, in the head mounted display 100, the reason why the first reprojection unit 84a for the camera image and the second reprojection unit 84b for the synthesized CG chroma key image are divided is that the rendering of the image generation device 200 takes time. This is because the amount of difference to be corrected differs depending on the reprojection. For example, while the first reprojection section 84a reprojects one frame forward, the second reprojection section 84b needs to reproject two frames forward.

The AR superimposing unit 88 converts the composite CG chromakey image reprojected by the second reprojection unit 84b into a low-delay and high-resolution camera image reprojected by the first reprojection unit 84a. , an augmented reality image is generated and supplied to the distortion processing unit 86 . A low-delay and high-resolution camera image is superimposed on the area where the chromakey color is specified on the head-mounted display 100 side.

A distortion processor 86 applies distortion processing to the augmented reality image. The generated augmented reality image is displayed on the display panel 32 .

According to the image generation system of the second embodiment, similar to the first embodiment, since an augmented reality image is generated by superimposing a synthetic CG chromakey image on a camera image, augmented reality without unnaturalness can be generated. In addition to the advantage of being able to generate images, there are the following advantages. Unlike the first embodiment, since the head-mounted display 100 performs reprojection processing on the camera image and the synthesized CG chromakey image, the viewpoint position and line-of-sight direction are adjusted to match the viewpoint position and line-of-sight direction immediately before being displayed on the display panel 32 . It can convert a camera image and a synthesized CG chromakey image, and can provide an augmented reality image with high accuracy and followability. In addition, since the load of the reprojection processing on the image generation device 200 side can be reduced, more resources can be used for rendering on the image generation device 200 side.

In the first and second embodiments, a configuration was described in which the chromakey image was not post-processed by the post-processing unit 236. However, as a modified example, the chromakey image may be adjusted for viewpoint adjustment or scaling. may be configured to apply post-processing to . In this case, if a method such as using the average value of surrounding pixels is adopted when interpolating pixels, the color of the boundary changes to a color different from the chromakey color. Therefore, it is necessary to apply a post-process so as not to change the chromakey color. Alternatively, after applying a normal post-process to the chromakey image that changes the color of the border, only the areas that exactly match the original chromakey color may be used as a mask.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the embodiments are examples, and that various modifications can be made to combinations of each component and each treatment process, and that such modifications are within the scope of the present invention. .

In the above description, processing such as depth-of-field adjustment, tone mapping, and anti-aliasing are examples of post-processing, but distortion processing, simple enlargement/reduction, trapezoidal transformation, and the like may also be called post-processing.

10 control unit 20 input interface 30 output interface 32 display panel 40 communication control unit 42 network adapter 44 antenna 50 storage unit 64 attitude sensor 70 external input/output terminal interface 72 external memory 80 camera unit , 82 image signal processing unit, 84 reprojection unit, 86 distortion processing unit, 88 AR superimposition unit, 100 head mounted display, 200 image generation device, 210 position/orientation acquisition unit, 220 viewpoint/line of sight setting unit, 230 image generation unit , 232 rendering unit, 234 superimposition unit, 235 chromakey generation unit, 236 post-processing unit, 238 scene depth buffer, 239 real space depth buffer, 240 reprojection unit, 242 distortion processing unit, 244 synthesis unit, 250 depth acquisition unit, 252 Camera image acquisition unit 260 Image storage unit 280 HDMI transmission/reception unit 300 Interface.

Claims (10)

a rendering unit that renders an object in a virtual space and an object in a real space, and renders a representation of light in the virtual space with respect to the real space to generate a computer graphics image;
a superimposing unit that superimposes the computer graphics image on the captured image of the real space to generate a temporary superimposed image;
a chromakey generating unit that generates a chromakey image by performing chromakey processing on the computer graphics image based on the depth information of the captured image of the physical space;
a synthesizing unit that generates a synthesized chromakey image that is used to generate an augmented reality image by superimposing the virtual superimposed image with the chromakey image to superimpose the captured image of the physical space on the superimposed image. ,
The chromakey generation unit is characterized in that a region of the real space in which the object in the virtual space is not rendered is set as a chromakey region, while a region of the real space in which an expression related to light in the virtual space exists is not set as a chromakey region. image generation device. The representation of the light in the virtual space with respect to the real space includes the shadow or reflection of the object in the virtual space on the object in the real space, the representation in which the back of the object in the virtual space can be seen through, and the representation of the virtual space in the virtual space. 2. The image generating apparatus according to claim 1, wherein the image generating apparatus is at least one lighting representation by a light source. The rendering unit deforms the polygon mesh structure obtained by spatially recognizing the physical space to generate a free space in which the virtual space is rendered, and then renders the physical space based on the deformed polygon mesh structure. 2. The image generation device of claim 1, wherein the device renders an object. further comprising a post-processing unit for post-processing the temporary superimposed image;
The synthesizing unit generates the synthetic chromakey image by masking the post-processed temporary superimposed image with the chromakey image that has not been post-processed. Item 4. The image generation device according to any one of Items 1 to 3. further comprising a reprojection unit that converts the post-processed temporary superimposed image and the chromakey image so as to match a new viewpoint position or line-of-sight direction;
The composition unit generates the composite chromakey image by masking the temporary superimposed image that has been subjected to the reprojection processing with the chromakey image that has been subjected to the reprojection processing. Item 5. The image generation device according to any one of Items 1 to 4. The captured image of the physical space used by the superimposing unit to generate the temporary superimposed image has a lower resolution than the captured image of the physical space used to generate the augmented reality image. 6. The image generation device according to any one of claims 1 to 5. 7. The image generating apparatus according to claim 6, wherein the representation of light in said virtual space with respect to said real space is superimposed on said captured image with low resolution as a translucent computer graphics image. An image generation system including a head-mounted display and an image generation device,
The image generation device is
a rendering unit that renders an object in a virtual space and an object in a real space, and renders a representation of light in the virtual space with respect to the real space to generate a computer graphics image;
a first superimposing unit that superimposes the computer graphics image on the captured image of the real space transmitted from the head-mounted display to generate a temporary superimposed image;
a chromakey generating unit that generates a chromakey image by chromakey processing the computer graphics image based on depth information of the captured image of the real space transmitted from the head-mounted display;
a synthesizing unit that generates a synthesized chromakey image that is used to generate an augmented reality image by superimposing the virtual superimposed image with the chromakey image to superimpose the captured image of the physical space on the superimposed image. ,
The head mounted display is
a second superimposing unit that generates an augmented reality image by synthesizing the composite chromakey image transmitted from the image generation device with the captured image of the physical space;
The chromakey generation unit is characterized in that a region of the real space in which the object in the virtual space is not rendered is set as a chromakey region, while a region of the real space in which an expression related to light in the virtual space exists is not set as a chromakey region. image generation system. a rendering step of rendering a virtual space object and a real space object, and rendering a representation of light in said virtual space relative to said real space to generate a computer graphics image;
a superimposing step of superimposing the computer graphics image on a photographed image of the real space to generate a temporary superimposed image;
a chromakey generating step of generating a chromakey image by chromakey processing the computer graphics image based on the depth information of the captured image of the physical space;
and a synthesizing step of generating a synthesized chromakey image that is superimposed on the captured image of the physical space and used to generate an augmented reality image by masking the temporary superimposed image with the chromakey image. ,
In the chromakey generating step, a real space area in which an object in the virtual space is not rendered is set as a chromakey area, and a real space area in which an expression related to light in the virtual space exists is not set as a chromakey area. image generation method. a rendering function that renders an object in a virtual space and an object in a real space and renders a representation of light in the virtual space with respect to the real space to generate a computer graphics image;
a superimposing function that superimposes the computer graphics image on the captured image of the real space to generate a temporary superimposed image;
a chromakey generation function for generating a chromakey image by performing chromakey processing on the computer graphics image based on depth information of the captured image of the physical space;
a synthesizing function for generating a synthesized chromakey image that is used to generate an augmented reality image by superimposing the virtual superimposed image with the chromakey image and superimposing it on the captured image of the physical space; to realize
The chromakey generation function is characterized in that a region of the real space in which the object in the virtual space is not rendered is set as a chromakey region, while a region of the real space in which an expression related to light in the virtual space exists is not set as a chromakey region. A program to

Images