JP2005135052A - Method for realizing fog effect in mixed reality space - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem wherein, for a dark overall scene, a conventional mixed reality (MR) technique using a video see-through HMD requires operations of darkening the brightness of captured real video and rendering CG in corresponding darkness, which require a real time change in the brightness of the real video to pose a high processing load. <P>SOLUTION: Real video is rendered in lower brightness with a FOG function of a graphics card and CG is also rendered in lower brightness with the FOG function to present an overall scene to a user as if it was dark. The use of the hardware function of the graphics card for a dark scene enables a dark scene without performance sacrifice. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  It is effective for equipment using MR (Mixed Reality) or AR (Augmented Reality) system using video see-through HMD.

When applying the FOG effect in an MR system using a video see-through HMD, it is necessary to reduce the brightness of the captured live-action video as the CG becomes darker due to FOG.
JP-A-11-136706

  However, in order to reduce the brightness of the live-action video, it is necessary to perform image processing for processing all pixels of the live-action video of 30 frames per second, which is heavy processing.

  The present invention solves this problem by using a FOG function that is supported by hardware in many graphic cards in order to change the brightness of a live-action image.

  As a specific means, when a live-action video is drawn as a background image, the brightness of the live-action video is adjusted by drawing using a FOG parameter different from the FOG parameter given to the CG video.

  At this time, by adjusting the FOG parameters of both so that the CG and the live-action video are in harmony, it is possible to give the FOG effect that the CG and the live-action video are fused without any sense of incongruity.

  As is apparent from the above description, according to the present invention, the FOG effect can be realized in the MR and AR systems without sacrificing the processing speed.

  An overall view of this example is shown in FIG. 1, and a schematic diagram is shown in FIG. The present embodiment is a game (hereinafter referred to as MR game) using MR technology using the FOG function. It consists of HMD 100, 3D sensor main body 200, 3D sensor fixed station 210, computer 300, and game field 400.

  HMD 100 is a video see-through HDM. As shown in FIG. 3, it consists of left-eye and right-eye cameras 110 and 111, a three-dimensional sensor mobile station 120, and left-eye and right-eye LCDs 130 and 131.

  Players play the game with HMD 100. On the LCDs 130 and 131 in the HMD 100, the live-action video shot by the cameras 110 and 111 and the CG image are combined and displayed (FIG. 4).

  In the present embodiment, the FOG-added MR video is presented to the user by individually applying FOG to the live-action video and the CG image (FIG. 5). The FOG mechanism assumes the use of the index official FOG mode of DirectX, which is a graphic API of Windows (registered trademark).

  The video displayed on the HDM 100 is created by the computer 300. The computer 300 includes a CPU 301, a memory 302, a PCI bridge 303, a serial I / O 310, video capture cards 320 and 321, and video boards 330 and 331.

  The three-dimensional position sensor 200 includes a three-dimensional position sensor fixed station 210, an HMD built-in position sensor 120, and a hand position sensor 500. The three-dimensional position sensor 200 measures the relative position of the three-dimensional position sensor fixed station 210 and the HMD built-in position sensor 120 by magnetism. The position information can measure 6 degrees of freedom of x, y, z, roll, pitch, yaw. The computer 300 communicates with a serial interface. The position of the three-dimensional position sensor fixed station 210 is accurately measured in advance, and by knowing the relative position of the HMD built-in position sensor, it is possible to know the absolute position of the HDM (the center of the game field 400 is the origin) .

  A flowchart of the HMD display thread of the computer 300 is shown in FIG. Since the left and right eye processes are basically the same, the right eye process will be described here.

  The live-action video from the video camera 100 is taken into the computer 300 through the video capture card 320 (S100).

  Before drawing a live-action video, FOG parameters for drawing the real-shot video are set (S101). Since the live-action video is drawn as a texture on a two-dimensional board with z buffer = 1, the following values are set for the FOG parameters (the variable foglevel indicates the FOG intensity to be set as 0 to 1).

FOG = 4 * (1-foglevel)
Since this FOG is set as a FOG parameter, the brightness of a live-action image rendered as a texture decreases according to the value of foglevel (when the color of the FOG is black).

  The photographed video is written into the video buffer on the memory 302 (S102). This video buffer is a work area for storing video that is being created. By writing the captured live-action video, the live-action video can be used as a CG background image.

  The position information of the HMD 100 is obtained from the three-dimensional position sensor 200 through the serial I / O 310 (S103). The HMD 100 position is stored after being converted to world coordinates. World coordinates are a coordinate system with the origin of the center of the game field 400 as the origin, and are used throughout the present embodiment. Conversion to the world coordinates can be easily performed by using the positions of the game field 400 and the three-dimensional position sensor fixed station 210 that are measured in advance.

  A matrix for projective transformation that uses CG for drawing and a matrix for modeling transformation are created (S104). Create modeling transformation matrix using world coordinates of HMD. Also, a projection transformation matrix is created from parameters such as the focal length of the camera that have been measured in advance (the projection matrix is created only once for the first time and is used later).

  Set FOG parameters for CG (S105).

FOG = 0.0015-(0.0015 / (1.0 + 0.2)) * (foglevel + 0.2)
Using the completed conversion matrix, CG data is drawn in the video buffer (S106). By overwriting the video buffer, CG is drawn on the live-action video. CG data is moved by another thread as needed. In this embodiment, the fish and rocket CGs are moved, but the method of moving the CGs is not related to the present invention, so the description is omitted.

  Since FOG parameters are set for CG, the screen will be subject to the FOG effect according to the foglevel. The reason why foglevel is not used as the FOG parameter is to adjust the brightness of the live-action video.

  When the drawing is finished, the video buffer is transferred to the frame buffer on the video board 330, and the video is drawn on the LCD 130 in the HMD 100 (S107).

  With the above processing, when realizing the FOG effect in the MR game, the brightness of the live-action image can be adjusted using the FOG mechanism implemented in hardware on many graphic boards. As a result, FOG effects can be realized in MR games without degrading system performance.

FIG. 1 is an overall view of this embodiment. It is a schematic diagram of a present Example. 1 is a schematic diagram of HMD 100. FIG. It is the figure which showed the outline of MR technique. It is the figure which showed a mode that FOG effect was implement | achieved in MR system by applying FOG to a real image and a CG image independently. It is the flowchart which showed the operation | movement (for one eye) of an image display thread | sled.

Explanation of symbols

100 HMD
110, 111 video camera
120 HMD built-in position sensor
130, 131 LCD
200 3D position sensor body
210 3D position sensor fixed station
300 calculator
301 CPU
302 memory
303 PCI bridge
310 serial I / O
320, 321 video capture board
330,331 video board

Claims (3)

  In a system that uses a video see-through HMD (Head Mount Display), when applying FOG to a CG image, adjust the brightness of the actual image used as the background image according to the color that makes the CG image darker by FOG. A system characterized by image processing.   2. The system according to claim 1, wherein the brightness of the live-action image is realized by applying FOG. 3. The system according to claim 2, wherein when applying FOG to the live-action video and the CG video, the brightness is adjusted by using different FOG parameters for the live-action video and the CG video, respectively.

Images