JP2006323450A - Simulation image generator, simulation image generation method, computation program, and recording medium recorded with program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow an interactive viewing of high real time property, without causing a speed reduction in use, even when using a large capacity of highly definition three-dimensional data to reproduce an object with high quality. <P>SOLUTION: This simulation image generator has an image data generated based on the three-dimensional data expressing an object surface within a three-dimensional scene, and based on camera information expressing a view point operation for photographing the object within the three-dimensional scene, and for indicating information as to the object surface in a corresponding point on the object surface corresponding to a position of each picture element in every of the picture elements, a function for inputting the camera information, a function for inputting a simulation condition, and a function for finding a picture element value when the object is brought into the simulation condition within the three-dimensional scene, in the every picture element, based on the image data and the camera information, and for displaying it. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for generating a simulation image.

  By using virtual reality technology as a means for digitizing and preserving three-dimensional objects such as arts and crafts, it is possible to interactively reproduce the appearance of an object under an arbitrary observation position and lighting conditions.

  The virtual reality technology here means 3D data (material information and texture image showing the 3D shape data of the object and the surface of the object) obtained by measuring a real 3D object such as arts and crafts with a live action. Based on the observation position, illumination conditions, etc., based on the observation position and lighting conditions, etc. To achieve a virtual experience.

  In order to improve the reality of the virtual experience, in addition to the image quality of the video, the real-time property such as the smoothness of the movement following the movement of the viewpoint is important. However, if it is attempted to accurately reproduce the details of the target object, the data amount of the three-dimensional shape data and the texture image increases, leading to a reduction in rendering processing speed.

  In addition, when display quality is given the highest priority, real-time rendering (virtual reality) technology is not used, and still images and videos are generated by high-precision rendering that takes a long time. In this case, virtual reality is used. The interactive function that is the feature of can not be realized.

  As described above, in the case of using the virtual reality technology, in order to improve the accuracy and detail of an image obtained as a result of the rendering process (hereinafter also referred to as a result image), the refinement of the three-dimensional shape data or the texture image As a result, the rendering process speed is reduced. On the other hand, in order to ensure the real-time performance of the rendering process, the detail level is reduced by reducing the amount of data. Full reproduction of details and subtle textures is difficult.

  By the way, the processing load for the pixels of the result image, such as color tone conversion of the entire screen, is naturally irrelevant to the level of detail of the three-dimensional shape data, and can be executed in a fixed time.

  However, such post-rendering processing is limited to color conversion, luminance conversion, filter processing, and the like based on image data such as RGB images, and is based on three-dimensional data such as highly useful illumination reproduction and texture reproduction. Processing cannot be executed.

  The present invention has been made in view of such background technology, and even when using high-definition and large-capacity three-dimensional data for high-quality reproduction of an object, real-time performance is not accompanied by a decrease in speed during use. The challenge is to enable highly interactive viewing.

In order to achieve the above object in the present invention, first, the invention of claim 1 represents three-dimensional data representing the surface of the object in the three-dimensional scene and a viewpoint operation for photographing the object in the three-dimensional scene. A function for inputting, for each pixel, image data indicating information on the object surface at a corresponding point of the object surface corresponding to the position of each pixel, and camera information, for each pixel;
The ability to enter simulation conditions;
A simulation characterized by having a function of obtaining and displaying a pixel value when an object is placed under a simulation condition in a three-dimensional scene for each pixel based on image data and camera information This is an image generation apparatus.

The invention according to claim 2 is a method executed by the simulation image generating apparatus according to claim 1,
Generated based on 3D data representing the surface of the object in the 3D scene and camera information representing the viewpoint operation for photographing the object in the 3D scene. For each pixel, the position of each pixel Image data indicating information on the object surface at corresponding points on the object surface corresponding to, and camera information is input,
A process for entering simulation conditions;
A simulation including a step of obtaining and displaying a pixel value when an object is placed under a simulation condition in a three-dimensional scene for each pixel based on image data and camera information This is an image generation method.

According to a third aspect of the present invention, it is obtained from three-dimensional data representing the surface of an object in a three-dimensional scene and camera information at each time point representing a viewpoint operation for photographing the object at each time point in the three-dimensional scene. When each frame is shot, each image data is generated for each frame of the obtained moving image, and each pixel is imaged, and each image data indicating information on the object surface at the corresponding point on the object surface corresponding to the position of each pixel. A function to input information about each camera used;
The ability to enter simulation conditions;
A function for inputting information for specifying a video sequence to be played;
Each frame of the video sequence is picked up sequentially, and the pixel value when the target object is placed under simulation conditions in the three-dimensional scene for each pixel based on the image data and camera information of the picked up frame. The simulation image generating apparatus is characterized by having a function of obtaining and displaying.

The invention according to claim 4 is a method executed by the simulation image generating apparatus according to claim 3,
For each frame of the video obtained by 3D data representing the surface of the object in the 3D scene and camera information at each time point representing the viewpoint operation for photographing the object at each time point in the 3D scene For each pixel, each image data indicating information on the object surface at the corresponding point of the object surface corresponding to the position of each pixel, and each camera information used when each frame is captured Input process,
A process for entering simulation conditions;
A process for inputting information specifying a video sequence to be played;
Each frame of the video sequence is picked up sequentially, and the pixel value when the target object is placed under simulation conditions in the three-dimensional scene for each pixel based on the image data and camera information of the picked up frame. And a step of obtaining and displaying the simulation image generation method.

Moreover, in invention of Claim 5, it is performed by the simulation image generation apparatus of Claim 1 or 3,
An arithmetic program that defines, for each pixel, a calculation for obtaining a pixel value when an object is placed under a simulation condition in a three-dimensional scene based on image data and camera information. It is what.

  According to a sixth aspect of the present invention, there is provided a recording medium on which the arithmetic program according to the fifth aspect is recorded.

  According to the first and second aspects of the present invention, even when high-definition and large-capacity three-dimensional data is used for high-quality reproduction of an object with respect to a still image, interactive with high real-time characteristics is not accompanied by a decrease in speed during use. This has the effect of making it possible to enjoy the appreciation.

  According to the third and fourth aspects of the present invention, even when high-definition and large-capacity three-dimensional data is used for high-quality reproduction of a target object for a moving image, the real-time interactive performance is not accompanied by a decrease in speed during use. This has the effect of making it possible to enjoy the appreciation.

  According to the fifth and sixth aspects of the present invention, in the simulation apparatus according to the first or third aspect, an object is placed in a simulation condition for each pixel in a three-dimensional scene based on image data and camera information. There is an effect that it is possible to execute an operation for obtaining a pixel value when it is set.

  The best embodiment of the present invention will be described below.

  As shown in FIG. 1, this embodiment relates to a computer, a simulation image generation apparatus including a browser device and a display.

  The computer stores three-dimensional data and camera path data in a storage device included in the computer, and includes an image generation calculation unit and an image output unit by executing a program.

  The browser device stores a calculation program in a storage device included in the browser device, and further includes an operation unit, a control unit, a moving image reproduction unit, and an image calculation unit, and the image calculation unit is connected to a display. is doing.

  A moving image file is input from the image output unit to the moving image playback unit via a storage medium or a network. The moving image file includes a color image sequence, a depth image sequence, a normal image sequence, a material image sequence, a camera information sequence, and a material list.

  First, the computer will be described.

  The three-dimensional data is composed of three-dimensional shape data of the object, material information indicating the property of the object surface, a texture image, and the like, and represents the object surface in the three-dimensional scene. The three-dimensional data may be obtained by measuring a real three-dimensional object such as arts and crafts with a live-action photograph, or may be obtained by drawing a fictitious three-dimensional object using a computer. .

The camera path data is composed of information in which camera information including a viewpoint position in a three-dimensional scene, a line-of-sight rotation angle (line-of-sight direction), and a viewing angle are arranged one by one in time order. The camera information at each time represents the viewpoint (camera) operation in the three-dimensional scene at each time, and by these viewpoint (camera) operations, the object represented by the three-dimensional data is displayed in the three-dimensional scene. A video can be obtained by shooting. This moving image is a sequence of still images called frames taken by a camera in the order of shooting.

  The image generation calculation unit performs rendering processing by a well-known method based on the camera information of the time corresponding to each frame for each frame of the moving image based on the three-dimensional data and the camera path data. Depth image data, normal image data, and material image data are sequentially generated.

  As shown in FIG. 2, the color image data is information indicating the color of reflected light (reflectance of each RGB) at a corresponding point on the surface of the object corresponding to the position of each pixel for each pixel on the screen. is there. However, it does not include components such as shadows and surface reflections that change according to environmental conditions such as lighting conditions and camera information.

  As shown in FIG. 3, the depth image data includes one-dimensional distance information (depth value D) from the viewpoint to the corresponding point on the surface of the object corresponding to the position of each pixel for each pixel on the screen. It is information which shows. The depth value D is expressed using a screen coordinate system.

  As shown in FIG. 4, the normal image data is information indicating a normal vector N (Nx, Ny, Nz) at a corresponding point on the surface of the object corresponding to the position of each pixel for each pixel on the screen. It is. This normal vector N (Nx, Ny, Nz) is represented using a three-dimensional scene coordinate system.

  As shown in FIG. 5, the material image data indicates, for each pixel on the screen, a reference number of a material list in which material information representing optical characteristics of the surface of the object corresponding to the position of each pixel is stored. Information. The material information includes specular reflectance and diffuse reflectance for each color of RGB. The material information is also used for the rendering process.

  The image output unit outputs a moving image file including a color image sequence, a depth image sequence, a normal image sequence, a material image sequence, a camera information sequence, and a material list. Here, the color image sequence is information in which color image data belonging to each frame of a moving image is arranged according to the order of each frame in the moving image. The depth image sequence is information in which depth image data belonging to each frame of a moving image is arranged according to the order of each frame in the moving image. The normal image sequence is information in which normal image data belonging to each frame of a moving image is arranged according to the order of each frame in the moving image. The material image sequence is information in which material images belonging to each frame of the moving image are arranged according to the order of the frames in the moving image. The camera information sequence is the camera information of the time corresponding to each frame used for sequentially generating color image data, depth image data, normal image data, and material image data for each frame of the moving image. The information is arranged according to the order of each frame in the moving image. The material list is a list in which material information used for the rendering process is stored.

  Here, the relationship between the three-dimensional scene coordinate system used for describing the three-dimensional data and the screen coordinate system will be described with reference to FIG.

  In virtual reality technology, to generate an image from a 3D scene, a method of generating perspective images by converting each coordinate point in the 3D scene to screen coordinates by coordinate transformation known as perspective projection is common. Is. Further, the value indicating the position of the coordinate point after coordinate conversion along the screen depth direction is the depth value.

In the present embodiment, when generating an image using the viewpoint A, the point P (X, Y, Z) in the three-dimensional scene is converted into a point S (Xs, Ys) on the screen and a depth value D. Function
Assume that (Xs, Ys, D) = F (X, Y, Z).

  The inverse function is (X, Y, Z) = Finv (Xs, Ys, D).

  These conversion functions F () and Finv () are determined by viewpoint operations such as a viewpoint position, a line-of-sight rotation angle (line-of-sight direction), and a viewing angle, and change depending on the viewpoint operation.

  Next, the browser device will be described.

  The operation unit causes the user to input information specifying a moving image sequence to be reproduced, and causes the user to input simulation conditions necessary for execution of the arithmetic program, including lighting conditions. Here, the moving image sequence is arranged in the order in which frames are reproduced.

  Information specifying the video sequence to be reproduced includes, for example, each frame shot between a specific frame shot before in time and a specific frame shot after in time, It may be specified to reproduce in the shooting order. Also, for example, each frame shot between a specific frame shot after time and a specific frame shot before time is played back in the reverse order of shooting. You may specify what to do. Further, for example, it may be specified that only one specific frame is reproduced.

  Based on the input from the operation unit, the control unit performs reproduction control of the moving image sequence, setting of processing conditions necessary for execution of the operation program including the lighting conditions, and control of the moving image reproduction unit and the image calculation unit.

  The video playback unit sequentially plays back color image data, depth image data, normal image data, material image data, and camera information for each frame constituting the video sequence from each sequence in the video file. The material list is read from the file, and each image data and camera information for each frame is passed to the image calculation unit together with the material list.

  The image calculation unit reproduces each frame by displaying on the display by performing calculation based on the calculation program on the data for each frame passed from the moving image reproduction unit. Real-time processing of high-definition moving images may be performed by using high-speed hardware for arithmetic logic.

  The calculation program defines the content of calculation in the image calculation unit.

  Hereinafter, an example of the flow of processing executed by the browser device for lighting simulation will be described according to the flowchart of FIG. 8 with reference to FIG.

S (STEP) 1;
The control unit places an omnidirectional natural point light source having a luminous intensity I at coordinates (Xl, Yl, Zl) in the three-dimensional scene based on the illumination condition input from the operation unit. Thus, the illuminance at the distance L from the light source is expressed as E = I / L ² .

S (STEP) 2;
The control unit sets the first frame of the moving image sequence specified by the information input from the operation unit as the frame of interest.

S (STEP) 3;
The video playback unit plays the color image data, depth image data, normal image data, material image data, and camera information of the frame of interest from each sequence in the video file, and reads the material list from the video file. Each image data and camera information of the frame of interest is passed to the image calculation unit together with the material list.

S (STEP) 4;
The image calculation unit sets the first pixel on the screen as the pixel of interest S (Xs, Ys).

S (STEP) 5;
The image calculation unit reads the diffuse reflection color (R, G, B) in the target image S (Xs, Ys) from the color image data of the target frame.

S (STEP) 6;
The image calculation unit reads the depth value D at the target pixel S (Xs, Ys) from the depth image data of the target frame.

S (STEP) 7;
The image calculation unit reads a normal vector N (Nx, Ny, Nz) at the target pixel S (Xs, Ys) from the normal image data of the target frame.

S (STEP) 8;
The image calculation unit determines a conversion function Finv () from the screen coordinate system to the three-dimensional scene coordinate system based on the camera information of the frame of interest.

S (STEP) 9;
The image calculation unit obtains a three-dimensional scene coordinate point P (X, Y, Z) = Finv (Xs, Ys, D) on the surface of the object corresponding to the target pixel S (Xs, Ys).

S (STEP) 10;
The image calculation unit calculates the position vector V (Vx, Vy, Vz) of the light source (Xl, Yl, Zl) with respect to P (X, Y, Z) = (Xl, Yl, Zl)-(X, Y, Z). Ask for.

S (STEP) 11;
The image calculation unit sets the length of the position vector V, that is, the distance between the light source and the coordinate point P as L = | V |, and the incident angle of light from the light source to the target surface corresponding to the target pixel S as θ. Cos θ = (Vx · Nx + Vy · Ny + Vz · Nz) / L is obtained.

S (STEP) 12;
The image calculation unit sets (R ′, G ′, B ′) = E ′ (R), where R ′, G ′, B ′ are the pixel values of the target frame to be displayed on the target pixel S (Xs, Ys). , G, B). Here, E ′ is the illuminance on the surface of the object corresponding to the target pixel S, and E ′ = I · Cos θ / L ^{2 according} to a well-known illuminance calculation formula.

S (STEP) 13;
The image calculation unit confirms whether or not the pixel value of the target frame has been obtained for all the pixels on the screen,
If the pixel value of the frame of interest has not been calculated for all the pixels on the screen, STE
Proceed to P14
When the pixel values of the target frame are obtained for all the pixels on the screen, the process proceeds to STEP15.

S (STEP) 14;
The image calculation unit newly sets the pixel next to the target pixel S (Xs, Ys) on the screen as the target pixel S (Xs, Ys), and then returns to STEP5.

S (STEP) 15;
The image calculation unit displays the target frame on the display based on the pixel value of the target frame at each pixel on the screen.

S (STEP) 16;
The control unit confirms whether all the frames of the moving image sequence specified by the information input from the operation unit are displayed on the display,
If not all frames are displayed on the display, go to STEP 17
If all frames are displayed on the display, go to END.

S (STEP) 17;
The control unit newly sets a frame next to the frame of interest in the moving image sequence specified by the information input from the operation unit, and then returns to STEP 3.

  The calculation program may be defined so as to cause the image calculation apparatus to perform the calculations shown in the following (1) to (4).

  (1) A specular reflection component may be added to the illumination simulation described above by using the specular reflectance included in the material image.

  (2) By increasing the depth value, the display pixel value may be decreased to add a depth effect that makes the near side brightly rise.

  (3) A depth-of-field effect may be added by performing a blur filter process in which the “blur amount” is changed according to the depth value. Specifically, the “blurring amount” is set to 0 for the depth value corresponding to the focal position, and the “blurring amount” is increased as the depth value moves away from the focal position.

  (4) By changing the material for each part of the target, partial color or texture change processing may be performed for each material.

  As can be seen from the above description, in this embodiment, various types of information obtained from three-dimensional data are recorded and reproduced in a format that depends only on the resolution of the final image that is independent of the definition of the three-dimensional data. In addition, various interactive simulations including illumination reproduction and texture reproduction can be performed on high-definition images in the form of still images or moving images.

The figure explaining the structure of each apparatus concerning this embodiment. The figure explaining color image data. The figure explaining depth image data. The figure explaining normal image data. The figure explaining material image data. The figure explaining the relationship between a three-dimensional scene coordinate system and a screen coordinate system. The figure explaining the example of illumination simulation. The flowchart which shows the example of the flow of the process which a browser apparatus performs for lighting simulation.

Claims (6)

Generated based on 3D data representing the surface of the object in the 3D scene and camera information representing the viewpoint operation for photographing the object in the 3D scene. For each pixel, the position of each pixel A function of inputting image data indicating information on the object surface at the corresponding point on the object surface corresponding to, and camera information;
The ability to enter simulation conditions;
A simulation characterized by having a function of obtaining and displaying a pixel value when an object is placed under a simulation condition in a three-dimensional scene for each pixel based on image data and camera information Image generation device. A method executed by the simulation image generating apparatus according to claim 1,
Generated based on 3D data representing the surface of the object in the 3D scene and camera information representing the viewpoint operation for photographing the object in the 3D scene. For each pixel, the position of each pixel Image data indicating information on the object surface at corresponding points on the object surface corresponding to, and camera information is input,
A process for entering simulation conditions;
A simulation including a step of obtaining and displaying a pixel value when an object is placed under a simulation condition in a three-dimensional scene for each pixel based on image data and camera information Image generation method. For each frame of the video obtained by 3D data representing the surface of the object in the 3D scene and camera information at each time point representing the viewpoint operation for photographing the object at each time point in the 3D scene For each pixel, each image data indicating information on the object surface at the corresponding point on the object surface corresponding to the position of each pixel, and each camera information used when each frame is captured A function to input,
The ability to enter simulation conditions;
A function for inputting information for specifying a video sequence to be played;
Each frame of the video sequence is picked up sequentially, and the pixel value when the target object is placed under simulation conditions in the three-dimensional scene for each pixel based on the image data and camera information of the picked up frame. A simulation image generation device having a function of obtaining and displaying the simulation image. A method executed by the simulation image generation device according to claim 3,
For each frame of the video obtained by 3D data representing the surface of the object in the 3D scene and camera information at each time point representing the viewpoint operation for photographing the object at each time point in the 3D scene For each pixel, each image data indicating information on the object surface at the corresponding point of the object surface corresponding to the position of each pixel, and each camera information used when each frame is captured Input process,
A process for entering simulation conditions;
A process for inputting information specifying a video sequence to be played;
Each frame of the video sequence is picked up sequentially, and the pixel value when the target object is placed under simulation conditions in the three-dimensional scene for each pixel based on the image data and camera information of the picked up frame. And a step of obtaining and displaying the simulation image generation method. Performed by the simulation image generating device according to claim 1 or 3,
An arithmetic program that defines, for each pixel, a calculation for obtaining a pixel value when an object is placed under a simulation condition in a three-dimensional scene based on image data and camera information. .   A recording medium on which the arithmetic program according to claim 5 is recorded.