WO2015196791A1

WO2015196791A1 - Binocular three-dimensional graphic rendering method and related system

Info

Publication number: WO2015196791A1
Application number: PCT/CN2015/070601
Authority: WO
Inventors: 王文敏; 张建龙; 王荣刚; 董胜富; 王振宇; 李英; 高文
Original assignee: 北京大学深圳研究生院
Priority date: 2014-06-27
Filing date: 2015-01-13
Publication date: 2015-12-30
Also published as: CN105224288A; CN105224288B

Abstract

The present invention relates to a binocular three-dimensional graphic rendering method and a related system. The method comprises a projection transformation step. The projection transformation step comprises: adding a middle plane between a near plane and a far plane as a projection plane, and projecting primitives between the near plane and the far plane onto the middle plane. The primitives between the near plane and the far plane are projected onto the middle plane through the added middle plane, so the primitives between the near plane and the middle plane achieve a stereo effect of out of the screen, and the primitives between the middle plane and the far plane achieve a stereo effect of into the screen. Therefore, no specific hardware is required for rendering the effects of "out of the screen" and "into the screen" when the existing rendering pipelines are used in a 3D display device.

Description

Binocular 3D graphics rendering method and related system

Technical field

The present invention relates to the field of stereoscopic vision processing technologies, and in particular, to a binocular three-dimensional (Stereoscopic 3D) graphics rendering method and related system.

Background technique

It is well known that the real world is a three-dimensional three-dimensional world. When the human eye is watching the three-dimensional world, since the eyes are horizontally separated at two different positions, the images of the objects seen are different. The visual angles of the images seen by the left and right eyes are different, and are called left view and right view, respectively. Due to the existence of parallax, through the convergence of the human brain, people can feel the stereoscopic 3D world with depth of field and layering. This is the principle of binocular parallax. According to this principle, if you can see two different views of the different angles of view, you can feel a stereoscopic 3D view with depth of field and layering.

The 3D display is designed based on the principle of binocular parallax. When real-time stereoscopic 3D images are to be generated, it is necessary to generate images of the left and right eyes in real time, so that the viewer continues to feel the stereoscopic effect. This process usually needs to be done by modifying the Rendering Pipeline. The 3D graphics rendering pipeline is responsible for performing a series of necessary steps to convert the 3D scene into a 2D image that can be displayed on the display. The 3D graphics rendering pipeline typically includes the following steps: conversion from a local coordinate system to a world coordinate system; conversion from a world coordinate system to a view coordinate system; projection transformation; and viewport transformation. At present, a popular graphics API (Application Programming Interface) and OpenGL (Open Graphic Library) have their own rendering pipeline.

OpenGL can be used for monocular rendering. It is a cross-platform, cross-programming API suitable for rendering stereoscopic 3D graphics on traditional 2D displays. OpenGL is usually supported on the GPU hardware platform. The GPU rendering pipeline is a hardware acceleration and efficient process for converting 3D information into 2D images. OpenGL also provides an application programming interface for binocular 3D rendering, but requires corresponding support on the GPU hardware, otherwise OpenGL will not be able to present binocular 3D effects on 3D displays.

In addition, based on the existing GPU (Graphic Processing Unit) rendering pipeline, the stereoscopic effect mainly includes two types of screen entry and screen release. The screen entry means that the object seen seems to be behind the screen, and the screen appears to be seen. The effect of the object is like the front of the screen. For example, when rendering a flame spray effect, the screen can give a sense of fire to the human sensory effect. In the stage of projection transformation of conventional GPU rendering, for example, using OpenGL for rendering, the viewpoint is at the origin and observed along the -z direction, forming a pyramid-shaped frustum, that is, two far and near, parallel to each other. A cone formed by truncation of planes (called far and near planes). Any primitives outside the cone will be cropped, and the primitives left in the cone will undergo a perspective transformation. Shadowing onto the near plane, the pseudo depth obtained by the perspective transformation is used as the basis for judging whether the pixel is visible or not. If you simply use the viewpoint displacement and depth information, only the primitives behind the near plane will be projected onto the near plane, which can only achieve the stereo effect of entering the screen.

Summary of the invention

According to a first aspect of the present invention, the present invention provides a binocular three-dimensional graphics rendering method, including a projection transformation step, the projection transformation step comprising: adding a midplane as a projection plane between a near plane and a far plane, and a near plane The primitives between the far planes are projected onto the midplane.

According to a second aspect of the present invention, the present invention provides a stereoscopic image reproduction method, including:

Creating step: creating at least two view frame buffers for respectively storing image data of different viewpoints;

a rendering step: receiving data of a three-dimensional graphic of at least two viewpoints, and rendering data of each viewpoint, the rendering comprising storing the rendering result into a corresponding video frame buffer using the binocular three-dimensional graphics rendering method as described above Area;

The synthesizing step is: synthesizing the rendering results in the at least two view frame buffers to obtain a stereoscopic frame, and outputting the stereoscopic frames.

According to a third aspect of the present invention, the present invention provides a stereoscopic image reproduction system, including:

Creating a module for creating at least two view frame buffers for respectively storing image data of different viewpoints;

a rendering module, configured to receive data of a three-dimensional graphic of at least two viewpoints, and render data of each view, the rendering comprising storing the rendering result into a corresponding view by using a binocular three-dimensional graphics rendering method as described above Frame buffer

And a synthesizing module, configured to synthesize the rendering result in the at least two view frame buffers to obtain a stereoscopic frame, and output the stereoscopic frame.

According to a fourth aspect of the present invention, the present invention provides a binocular three-dimensional graphics rendering and display system, comprising:

a storage device for saving a data file containing a three-dimensional graphic;

a processor, configured to parse the data file in the storage device;

Processor memory for providing at least two view frame buffers for respectively storing data of different viewpoints;

a graphics processor, configured to implement three-dimensional graphics rendering on the processed data file by the processor, where the rendering includes generating a view frame of different viewpoints by using a binocular three-dimensional graphics rendering method as described above;

The processor memory is further configured to store a view frame of different views generated by the graphics processor;

The processor is further configured to synthesize the view frames of the different views to obtain a stereo frame;

A three-dimensional display for displaying the stereoscopic frame.

The binocular three-dimensional graphics rendering method and related system according to the present invention are increased by In the middle plane, the primitive between the near plane and the far plane is projected onto the midplane, so that the primitive between the near plane and the middle plane has a stereoscopic effect, and the primitive between the middle plane and the far plane will have The stereo effect of entering the screen; thus, the existing "output screen" and "on screen" effects can be rendered without using special hardware when using the existing rendering pipeline in the 3D display device.

DRAWINGS

1 is a three-dimensional schematic diagram of a projection transformation stage extension in a GPU rendering pipeline;

Figure 2 is a schematic diagram of the principle of binocular parallax;

3 is a schematic structural diagram of a binocular three-dimensional graphics rendering and display system according to an embodiment of the present invention;

4 is a graphics processing pipeline of a GPU in an embodiment of the present invention;

FIG. 5 is a schematic flow chart of a method for reproducing a stereo image according to an embodiment of the present invention.

detailed description

The present invention will be further described in detail below with reference to the accompanying drawings.

First explain some of the terms or concepts used below. The left-view frame refers to a two-dimensional view indicating the left eye of the viewer in the three-dimensional image display; similarly, the right-view frame refers to a two-dimensional view indicating the right eye of the viewer in the three-dimensional image display. . A stereoscopic frame refers to a 3D view frame obtained by combining a rendered left view frame and a right view frame according to the type of the 3D display.

The OpenGL rendering pipeline is known to convert 3D information into 2D images in real time and efficiently. However, as described in the foregoing background art, the OpenGL rendering pipeline used by the existing GPU cannot simultaneously consider the screen entry and the screen when rendering, and can realize the 3D rendering based on binocular parallax without the GPU hardware support.

Example 1:

In order to enable the existing GPU rendering pipeline to render the in-screen effect and render the screen effect without corresponding support on the GPU hardware, a binocular 3D graphics rendering method in this embodiment adds a midplane. (middle plane), as shown in FIG. 1, the middle plane is between the near plane and the far plane, and the middle plane is regarded as a projection plane, and the primitive between the near plane and the far plane is projected onto the middle plane, and then The primitive between the plane and the mid-plane has a three-dimensional effect of the screen, and the primitive between the middle plane and the far plane has a stereo effect of entering the screen. In order to ensure that only the primitives between the near plane and the far plane are preserved during the cropping phase, the pseudo depth must remain unchanged, ie between [-1, 1].

Let the distances of the near plane, the middle plane and the far plane from the origin be N, M, F, respectively. The pseudo depth between the near plane and the far plane is [-1, 1], then the perspective transformation matrix M _{1 is} as follows:

After the perspective transformation, it is also necessary to shift and scale the result of the perspective into the cube between [-1, 1]. Let the coordinates of the top, bottom, left, and right of the midplane be top, bottom, left, and right, respectively. The matrix representation M _{2 of} the displacement and scaling operations is as follows:

You can use θ to represent the size of the observer's perspective. The projection surface is the mid-plane. Top, bottom, left, and right can be calculated by the following equation:

Bottom=-top

Left=-right

Since the model view transformation stage shifts the viewpoint from the origin to the horizontal direction by e/2 and -e/2, it is necessary to make some corrections to the projection transformation. The correction is related to the displacement, and the displacement amount is represented by s, and the correction matrix M _{3 is} as follows:

In summary, the matrix of the projection transformation stage can be represented by the matrix M _PT as follows:

In the projection matrix M _PT of the left eye, s=-e/2; in the projection matrix M _PT of the right eye, s=e/2.

Example 2:

2 shows a view in which the angles of the left and right eyes are different due to binocular parallax, and the buffers for storing the left and right eye views are the left view frame and the right view frame, respectively, and the object objects in the left and right view frames are related to the depth of field.

The present embodiment provides a binocular three-dimensional graphics rendering and display system based on the binocular parallax principle and the GPU rendering pipeline. As shown in FIG. 3, it is a schematic structural diagram of the system, and the system includes five modules:

(1) The external storage device 200 is configured to store scene data, such as 3D grid data, image data, configuration data, and the like;

(2) a processor (CPU) 201: for parsing files, processing of scene data, and synthesizing operations of stereoscopic frames;

(3) GPU 202: implements the main components of the graphics rendering pipeline to generate left and right views;

(4) processor memory 204: storing left and right view frame buffers and program data;

(5) 3D display 203: for displaying stereoscopic frames.

The CPU parses the scene data from the file of the external memory, stores it in the processor memory, and selectively sends the scene data to the GPU hardware through a CPU or DMA (Direct Memory Access) according to the rendering command. The 3D rendering of the scene is done through the GPU's rendering pipeline. The GPU needs to perform left and right view frame rendering on the scene data. After each rendering, the data in the frame buffer needs to be transferred from the GPU to the processor memory. After the rendering of the left and right view frames is completed, according to the type of the 3D display, the left and right view frames in the processor memory are combined into a binocular 3D view frame, and then the 3D view frame is transmitted to the frame buffer of the GPU, and finally displayed on the 3D display. .

4 is a rendering pipeline of the GPU. The vertex data 300 stored in the memory is called object coordinates. First, the transformation of the object view coordinates into viewpoint coordinates is performed through the transformation of the model view matrix 301. The origin of the viewpoint coordinates is the position of the camera. The viewpoint coordinates are transformed by the transformation matrix 302 stage, including three steps of perspective, displacement and scaling transformation, and the coordinate control is transformed into a cube between [-1, 1]. The coordinates obtained at this time are called the clipping coordinates, and the z component of the coordinates. Called pseudo depth. In the transformation of the projection matrix stage, plane shading or smooth shading is also performed. If the lighting and texture are added, the calculation of the pixel values between the vertex is also implemented. Next is the cropping 303 stage, cropping the primitives in the non-[-1,1] cube area, and only the primitives in the [-1,1] solid area are visible to the user. Performing the perspective division 304 operation on the quaternary homogeneous coordinates of the visible segment, A ternary normalized device coordinate is obtained; it is then converted to coordinates on the screen by window transformation 305. In order to improve the processing efficiency of texture data, such as texture storage format conversion between memory and GPU, GPU provides a dedicated data channel. Through the GPU rendering pipeline, the 3D information can be converted into a 2D image in real time and efficiently, and stored in a frame buffer in the GPU for display of the display.

The projection transformation process in the rendering according to the embodiment may adopt the method of Embodiment 1, and other processes involved in the rendering, such as model view transformation, and subsequent synthesis, display, and the like may be implemented by referring to commonly used related technologies, and are not used herein. Detailed.

Example 3:

According to an embodiment of the present invention, a stereoscopic image reproduction method is provided, including:

a rendering step: receiving data of a three-dimensional graphic of at least two viewpoints, and rendering data of each viewpoint, the rendering is using the binocular three-dimensional graphics rendering method of Embodiment 1, and storing the rendering result in a corresponding view frame buffer ;

Synthesizing step: synthesizing the rendering results in at least two view frame buffers to obtain a stereoscopic frame, and outputting the stereoscopic frame.

Taking the two viewpoints of the left viewpoint and the right viewpoint as an example, as shown in FIG. 5, the stereoscopic image reproduction method of this embodiment. First read the attribute information of the 3D display, including the length, width, resolution, type of the display (such as side by side or side by side, etc.) and other information; initialize the buffer of the left and right viewpoints in the processor memory, and clear the two buffers; Set the left viewpoint in the GPU rendering pipeline, clear the two buffers; set the projection matrix and model view matrix of the left viewpoint in the GPU rendering pipeline; render the entire scene, store the result in the left buffer; set the right viewpoint in the GPU rendering pipeline Projection matrix and model view matrix; render the entire scene, store the rendering result into the right buffer; according to the 3D display type, format or synthesize the left and right frames, and then copy to the frame buffer; finally, binocular 3D The view frame is rendered onto the 3D display.

Based on the foregoing method embodiment, the present invention further provides a stereoscopic image reproduction system, including:

a rendering module, configured to receive data of the three-dimensional graphics of the at least two viewpoints, and render the data of each of the viewpoints, where the rendering is performed by using the binocular three-dimensional graphics rendering method of the foregoing embodiment, and the rendering result is stored in the corresponding view. Frame buffer

The synthesizing module is configured to synthesize the rendering results in the at least two view frame buffers to obtain a stereoscopic frame, and output the stereoscopic frame.

The implementation of each module refers to the foregoing embodiment, and is not repeated herein.

In summary, the present invention relates to a binocular 3D graphics rendering method and system thereof, which is based on a GPU rendering pipeline and a binocular parallax principle, and provides an OpenGL-compatible binocular 3D rendering method and system capable of shutter, polarization, naked eye, and the like. The 3D display shows stereoscopic 3D effects and is compatible with special effects rendering algorithms in traditional graphics, such as particle systems, texture shading, and shadows. The binocular 3D graphics obtained by the method can present a stereoscopic 3D world with depth of field and layering as well as 3D video.

The invention presents a stereoscopic 3D effect through the 3D display, and solves the problem of how to render the stereoscopic 3D effect when the GPU hardware does not support the OpenGL binocular 3D rendering API. The method adopted is based on the existing GPU rendering pipeline, fully utilizing the hardware acceleration characteristics of the pipeline, improving processing efficiency, compatible with most application programming interfaces of OpenGL, and real-time rendering stereoscopic 3D scenes using the existing OpenGL application programming interface. . Based on the binocular parallax principle, different left and right viewing frames are generated according to the parallax of the left and right eyes and the depth of the object, and finally a binocular 3D viewing frame is synthesized according to the type of the 3D display.

For a hardware platform that does not support binocular 3D rendering (ie, does not provide stereoscopic rendering left and right frame buffers), the binocular 3D rendering method and system provided by the present invention require two left and right viewing frames (required for multi-view situations) Multiple view frames), two view frame buffers can be created in the processor memory, one for storing the left view and the other for storing the right view; and then synthesizing according to the stereoscopic 3D view frame format of the 3D display, the method It is also compatible with most of the special effects rendering algorithms in traditional graphics.

There is only one scene constructed by vertices, textures, etc., and the same scene data can be rendered twice. The model view matrix and the projection matrix are adjusted each time the rendering is performed, and the results of the two renderings are respectively saved into the left and right frame buffers, that is, Copy the data in the GPU frame buffer to the view frame buffer in the processor memory. After obtaining the image data of the left and right view frames, the left and right view frames are combined into a binocular 3D view frame according to the type of the 3D display; finally, the 3D view frame is copied to the frame buffer on the GPU and presented on the 3D display. The rendering process involved includes: setting the model view matrix and projection matrix of the left viewpoint, rendering the scene data, and the frame buffer Copy the data into the left view frame buffer of the processor memory; set the model view matrix and projection matrix of the right view, render the scene data, and copy the data in the frame buffer to the right view frame buffer of the processor memory; The type of 3D display combines two frames of data into a binocular 3D view frame, which is sent to the frame buffer on the GPU; finally, it is rendered onto the 3D display.

A person skilled in the art may understand that all or part of the steps of the various methods in the above embodiments may be completed by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, and the storage medium may include: a read only memory, Random access memory, disk or optical disk, etc.

The invention has been described above in terms of specific examples, which are merely used to help the understanding of the invention, but do not limit the scope of application of the invention. Variations on the basis of the above-described specific embodiments can be made by those skilled in the art based on the idea of the present invention.

Claims

A binocular three-dimensional graphics rendering method includes a projection transformation step, wherein a midplane is added as a projection plane between a near plane and a far plane, and a primitive between the near plane and the far plane is projected onto the midplane.
The binocular three-dimensional graphics rendering method according to claim 1, wherein the projection matrix used in the projection transformation step is:

Where M PT represents the projection matrix, M represents the distance between the midplane and the origin, N represents the distance between the near plane and the origin, and F represents the distance between the far plane and the origin.
Bottom=-top,
Left=-right, θ represents the size of the angle of view, and s represents the amount of displacement of the viewpoint from the origin to the horizontal direction.
A stereoscopic image reproduction method, comprising:

Creating step: creating at least two view frame buffers for respectively storing image data of different viewpoints;

a rendering step: receiving data of a three-dimensional graphic of at least two viewpoints, rendering data of each viewpoint, the rendering comprising storing the rendering result by using the binocular three-dimensional graphics rendering method according to claim 1 or 2 Corresponding view frame buffer;

The synthesizing step is: synthesizing the rendering results in the at least two view frame buffers to obtain a stereoscopic frame, and outputting the stereoscopic frames.
The stereoscopic image reproduction method according to claim 3, wherein

The creating step includes: creating and initializing a left-view frame buffer and a right-view frame buffer in the memory;

The rendering step includes: receiving data of a left-view frame including a three-dimensional graphic, rendering data of the left-view frame, storing the rendering result in the left-view frame buffer, and receiving data of a right-view frame including the three-dimensional graphic, Rendering data of a right-view frame, and storing the rendering result in the right-view frame buffer;

The synthesizing step includes: splicing the rendering result in the left-view frame buffer and the rendering result in the right-view frame buffer to obtain a stereoscopic frame, and storing the stereoscopic frame in the graphic The frame buffer of the processor.
The stereoscopic image reproduction method according to claim 3, further comprising a displaying step of displaying the stereoscopic frame.
A stereoscopic image reproduction system, comprising:

Creating a module for creating at least two view frame buffers for respectively storing image data of different viewpoints;

a rendering module, configured to receive data of a three-dimensional graphic of at least two viewpoints, and render data of each viewpoint, the rendering comprising using the binocular three-dimensional graphics rendering method according to claim 1 or 2 to render the rendering result Stored in the corresponding view frame buffer;

And a synthesizing module, configured to synthesize the rendering result in the at least two view frame buffers to obtain a stereoscopic frame, and output the stereoscopic frame.
The stereoscopic image reproduction system according to claim 6, further comprising a display module for displaying the stereoscopic frame.
A binocular three-dimensional graphics rendering and display system, comprising:

a storage device for saving a data file containing a three-dimensional graphic;

a processor, configured to parse the data file in the storage device;

Processor memory for providing at least two view frame buffers for respectively storing data of different viewpoints;

a graphics processor, configured to implement three-dimensional graphics rendering on the data file processed by the processor, where the rendering comprises using a binocular three-dimensional graphics rendering method according to claim 1 or 2 to generate a view frame of different viewpoints;

The processor memory is further configured to store a view frame of different views generated by the graphics processor;

The processor is further configured to synthesize the view frames of the different views to obtain a stereo frame;

A three-dimensional display for displaying the stereoscopic frame.