CN102802003A - Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras - Google Patents

Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras Download PDF

Info

Publication number
CN102802003A
CN102802003A CN201210289898XA CN201210289898A CN102802003A CN 102802003 A CN102802003 A CN 102802003A CN 201210289898X A CN201210289898X A CN 201210289898XA CN 201210289898 A CN201210289898 A CN 201210289898A CN 102802003 A CN102802003 A CN 102802003A
Authority
CN
China
Prior art keywords
real
time
gpu
network
network camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201210289898XA
Other languages
Chinese (zh)
Inventor
王琼华
张�杰
李磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN201210289898XA priority Critical patent/CN102802003A/en
Publication of CN102802003A publication Critical patent/CN102802003A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Stereoscopic And Panoramic Photography (AREA)

Abstract

The invention provides a real-time shooting and real-time free stereoscopic display system based on both a GPU (Graphics Processing Unit) and network cameras. The system comprises a stereoscopic network camera, a computer and a free stereoscopic display, wherein the stereoscopic network camera consists of a plurality of network cameras. According to the system, real scenes are real-timely shot through the stereoscopic network camera to obtain a plurality of anaglyphs, then anaglyph correction, real-time anaglyph translational cutting and real-time composite image generation are carried out in the computer, and finally, the composite images are real-timely displayed on the free stereoscopic display, so that a viewer can see stereoscopic images.

Description

Real-time shooting and real-time free stereo display system based on GPU and network camera
Technical Field
The invention relates to the technical field of 3D (three-dimensional) stereo display, in particular to a real-time shooting and real-time free stereo display system based on a GPU (graphic processing unit) and a network camera.
Background
Nowadays, 3D display technology is developed vigorously, and great results are obtained. Among many 3D displays, autostereoscopic display is an important one, which is also called multi-view raster 3D display. The film source required for the autostereoscopic display is generally obtained by means of stereo photography. Stereo photography uses a stereo camera to photograph the same scene to obtain two or more parallax images, the stereo camera is a camera array composed of a plurality of cameras with the same specification, each camera needs to set the same parameters and must photograph synchronously. The stereo camera can conveniently shoot scenes in reality and can provide rich 3D film sources. However, to achieve real-time capturing and displaying, each frame of image captured by each camera needs to be processed in real time, and the amount of data to be processed is very large. Moreover, the existing stereo shooting system is huge in size, has high requirements on the performance of a computer, and is difficult to realize in wide application. Therefore, there is an urgent need for a simple and portable high-performance photographing system.
The network camera is a new generation camera combining the traditional camera and the network technology, and has all the image capturing functions of the traditional camera, and is also internally provided with a digital compression controller and a network-based operating system, so that video data is compressed and encrypted and then is sent to a terminal user through a local area network, the internet or a wireless network. Web-based cameras are becoming more popular but are also limited to 2D (two-dimensional) shots, or non-real-time 3D shots. Compared with the traditional camera, the network camera has the advantages of low cost, convenience in carrying, no need of an external power supply, convenience in connection with a computer and the like. The kernel of the computer display card, namely the GPU, has tens to thousands of Shader kernels, each Shader kernel is similar to a CPU kernel and can complete conventional numerical calculation and simple control functions, the GPU has great advantages in floating point operation, parallel calculation and the like compared with the CPU, and the GPU can provide tens of times or hundreds of times of the performance of the CPU. Direct3D is an API (application programming interface) used by microsoft corporation for operating a GPU, and Direct Show in Direct3D has a good support for a webcam, so that the webcam can be used to better utilize the hardware acceleration function of the GPU to realize processing of very large-scale image data.
Disclosure of Invention
The invention provides a real-time shooting and real-time free stereo display system based on a GPU and a network camera. As shown in fig. 1, the system includes a stereo webcam, a computer, and an autostereoscopic display, the stereo webcam being composed of a plurality of webcams. The system obtains a plurality of parallax images by shooting a real scene in real time through a stereo network camera, then carries out real-time parallax image correction, real-time parallax image translation cutting and real-time synthetic image generation in a computer, and finally displays the synthetic image on an auto-stereo display in real time, so that a viewer can see the stereo image.
Aiming at the problem that a real-time shooting and real-time free stereo display system needs to process overlarge data volume, the technical scheme of the invention is as follows: the real-time parallax image correction, the real-time parallax image translation cutting and the real-time synthetic image generation are realized by fully utilizing the hardware acceleration function of the GPU, and the method is characterized in that: as shown in fig. 2, each network camera of the stereo network camera adopts a parallel arrangement mode, that is, the optical axes of the network cameras are parallel to each other, and the optical centers are located on the same straight line; the distances among all the network cameras are equal, the network cameras are connected with a computer through a PCI card, videos are collected in real time by adopting DirectShow, the same network camera parameters are set, and synchronous shooting of all the network cameras is realized.
Preferably, the invention adopts a real-time parallax image correction method based on the GPU, and videos collected by all the network cameras are corrected by the GPU in real time. The method solves the problem of difficult calibration among a plurality of network cameras, and can correct the deformed parallax images. As shown in fig. 3, the present invention uses a re-projection method to project the original imaging plane of each network camera onto an ideal imaging plane, where (x, y) is a point on the original imaging plane, and (x ', y') is a point on the ideal imaging plane transformed by the re-projection of the point, and the relationship between these two points is as follows:
(1)
wherein, a1, …, a8Are all constant.
Preferably, the invention uses the GPU to implement real-time parallax image panning cropping, thereby changing the position of the zero parallax plane. A plurality of network cameras of the three-dimensional network camera are arranged in parallel and are focused at infinity, so that the whole scene only has negative parallax; as shown in fig. 4, the position of the zero parallax plane can be changed by performing translational cropping on the parallax image in the imaging plane of the network camera. In fig. 5, the thick solid line part is a zero parallax plane after clipping, and the thick dotted line part is a clipping region. The invention carries out translation cutting on the parallax image in real time, so that the zero parallax plane after cutting can be adjusted in real time, each time, one pixel is taken as a unit, and the geometric relationship can be knownnThe left side of the parallax image shot by the network camera in the horizontal direction should be cut off (N-n) Pixel, right side should be cropped: (n-1) pixels, where N represents the number of webcams. The position of the zero parallax plane is changed through real-time parallax image translation clipping, so that the shot scene has good stereoscopic impression.
Preferably, the parallax image after correction and translational cutting is generated into a synthetic image in real time by using a method of GPU multi-time rendering and superposition. Suppose the number of views of an autostereoscopic display isN(equal to the number of the network cameras), the raster inclination angle of the autostereoscopic display is alpha, the width-to-height ratio of the sub-pixels is 1: 3, thennThe offset of the ith row of the parallax image in the autostereoscopic display is δ i, and the value thereof is determined by the following formula:
δi=3×i×tanα+n-1 (2)
wherein 1 is less than or equal tonN. The matrix of the number of viewpoints obtained according to the formula (2) is shown in FIG. 5, in which,1,2,…, NThe image displayed by the sub-pixels is represented by which parallax image, and RGB represents red, green and blue sub-pixels respectively.
The real-time shooting and real-time free three-dimensional display system based on the GPU and the network camera has the characteristics of low cost, simple structure, small size and the like. The system fully utilizes the hardware acceleration function of the GPU to realize real-time parallax image correction, real-time parallax image translation clipping and real-time synthetic image generation, solves the processing problem of ultra-large data volume, and reduces the requirements on the performance of a computer.
Drawings
FIG. 1 is a schematic diagram of a real-time photographing and real-time autostereoscopic display system based on a GPU and a network camera.
Fig. 2 is a schematic diagram of a parallel arrangement of the network cameras of the stereo network camera.
Fig. 3 is a schematic diagram of a reprojection transformation.
Fig. 4 is a schematic diagram of real-time parallax image translation cropping.
Fig. 5 is a view number matrix corresponding to the synthesized image.
The figures in the above figures are numbered:
1 stereo network camera, 2 computer, 3 auto-stereo display, 4 network camera, 5 optical axis of network camera, 6 projection center, 7 original image plane, 8 ideal image plane, 9 zero parallax plane after cutting, 10 cutting area
It should be understood that the above-described figures are merely schematic and are not drawn to scale.
Detailed Description
The present invention will be described in detail with reference to the accompanying drawings, which illustrate embodiments of a real-time capturing and real-time autostereoscopic display system based on a GPU and a webcam in detail. It should be noted that the following examples are only for illustrative purposes and should not be construed as limiting the scope of the present invention, and that the skilled person in the art may make modifications and adaptations of the present invention without departing from the scope of the present invention.
Fig. 1 is a schematic diagram of a real-time shooting and real-time autostereoscopic display system based on a GPU and a web camera according to the present invention, which includes a stereoscopic web camera, a computer, and an autostereoscopic display, wherein the stereoscopic web camera is composed of a plurality of web cameras. In this embodiment, the autostereoscopic display is a 21.5-inch lenticular-grating autostereoscopic display, the 2D resolution is 1920 × 1080, the tilt angle of the grating relative to the display screen is arctan (-0.4550), the number of views is 8, the corresponding stereoscopic webcam adopts 8 webcams, and each webcam acquires an image according to 640 × 480 resolution. The network camera is connected to a computer through a PCI card, and the GPU adopts an NVIDIA GeForce GTX 460 chip.
The invention realizes the real-time parallax image correction by the following steps: firstly, shooting a rectangular pattern placed in a scene, wherein the rectangular pattern needs to contain a plurality of characteristic points and is filled with parallax images as much as possible, shooting to obtain matching points of the characteristic points of the rectangular pattern in 8 parallax images, selecting one parallax image as a reference, easily obtaining coordinates of 4 corner points in the rectangle in the parallax images, manually selecting the coordinates of the 4 matching points in the parallax images, and obtaining the coordinates of the 8 points to obtain the coordinates of the parallax images in the calculation formula (1)a 1, …, a 8And (4) the coefficient.
And secondly, establishing a four-channel rectangular texture with the size of 1920 multiplied by 1080 in a video memory, wherein the texture format is RGBA, and reading a first parallax image captured by the stereo network camera into the texture memory as the texture. Setting a rendering mode as alpha color mixing, setting a texture addressing mode as frame addressing, and setting a texture filter as a linear texture filtering mode.
Thirdly, obtaining the coordinates of each pixel in the parallax image according to the pixel Shader, and (C)XY) And the texture coordinate in the GPU is fixed to (0, 1), and each pixel coordinate is respectively positioned inXAndYinterpolation is carried out according to (0, 1) in the direction, thereby obtaining the texture coordinate of each pixel, andxy) And (4) showing.
A fourth step of calculating the correction parameters and texture coordinates of each pixel by the first step (xy) And transforming the texture coordinates of each pixel in each parallax image in the GPU by using a formula (1), wherein the image after coordinate transformation is the corrected parallax image.
The method adopts a method of rendering and superposing for multiple times to generate the synthetic image, namely a Shader program only needs to be responsible for sampling 1 parallax image, and then the synthetic image is obtained by superposing rendering results for multiple times.
The invention realizes the generation of real-time synthetic images by the following steps:
firstly, the total number of parallax images 8 and the number of parallax images at present are transmitted from an application programn(1,2, …,8) and the inclination angle arctan (-0.4550) of the grating relative to the vertical direction of the display.
A second step ofnCutting off each line on the left side of the parallax imagen-1One pixel, cutting off each line at the right side of the parallax image (8-n-1) The pixel clipping method is that each pixel coordinate is respectively arranged in the GPUXAndYin the direction of [ (n-1)/1920, 1- (8-n-1)/1920]Interpolation is performed, and since the display aspect ratio 1920: 1080, the parallax image is trimmed to (8-1) × 1080/1920 pixels in the vertical direction by using an interpolation method.
And thirdly, according to the width-height ratio 1 of the display sub-pixel: 3 and raster slope-0.4550nNumber of viewpoints of amplitude parallax imageThe matrix is calculated by the formula of:x*3+3*y*(-0.4550)]%8-nwherein (a)xy) For the texture coordinates of each pixel, if one of the R, G or B sub-pixels in the pixel belongs to the second sub-pixelnThe magnitude parallax image is aligned with the corresponding positionnAnd sampling the amplitude parallax image, and rendering the sampled texture.
And fourthly, rendering the rest 7 parallax images after sampling according to the steps, wherein the result of rendering and superposing the 8 parallax images is a composite image, the composite image is displayed on a free stereoscopic display, and a viewer can see the stereoscopic image.
Therefore, the real-time shooting and real-time free three-dimensional display system based on the GPU and the network camera can display a good three-dimensional effect, and the running frame rate of the system can reach 30 frames per second.

Claims (5)

1. A real-time shooting and real-time free stereo display system based on a GPU and a network camera is characterized by comprising a stereo network camera, a computer and a free stereo display, wherein the stereo network camera is composed of a plurality of network cameras, the system obtains a plurality of parallax images by shooting a real scene in real time through the stereo network camera, then carries out real-time parallax image correction, real-time parallax image translation cutting and real-time synthetic image generation in the computer, and finally displays the synthetic image on the free stereo display in real time, so that a viewer can see the stereo image.
2. The real-time shooting and real-time auto-stereoscopic display system based on the GPU and the network cameras as claimed in claim 1, wherein the optical axes of the network cameras in the stereoscopic network camera are parallel to each other, the optical centers are positioned on the same straight line, the intervals between the cameras are equal, the network cameras are connected with a computer through a PCI card, a DirectShow is adopted to collect videos in real time, the same network camera parameters are set, and synchronous shooting of the network cameras is realized.
3. The real-time shooting and real-time auto-stereoscopic display system based on GPU and network cameras as claimed in claim 1, wherein the real-time parallax image correction method based on GPU is adopted to make the video collected by each network camera pass through GPU correction in real time, the method utilizes re-projection to project the original imaging surface of each network camera to an ideal imaging plane, and the method is characterized in that (1)x, y) Is a point on the original imaging plane, ((ii))x', y') is the point that the point re-projection transforms to the ideal imaging plane, the relationship between these two points is determined by the formula:
Figure 201210289898X100001DEST_PATH_IMAGE001
wherein,a 1, …, a 8are all constant.
4. The system as claimed in claim 1, wherein the GPU is used for real-time parallax image translation cropping by the method of the GPUnLeft side of parallax image shot by the network camera is cut off in horizontal direction (N-n) One pixel, the right side is cut out: (n-1) pixels, whereinNIndicates the number of network cameras.
5. The system as claimed in claim 1, wherein the parallax images obtained by correcting, translating and cutting are real-time rendered and superimposed by the GPU to form a composite image, the Shader program only needs to sample 1 parallax image, and the composite image is obtained by superimposing the rendering results for a plurality of times, and the method comprisesnThe amplitude parallax image is displayed in an auto-stereoscopic displayiThe offset of the row isδ iCan be calculated by a formulaδ i=3×i×tanα+n-1 obtaining whereinαRepresenting the tilt angle of the grating of an autostereoscopic display, 1 ≦nNNThe number of views of the autostereoscopic display is equal to the number of webcams.
CN201210289898XA 2012-08-15 2012-08-15 Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras Pending CN102802003A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201210289898XA CN102802003A (en) 2012-08-15 2012-08-15 Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201210289898XA CN102802003A (en) 2012-08-15 2012-08-15 Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras

Publications (1)

Publication Number Publication Date
CN102802003A true CN102802003A (en) 2012-11-28

Family

ID=47200958

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201210289898XA Pending CN102802003A (en) 2012-08-15 2012-08-15 Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras

Country Status (1)

Country Link
CN (1) CN102802003A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103698967A (en) * 2013-12-09 2014-04-02 四川大学 Projection-type integrated imaging 3D display device
CN103745448A (en) * 2013-12-24 2014-04-23 四川大学 Rapid generating method for ultrahigh-resolution composite image in raster 3D displaying
WO2020082286A1 (en) * 2018-10-25 2020-04-30 郑卜元 Virtual reality real-time image capturing and monitoring system, and control method

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1580947A (en) * 2003-08-16 2005-02-16 劳国华 System and method for generating amnibearing stereo digital image
CN101453662A (en) * 2007-12-03 2009-06-10 华为技术有限公司 Stereo video communication terminal, system and method
CN101505433A (en) * 2009-03-13 2009-08-12 四川大学 Real acquisition real display multi-lens digital stereo system
CN101600123A (en) * 2009-07-06 2009-12-09 四川大学 Obtain the method for shifting parallel vidicon array of anaglyph
CN102209254A (en) * 2011-05-30 2011-10-05 四川大学 One-dimensional integrated imaging method and device
US20110304710A1 (en) * 2010-06-14 2011-12-15 Hal Laboratory, Inc. Storage medium having stored therein stereoscopic image display program, stereoscopic image display device, stereoscopic image display system, and stereoscopic image display method
CN102378024A (en) * 2010-08-11 2012-03-14 索尼公司 Image processor, stereoscopic display, method of detecting parallax displacement in a stereoscopic display
EP2432228A2 (en) * 2010-09-17 2012-03-21 Samsung Electronics Co., Ltd. Display apparatus and control method thereof

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1580947A (en) * 2003-08-16 2005-02-16 劳国华 System and method for generating amnibearing stereo digital image
CN101453662A (en) * 2007-12-03 2009-06-10 华为技术有限公司 Stereo video communication terminal, system and method
CN101505433A (en) * 2009-03-13 2009-08-12 四川大学 Real acquisition real display multi-lens digital stereo system
CN101600123A (en) * 2009-07-06 2009-12-09 四川大学 Obtain the method for shifting parallel vidicon array of anaglyph
US20110304710A1 (en) * 2010-06-14 2011-12-15 Hal Laboratory, Inc. Storage medium having stored therein stereoscopic image display program, stereoscopic image display device, stereoscopic image display system, and stereoscopic image display method
CN102378024A (en) * 2010-08-11 2012-03-14 索尼公司 Image processor, stereoscopic display, method of detecting parallax displacement in a stereoscopic display
EP2432228A2 (en) * 2010-09-17 2012-03-21 Samsung Electronics Co., Ltd. Display apparatus and control method thereof
CN102209254A (en) * 2011-05-30 2011-10-05 四川大学 One-dimensional integrated imaging method and device

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
徐小清,等: "利用GPU合成自由立体显示图像的方法", 《第四届立体图象技术专业委员会学术研讨会论文集》 *
王琼华,等: "立体拍摄与自由立体显示", 《2010中国平板显示学术会议论文集》 *
王琼华: "《3D显示技术与器件》", 30 April 2011, 科学出版社 *
谷俊,等: "多摄像机拍摄立体图像的方法", 《中国光学学会2010年光学大会论文集》 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103698967A (en) * 2013-12-09 2014-04-02 四川大学 Projection-type integrated imaging 3D display device
CN103698967B (en) * 2013-12-09 2016-01-13 四川大学 A kind of projection integration imaging 3D display device
CN103745448A (en) * 2013-12-24 2014-04-23 四川大学 Rapid generating method for ultrahigh-resolution composite image in raster 3D displaying
CN103745448B (en) * 2013-12-24 2016-08-17 四川大学 Grating 3D show in the rapid generation of ultrahigh resolution composograph
WO2020082286A1 (en) * 2018-10-25 2020-04-30 郑卜元 Virtual reality real-time image capturing and monitoring system, and control method

Similar Documents

Publication Publication Date Title
Anderson et al. Jump: virtual reality video
EP2603834B1 (en) Method for forming images
US7643025B2 (en) Method and apparatus for applying stereoscopic imagery to three-dimensionally defined substrates
JP4476905B2 (en) Structure of stereoscopic display image data, recording method of stereoscopic display image data, display reproduction method, recording program, and display reproduction program
JP5300258B2 (en) Method and system for acquiring, encoding, decoding and displaying a three-dimensional light field
US20080246757A1 (en) 3D Image Generation and Display System
WO2016120719A1 (en) Digital multi-dimensional image photon platform system and methods of use
JP2008257686A (en) Method and system for processing 3d scene light field
CN102572486A (en) Acquisition system and method for stereoscopic video
US20180288241A1 (en) Digital multi-dimensional image photon platform system and methods of use
JP2006515128A (en) Stereo panoramic image capturing device
KR101725024B1 (en) System for real time making of 360 degree VR video base on lookup table and Method for using the same
US8019180B2 (en) Constructing arbitrary-plane and multi-arbitrary-plane mosaic composite images from a multi-imager
KR20200064998A (en) Playback apparatus and method, and generating apparatus and method
Ainsworth et al. Acquisition of stereo panoramas for display in VR environments
CA2540538C (en) Stereoscopic imaging
KR20080034419A (en) 3d image generation and display system
CN102802003A (en) Real-time shooting and real-time free stereoscopic display system based on both GPU and network cameras
CN108450031B (en) Image capturing apparatus
CN109961395B (en) Method, device and system for generating and displaying depth image and readable medium
Gurrieri et al. Stereoscopic cameras for the real-time acquisition of panoramic 3D images and videos
Knorr et al. From 2D-to stereo-to multi-view video
Isogai et al. A panoramic video rendering system using a probability mapping method
Frick et al. Real-time preview for layered depth video in 3D-TV
CN115442580B (en) Naked eye 3D picture effect processing method for portable intelligent equipment

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C02 Deemed withdrawal of patent application after publication (patent law 2001)
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20121128