KR100208476B1 - Method and device of composing three-dimensional images in real time, using digital masking - Google Patents

Abstract

The present invention relates to a method and apparatus for real-time stereoscopic image synthesis using a digital masking technique. In the conventional lenticular or parallax barrier system, a stereoscopic image method can be divided into an analog method and a digital method. On the other hand, the input is fixed to two channels, which limits the scalability of the system and makes it difficult to process the previous image. The digital method also has the problem of limiting the flexibility of the system. In view of these problems, the left and right input images converted into digital signals are obtained by performing a logical AND operation on the mask pattern image of each of the left and right images and the respective pixels to obtain a stripe-shaped image, and the two stripe-shaped images are pixels. By performing a logical sum operation to output a stereoscopic image, it is easy to connect to a computer and apply an image processing technique, to easily store and transmit image data, and to change patterns through a software data processing method. It is easy to flexibly expand the number of camera channels, making it easy to apply to experiments.

Description

Real time stereoscopic image synthesis method and device using digital masking

1 is a principle diagram of binocular disparity.

2 is a view of the lenticular lens plate.

3 is a diagram of a parallax barrier type plate.

4 is a three-dimensional image display principle of the lenticular method.

5 is a three-dimensional image display principle of the parallax barrier method.

Figure 6 is a three-dimensional image synthesis principle using a dedicated hardware board.

7 is a principle diagram of digital multiplexing.

Figure 8 (a), (b) is an exemplary mask pattern image of the left and right image according to the present invention.

Figure 9 (a), (b) is an explanatory diagram of the logical product operation process according to the present invention.

10 is an explanatory view of the OR operation process according to the present invention.

11 is a real-time stereoscopic image synthesis device block diagram using the digital masking technique of the present invention.

12 is a block diagram of a real-time stereoscopic image synthesizing apparatus using a digital masking technique using four cameras of the present invention.

* Explanation of symbols for main parts of the drawings

11a-11d: Camera 12a-12d: Analog / Digital Converter

13a-13d: 3D image storage memory 14a-14d: Mask storage memory

15a-15d: Calculator 16: Alternating Video Storage Memory

17: output controller 18: video medium

The present invention relates to a method for synthesizing three-dimensional images, which is part of three-dimensional image display technology. In particular, digital masking allows a human to feel a natural and realistic sense in a real world through a flat medium such as a TV. A method and apparatus for real-time stereoscopic image synthesis using a masking technique are provided.

There are two physiological and psychological factors in humans, and in three-dimensional image display technology, the object is created by using the binocular parallax, which is the biggest factor that generates three-dimensional effect at a short distance (usually within 10m). Represents a three-dimensional appearance. FIG. 1 is a principle diagram of binocular disparity. As can be seen from FIG. 1, binocular disparity refers to a visual angle difference that occurs when the same object is seen in the right eye and the left eye due to the distance between the two eyes (about 65 mm). The parallax between both eyes makes it possible to feel the three-dimensional effect of the same object.

The stereoscopic image display method of displaying stereoscopic effect using the binocular vision difference can be easily combined with general TV technology so that people can easily use it. The three-dimensional image display method can be largely divided into a wearing method and a non-wearing method.

The spectacle wearing method is an anaglyph method using blue and red sunglasses on both eyes, a polarization type using glasses with different polarizations, and a polarization shutter synchronized with this period by periodically repeating polarization. There is a time-sharing method of wearing glasses installed.

In particular, the time-division method has an advantage that it can be easily obtained three-dimensional effect compared to other methods, while being practical from the early days, the inconvenience of having to wear other glasses other than vision correction glasses and other objects other than the image while wearing glasses There is a problem in that it is difficult to observe and also a problem that multi-view image display becomes impossible.

Therefore, a study on the non-wearing method for viewing a stereoscopic image without using glasses has been continued.

The most practical method proposed so far is the lenticular method in which a lenticular lens plate in which cylindrical lenses are arranged vertically is installed in front of the TV, and the light that is shined on the screen using a specially designed plate is used. The parallax barier system which obtains a three-dimensional effect by adjustment is mentioned.

FIG. 2 is a shape of the lenticular lens plate. As shown therein, a cylindrical lens lens having a predetermined pitch is vertically arranged to form a lenticular lens plate, and FIG. 3 is a parallax barrier type plate. As shown in the figure, the opaque rods on the glass plate are arranged vertically at regular intervals as shown.

Each of the three-dimensional image display techniques described above requires an image structure having a unique shape. For example, for the anagraph method of wearing glasses, the left image is red (or blue) and the right image is blue (or red). In the time division method, the left and right images are repeatedly displayed in synchronization with the TV interlaced signal. In the lenticular and parallax barrier methods, the left and right images are alternately arranged pixel by pixel. Should be.

4 is a 3D image display principle of a lenticular method. As shown in FIG. 4, left and right images of the left and right cameras are input to an image processor, and an LCD panel (from the image processor) is input. When the stereoscopic images are arranged alternately in the order of the left and right images on the LCD panel, the left and right image pixels intersect on the focal plane of the lenticular plate according to the directivity characteristic of the lenticular sheet. Therefore, only the left image pixel and the right image pixel are projected on the left eye of the observer, so that the two images can be seen separately, so that a stereoscopic feeling can be felt.

5 is a diagram illustrating a three-dimensional image display of a parallax barrier method. As shown therein, an image viewed from the left eye of the observer and an image viewed from the right eye are combined and output in a vertical direction on the LC plate. Behind the LCD panel is an opaque rod (called a video splitter), a bar-shaped partition placed perpendicular to the glass plate. This video splitter is the most essential element of the parallax barrier method, and the light of the backlight behind it is affected by the video splitter. That is, while the viewer's left eye is seen as a pixel of the left image from which light is projected, the pixel of the right image is not visible because the light is not reflected due to the vertical partition of the image splitter. In the same way, only the pixels of the right image are visible to the right eye, and the pixels of the left image are not visible. As a result, the viewer's left and right eyes will see different images, so they will feel three-dimensional. Here, the distance between the viewer and the LC panel to which the image is output may be adjusted using the thickness of the glass plate of the image splitter.

Most of the existing techniques for image structure processing for the lenticular or parallax barrier methods mentioned above can be divided into analog and digital methods.

The analog method is based on NTSC's interlacing method, which focuses on inputting images in field units instead of frame units, and rotates both cameras and TVs by 90 °. A natural left / right composite image is realized by inputting an image into an even field (or odd field) and a right image into an odd field (or even field), respectively.

This method has been used in three-dimensional video telephone systems developed by the Japanese entity (NTT). However, this method has the advantage of easy image synthesis, while the input is fixed to two channels, which limits the scalability of the system, and has the inherent disadvantages of analog methods that have difficulty in image processing before synthesis.

On the other hand, NHK has developed a lenticular digital 3D TV technology, which uses a dedicated hardware board for synthesizing the input image and synthesizes it through the digitally converted input image multiplexing method.

FIG. 6 is a stereoscopic composite principle using a dedicated hardware board. As shown therein, analog video signals input by the cameras A to D are digitalized by the converter 1, respectively, to determine the RGB signals. (R (A) -R (D), G (A) -G (D), B (A) -B (D)). The separated Algivy signal is made into a stripe-shaped image alternately pixel-by-pixel via the digital multiplexer 2 and displayed on the display unit 3.

That is, FIG. 7 is a principle diagram of digital multiplexing. As shown therein, an RGB component of the first camera A forms one pixel and is positioned at the (0,0) th position. The Algivy components of the two cameras B become the (0,1) th pixel. The images input in this form can produce an alternating image. In general, since the hardware processing speed of multiplexing is very fast, it has the advantage of making stereoscopic images in real time (usually 30 frames per second), but since the input of the multiplexer is fixed, it is difficult to change the number of input images. There are disadvantages in that various attempts to produce optimal stereoscopic images in a variable experimental environment are restricted.

Accordingly, an object of the present invention is to provide a method and apparatus for synthesizing a stereoscopic image using a digital masking technique capable of synthesizing a stereoscopic image in real time by solving the above disadvantages.

In addition, another object of the present invention is to provide the convenience of connecting to a computer and applying image processing techniques, the convenience of storing and transmitting image data, and the digital multiplexing method through dedicated hardware. In order to overcome the problems caused by structural constraints, it is to provide a user-oriented stereoscopic image production environment that can be easily applied to various experiments through software data processing.

The object of the present invention is to perform an AND operation on a mask pattern image of left and right images and to perform an AND operation, respectively, to obtain an image of a stripe shape, and to perform OR operation on the two stripe images. This is achieved by outputting a stereoscopic image, which will be described in detail with reference to the accompanying drawings.

(A) and (b) of FIG. 8 are exemplary mask pattern images of the left and right images according to the present invention. The mask pattern images of the left image are black and white portions as shown in (a) of FIG. This alternately repeated, as shown in (b) of FIG. 8, the mask pattern image of the right image alternately repeats the white and black portions as opposed to the mask pattern image of the left image. In the mask pattern image, the black part means 0 in the logic circuit, which is off when viewed by a switch, and the white part means 1 in the logic circuit, which is on, when viewed by a switch.

Such two mask pattern images are used when two eyes, that is, two input images.

(A) and (b) of FIG. 9 are explanatory diagrams of a logical AND operation process according to the present invention. As shown in FIG. A stripe-shaped image is obtained by performing a logical AND operation on the mask pattern image of the right image. First, it is assumed that the pixel value of the left image of the input image is A and the pixel value of the right image is B. In the case of the left image, the (0,0) th pixel is input, and a logical AND operation is performed with the (0,0) th pixel (meaning logical 0 as the black portion) of the mask pattern image for the left image. At this time, since A ∩ O = O in the logic circuit, the black part of the mask pattern image of the left image is positioned at the (0,0) th pixel of the output image. The next (0,1) pixel is logically 1 since the (0,1) pixel of the mask pattern image of the left image is a white portion. Therefore, since A ∩ 1 = A, the (0,1) th pixel of the input image is placed at the (0,1) th pixel position of the output image.

In the case of the right image, a logical output operation is performed in the same manner as in the case of the left image described above, so that an output image having a stripe shape for the left and right images can be obtained.

FIG. 10 is an explanatory diagram of a logical OR operation process according to the present invention. As shown in FIG. 9, logical outputs of stripe-shaped output images of left and right images obtained as shown in (a) and (b) of FIG. By performing a (logical OR) operation, the final synthesized stereoscopic image can be obtained. That is, since the (0,0) th pixel of the sprite-type left image is a black portion, it is logical 0, and the (0,0) th pixel of the right image is B because it is an input image. Therefore, the result of the logical sum of the (0,0) th pixels of the left and right images is O ∪ B = B, and the (0,0) th pixel value of the final output image is B. In the case of the (0,1) -th pixel of the left and right images, A ∪ O = A, and the (0,1) -th pixel value of the final output image is A when the logical sum is performed in the same manner as described above. As a result, a stereoscopic image of a form in which pixels B, A, B, and A of the left and right images are repeated is finally made.

FIG. 11 is a block diagram of an apparatus for synthesizing a real-time stereoscopic image using the digital masking method of the present invention.As shown in FIG. 11, an analog / digital signal for converting an output image signal from left and right cameras 11a and 11b into a digital signal is shown. An input image storage memory (13a) and (13b) for receiving and storing the output image signals of the digital (A / D) converters 12a and 12b and the analog / digital converters 12a and 12b; Mask storage memories 14a and 14b for storing mask pattern images of left and right images, and input image signals stored and output in the input image storage memories 13a and 13b. Arithmetic operations of the mask pattern image signals of Figs. 14b and 14b and then outputting the image signals according to the input image signals alternately, and the arithmetic units 15a and 15b. Alternate image storage memory 16 for storing an alternating image signal output alternately from That receives the alternating images stored in the image storage alternate memory 16 constituted by the output controller 17 to display the video medium 18 will be described its operation process.

Left and right video signals captured and output by the left and right cameras 11a and 11b are converted into digital video signals by the analog / digital converters 12a and 12b to be input image storage memories 13a and 13b. ) And the mask pattern image of the left and right images stored and output in the mask storage memories 14a and 14b are inputted from the calculators 15a and 15b and logically multiplied by each pixel. logical AND) results in a stripe-shaped image as described in (a) and (b) of FIG. 9, and an image according to the output of the left and right input images, that is, a black portion, is The white portion of the image is alternately outputted by the calculators 15a and 15b to form an alternating image of the left and right input images. In this way, the alternating images of the left and right input images output from the calculators 15a and 15b are stored in the alternating image storage memory 16, and the output controller 17 outputs the image output from the alternating image storage memory 16. The display image is a stereoscopic image as it is received from the display medium and displayed on the image medium 18.

FIG. 12 is a block diagram of an apparatus for synthesizing a real-time stereoscopic image using a digital masking technique using four cameras of the present invention. As shown in FIG. 12, the number of mask patterns is increased to four, which is the same as the technical configuration of FIG. To configure. That is, the analog / digital converters 12c and 12d that receive the output video signals of the cameras 11c and 11d and convert them into digital signals, and the output images of the analog / digital converters 12c and 12d. Input image storage memories 13c and 13d for receiving and storing signals, mask storage memories 14c and 14d for storing mask pattern images, and input image storage memories 13c and 13d. An operator 15c for performing an AND operation on the mask pattern image signal of the mask storage memories 14c and 14d and then outputting the image signal according to the input image signal alternately. 15d is added to the technical configuration of FIG. 11, and the operation of FIG. 12 is performed in the same manner as the operation described with reference to FIG.

As described in detail above, the present invention can be easily applied to a general personal computer, a dedicated hardware system or a general-purpose hardware system because the processing method and the required specifications are simple, and various forms of stereoscopic image implementation experiments according to the user's software implementation. This is possible, and when applied to a general-purpose image processing hardware system that has a high computational speed, there is an effect of synthesizing left and right images by real-time processing.

Claims (4)

Obtaining a stripe-shaped image by performing a logical AND operation on the left and right input image converted into a digital signal, and a mask pattern image of each of the left and right images, and for each pixel; A real-time stereoscopic image synthesis method using a digital masking technique comprising the step of performing a logical OR operation for each pixel to output a stereoscopic image. First and second analog / digital conversion means for receiving output video signals from left and right cameras and converting them into digital signals, and first and second output signals for first and second analog / digital conversion means. A second input image storage means, first and second mask storage means for storing a mask pattern image of left and right images, and an input image signal output from the first and second input image storage means. First and second calculation means for performing an AND operation on the mask pattern video signal of the second mask storage means and alternately outputting the video signal according to the input video signal, and from the first and second calculation means. A digital image comprising alternating image storing means for storing an alternating image signal outputted alternately and an output control means for receiving an alternating image stored in the alternating image storing means and displaying the alternating image on the image medium; Real-time stereoscopic image synthesis device using masking technique. 3. The real-time stereoscopic image of claim 2, wherein the mask pattern images of the left and right images stored in the first and second mask storage units are arranged by arranging logic 0 parts and logic 1 parts opposite to each other. 4. Synthesis device. The apparatus of claim 2, wherein the number of mask patterns is increased to four using four cameras.