KR100424401B1 - 3D Stereoscopic Multiview video system include Searching function - Google Patents

Abstract

The present invention relates to a multi-view video communication system, and more specifically, to display more images than the original image through the process of preprocessing, compressing, multiplexing, demultiplexing, reconstructing, and synthesizing the original image inputted from multiple image input devices. The present invention relates to a multi-view video communication system capable of outputting from a device, viewing 3D stereoscopic images at various angles and various distances, and including 3D stereoscopic images.

The present invention is largely divided into a transmitter and a receiver. The transmitter includes an image acquisition unit for obtaining a 2D image from multiple cameras, a multi-view image preprocessing unit 100 for correcting a parameter due to a characteristic difference of the camera, and each camera. Multi-view image compression unit 200 to remove the temporal and spatial redundancy between the images output from the image and metadata having information that can be retrieved directly from the two-dimensional image obtained by the image acquisition unit Multi-view image retrieval system 300 for extracting the multi-view image and the multi-view image multiplexer 400 for multiplexing each image stream and the metadata in one stream,

The receiving end restores the metadata and each independent video stream from the multiplexed video stream and reproduces the multiplexed stream excluding metadata for storage, and the multi-view video demultiplexer (500). A multimedia storage and retrieval system (Multimedia Storage Search system) for temporarily storing the 3DSS stream that does not include the metadata and metadata generated in the 500 and search using the metadata, the multi-view image demultiplexer (500) and A multi-view image restoring unit 600 which decompresses and restores each compressed image of the 3DSS stream retrieved from the multimedia storage and retrieval system, and a multi-view intermediate image synthesizing unit generating an intermediate image between the restored multi-view images. 700 and the 3D display controller 800 for outputting the 2D image formed in the electrical stage to the 3D display apparatus. Include.

Therefore, the present invention has a merit that a large number of people can watch at various angles or views at the same time, unlike a conventional technology in which only a limited number of people can see a 3D image, so that more viewpoints can be obtained from a camera. . In addition to the three-dimensional multi-view video system, each means constituting the present invention can be independently applied to products such as digital broadcasting, medical field, game / entertainment, etc. In addition to unconditional reproduction, the present invention can be reproduced at various angles and viewpoints in three dimensions depending on the user's preferences and selections.

Description

Multi-view video communication system for 3D stereoscopic image with search function {3D Stereoscopic Multiview video system include Searching function}

The present invention relates to a multi-view video communication system, and more specifically, to display more images than the original image through the process of preprocessing, compressing, multiplexing, demultiplexing, reconstructing, and synthesizing the original image inputted from multiple image input devices. The present invention relates to a multi-view video communication system capable of outputting from a device, viewing 3D stereoscopic images at various angles and various distances, and including 3D stereoscopic images.

1 shows a series of methods for implementing a 3D image from a multiview. That is, as a method of implementing a three-dimensional image, two-dimensional image data is obtained from several cameras 1, each of which acquires two-dimensional image data. The 3D image is realized through the 3D display apparatus 2, that is, a lenticular method, by restoring the remote apparatus. The existing multiview video system is a stereoscopic video system that satisfies the multiview profile included in the MPEG-2 standard, as shown in FIG. The system configuration consists of a MPEG-2 MP @ ML base layer left eye motion compensation DCT encoder block (3), a right eye motion compensation DCT encoder block (4), a disparity estimator (DE) (5), and a floating compensator. Operation consisting of (Disparity Compensator, DC) (6) and the like for a left eye composed of a base layer of MPEG-2 MP @ ML as well as a stereo encoder module comprising a floating compensation DCT encoder block 50 and a system multiplexer 7 Operation consists of a compensation DCT decoder block 8, a motion compensation DCT decoder block 9 for the right eye, a floating compensator 10 and the like, and a stereoscopic decoder module including a floating compensation DCT decoder block 60 and a system demultiplexer 11. It is done.

The operation of the system is as follows. In operation of the system, the left image is selected as the base image, the left eye motion compensation DCT encoder block 3 is encoded by the MPEG-2 standard algorithm, and the encoded stream is transmitted to the system multiplexer 7. On the other hand, the block generates a reconstructed image for encoding the next image and the right image, and transmits the reconstructed image to the DE (5) and the DC block (6). The right original image is transmitted to the motion compensation DCT encoder block (4) and the DE block (5) for the right eye. In the DE block (5), the reconstruction image of the left image and the original image of the right image are compared and evaluated for disparity vector, DV) is obtained and transferred to the right eye compensation DCT encoder block 4 and the DC block 6. The DV delivered to the DC block 6 generates a compensation image for the next right image with reference to the left restored image, and the DV delivered to the right eye compensation DCT encoder block 4 is encoded in the same manner as the MV of the MPEG-2 standard. do. In addition, the right eye motion compensation DCT encoder block 4 uses an MPEG-2 encoding algorithm using a DCT (discrete cosine transform, hereinafter DCT) for the difference between the compensation image generated from the DC block 6 and the right original image. Is encoded. The encoded stream for the right picture is separately transmitted to the system multiplexer 7 in the same manner as the left picture. The system multiplexer 7 follows the standard of the MPEG-2 system. The multiplexed streams from the system multiplexer 7 are passed to the system demultiplexer 11 and demultiplexed to restore two video encoding streams, respectively. Is passed to (9). The left eye motion compensated DCT decoder block 8 that obtains the encoded stream of the left image reconstructs the image by an algorithm satisfying the MPEG-2 standard, and transmits the reconstructed image to the output and the DC block 10. In addition, the right-eye motion compensation DCT decoder block 9 that receives the encoded stream for the right picture decodes the stream and performs a difference picture between the compensation picture and the original picture with reference to the left picture generated by the DV and DC block 10. Acquire. The acquired DV is transferred to the DC block to generate a compensation image by referring to the left image previously restored, and the compensated image generated from the DC block is transferred to the right eye motion compensation DCT decoder block 9 and the reconstructed difference image. It is synthesized to restore the right image.

From the application's point of view, the operation of the system is obtained from two horizontally parallel cameras at a distance similar to the left and right eyes of a person. It is delivered to the decoder through a remote. After decoding the demultiplexer of the decoder, each decoded image coded stream is reconstructed into an image through the decoder. Each reconstructed image is output to the user through a three-dimensional display device. The system can implement a three-dimensional image only at one viewpoint and within a limited distance.

As described above, the conventional method is a binocular system that constructs a 3D image only at one point when the 3D image is viewed from the user's point of view. That is, the image is shown in three dimensions only when there is a viewer at about 1 to 2 m in front of the display device. Therefore, the existing technology has an extremely limited visual and visual field of vision so that only one person or a very small number of people can see the 3D image. As a result, the conventional method has a disadvantage in that it cannot be applied to an application field targeting a plurality of users such as a 3D information terminal and a 3D TV. In addition, it is impossible to reproduce a multiview image selectively based on the content of a multiview image to be stored or transmitted by the user.

As a solution to the problems of the existing technology as described above, the present inventors propose a three-dimensional multi-view video system that can view objects and scenes from multiple angles simultaneously on a lenticular three-dimensional display. To this end, in the present invention, a large amount of data can be compressed to be applied to a channel environment that is possible at the present point of view, and it is possible to generate an image of more viewpoints than the acquired point of view, thereby realizing three-dimensional images at various viewpoints and distances. It is an object of the present invention to be able to.

In addition, a device having an image retrieval function is added to a multi-view video communication system to provide a multi-view video communication system that is reproduced according to a user's preference and selection as well as unconditional reproduction.

1 is a schematic explanatory diagram of a three-dimensional multiview imaging system;

2 is a block diagram of a conventional stereoscopic image system.

3 is a block diagram of a three-dimensional multi-view video communication system of the present invention.

4 is a detailed embodiment of the present invention (multi-view image preprocessing unit and multi-view image compression unit)

5 is a specific embodiment of the present invention (multi-view multiplexer)

6 is a specific embodiment of the present invention (multi-view image demultiplexer)

Figure 7 is a specific embodiment of the present invention (multi-view image restorer and multi-view intermediate image generation unit)

<Description of the symbols for the main parts of the drawings>

1: camera 2: 3D display device

3: left eye motion compensation DCT encoder block

4: motion compensation DCT encoder block for right eye

5: floating estimator 6,10: floating compensator

7: system multiplexer

8: motion compensation DCT decoder block for left eye

9: motion compensation DCT decoder block for right eye 11: system demultiplexer

50: motion and floating compensation DCT encoder block

60: motion and floating compensation DCT decoder block

100: multiview image preprocessing unit 101: imbalance reduction filter

102: noise reduction filter 200: multi-view image compression unit

201,204,207: DCT-based encoder 202,205: Float estimator

203,206,602: floating compensator

300: Multiview Image Search System

310: multiview image feature extraction device

320: again low image feature technology device

400: multiview image multiplexer 410: metadata processor

411 metadata packetizer 412 stream header analyzer

413 Video Packetizer 414 Buffer Controller

415: System Header Maker 416: DSS Packetizer

500: multiview image demultiplexer 511: packet synchronizer

512: error detector 513: system clock extractor

514: System Clock Restorer 515: Program Information Decoder

516 PID processor 517 System header processor

518: video packet header processor

519: Timing recovery and storage buffer

600: multi-view image restoring unit 601: DCT-based decoder

700: multiview intermediate image synthesis unit 701: adaptive point extractor

702: Occlusion area maker

703: the final intermediate image generator

800: 3D display control unit

Figure 3 is a block diagram of a three-dimensional multi-view video communication system of the present invention, Figure 4 is a specific embodiment of the present invention, a multi-view image pre-processing unit and a multi-view image compression unit, Figure 5 is a view of the present invention again 6 shows a multi-view image demultiplexing unit of the present invention, and FIG. 7 shows a multi-view image restoring unit and a multi-view intermediate image generating unit of the present invention.

Referring to FIG. 3, the system is largely divided into a transmitter and a receiver. The transmitter includes an image acquisition unit for acquiring a 2D image from multiple cameras, and a multi-view image preprocessing for correcting parameters due to camera characteristic differences. Meta data from the multi-view image compression unit 200, which removes temporal and spatial redundancy between the images output from each camera, the image 100, and the 2D image obtained by the image acquisition unit. Multi-view image retrieval system 300 for extracting the multi-view image and the multi-view image multiplexer 400 for multiplexing each image stream and the metadata compressed by the multi-view image compression unit 200 into a single stream And

The multi-view image demultiplexer 500 restores the metadata and each independent video stream from the multiplexed stream (3DSS stream) and reproduces the multiplexed stream except metadata for storage. A multimedia storage search system for temporarily storing 3DSS streams that do not include metadata and metadata generated by the image demultiplexer 500 and performing a search using metadata, and the multi-view The image demultiplexer 500 and the multiview image restoration unit 600 which decompresses and restores each compressed image of the 3DSS stream retrieved from the multimedia storage and retrieval system, and generate an intermediate image between the restored multiview images. The multi-view intermediate image synthesis unit 700, and outputs the two-dimensional image formed in the electrical step to the three-dimensional display device 3D display control unit 800 is configured to include.

Referring to FIG. 4, the acquired multi-view image (shown at M point of view) is used to recall the luminance and chromaticity of the image based on one reference camera to correct an error due to a characteristic difference between the cameras. The multi-viewpoint image preprocessing unit 100 is in charge. Preferably, the multi-view image preprocessing unit 100 includes an imbalance reduction filter 101 and a noise reduction filter 102, and the imbalance reduction filter 101 includes a block using a least square error (LSE) and an Affine Transform coefficient. Based on the balanced algorithm, luminance and chromaticity are corrected by matching the average value and the variance. In addition, the image signal passing through the unbalance reduction filter 101 is transmitted to the multi-viewpoint image compression unit 200 through the noise reduction filter 102 for removing each Gaussian noise.

The multi-view image compression unit 200 is a temporal and spatial redundancy between adjacent images for each camera for the multi-view images corrected by the multi-view image preprocessor 100 and the time between the images output from each camera In addition, image compression is performed to remove spatial redundancy. To this end, the multi-viewpoint image compression unit 200 preferably includes an overlapped block disparity estimation / overlapped block disparity compensation and an existing MPEG-2 algorithm.

The video compression means 200 according to the preferred embodiment of the present invention shown in FIG. 4 includes a DCT-based encoder 201 including an overlapped block motion estimation / overlapped block mobility (not shown). ), A floating estimator 202, and a floating compensator 203. The video compression unit 200 provides the temporal and spatial redundancy according to the homogeneity between the closest camera images at the same time with respect to the video frames continuously input from each camera, by the floating estimator 202 and the floating compensator 203. Are encoded by using the encoding that satisfies the MPEG-2 standard. In addition, temporal and spatial redundancy according to homogeneity between consecutive frames input from each camera is encoded using a motion estimator / motion compensator, and the remaining DCT coefficients are compressed through encoding that satisfies the MPEG-2 standard.

As an example according to the embodiment shown in the figure, the corrected image of the camera 2 (hereinafter, abbreviated as Ca_2) is independently encoded by the DCT based encoder 26 including a motion estimator / motion compensator to encode the Ca_2 encoded stream. The reconstructed image is provided as a reference image to the floating estimator 202 and 205 / floating compensator 203 and 206 block. Referring to the Ca_1 correction image and the Ca_2 correction image, the image is transferred to the floating estimator 202 and the DCT-based encoder 201, and the floating estimator 202 refers to the reconstructed image of the Ca_2 correction image by referring to a disparity vector, Or DV) and deliver the DV to the floating compensator 203 and the DCT based encoder 201. The floating compensator 203 refers to the reconstructed image of the Ca_2 corrected image, generates a compensation image using DV, and transfers the image information to the DCT-based encoder 201. This process is applied equally between Ca_2 and Ca_3 correction image. Finally, DCT-based encoders 201 and 207 are used to compare DV and original image (Ca_1 or Ca_3 correction image) transmitted from floating estimator and image compensated in floating compensation. The difference image is encoded and output as a Ca_1 encoded stream and a Ca_3 encoded stream, respectively.

The metadata of the transmitter is a multi-view image retrieval system 300 composed of a multi-view image feature extraction device 310 and a multi-view image feature description device 320. Is extracted by.

As shown in FIG. 5, the multi-view image multiplexer 400 is a system in which each of the compressed video streams and the metadata are multiplexed into a single stream, and identifiers, time information, and program information for each independent video stream are provided. Etc. are added and the detailed configuration as the specific embodiment is as shown in FIG. The multi-view image multiplexer 400 shown in the figure is preferably a stream header analyzer 412, a video buffer, a video packetizer 413, a buffer controller 414, a system header maker 415, a DSS packetizer ( 416 and metadata packetizer 411.

Looking at the working relationship, the video stream independently encoded by the multi-view video compression unit 200 is separated into AU units in the stream header analyzer 412 and sequentially delivered to the video buffer. Next, the video stream divided into AU units is packetized by adding time information and stream information about each AU to a header in the video packetizer 413, and the packetized video stream is transferred to the video buffer. Next, the video stream delivered to the video buffer is transferred to the system header maker 415 in a round robin manner under the control of the buffer controller 414 to generate a system header including an identifier and other information for each video stream. The DSS packetizer 416 generates a PES packet (PES_1, PES_2, etc.) into one DSS packet and sequences it.

Meanwhile, the metadata extracted by the multiview image retrieval system 300 is inserted into the 3DSS stream by the metadata processor 410 in the multiview system multiplexer. Referring to the insertion process, the metadata are sequentially Stored in a metadata buffer and packetized to size by the metadata packetizer 411 and inserted into a payload of packets of a 3DSS stream to be inserted and then passed to the system header manufacturer 415. A DSS packetizer 416 is used to output the 3DSS stream. Here, the identification of the data is distinguished using a unique identifier (Packet_ID), and the data type and identification information of the metadata are inserted into the PSI (Program Specific Information) information existing in the 3DSS stream.

The receiving unit processes the 3DSS stream including metadata in three ways. Firstly, multi-view video is played by processing 3DSS streams in real time. Secondly, 3DSS streams are stored. Finally, playback and storage are simultaneously performed. In the first case, the metadata is extracted from the 3DSS stream and the search system does not need to perform a search function since it is already being played as in the broadcast case. In the second case, the search function plays an important role. When storing, it extracts the metadata including the search function separately and regenerates the remaining 3DSS streams except the metadata to recognize the associations with each other in the storage medium. If you store them independently, the search system of the receiver can process this metadata and play back only the multi-view images you want. Lastly, playback and storage are executed simultaneously. In this case, the 3DSS streams except metadata are transferred to the video buffer at the same time for playback. Referring to Figure 6 will be described in more detail the process of the three cases.

As shown in FIG. 6, the multi-view video demultiplexer 500 restores the metadata and the respective independent video streams from the multiplexed video stream including the metadata, and stores the multiplexed video streams except the metadata for storage. As a system for regenerating the data, a packet synchronizer 511, an error detector 512, a packet buffer, a system header processor 517, a PID processor 516, a system clock extractor 513, a system clock restorer 514, A program information decoder 515, a video packet header processor 518, a video buffer (V_buffer), a timing recovery and storage buffer 519, and a metadata buffer for storing metadata.

First, referring to the actual reproduction process, the 3DSS stream including metadata is sequentially input to the packet synchronizer 511, and the packet synchronizer 511 retrieves the synchronization bytes of each packet and stores the synchronized packets. Send to error detector 512. The error detector 512 performs an error check and sends a DSS packet to the system clock extractor 513 and a packet buffer (FIFO). The system clock extractor 513 receives the PCR_PID from the program information decoder 515, extracts the PCR information from the corresponding DSS packet header, and restores the correct system clock using the system clock recoverer 514. The 3DSS stream packets stored in the packet buffer (FIFO) are sequentially transmitted to the PID processor 516, and the PID processor 516 identifies the Packet_ID to determine whether the 3DSS stream packets are valid. In this case, the validity judgment is valid only when Packet_ID is Packet_ID for identifying program recognition information and Packet_ID for identifying multi-view video streams, and all others are invalid.

If it is determined that the packet is valid, the packet is transferred to the system header processor 517 to extract and analyze the system header, and only the payload is transmitted to the video packet header processor 518. If it is determined not valid, the packet is ignored. The data transmitted to the video packet header processor 518 analyzes the video packet header and passes it to the video buffer (V_buffer) as it is. do.

Next, referring to the process of storing the 3DSS stream, the validity determination is made in the PID processor 516, which is valid only when the Packet_ID identifies metadata, and all of the remaining invalid data are determined to be invalid. The packets are transferred to a timing recovery and storage buffer 519 to reconstruct and sequentially store time information of the 3DSS stream. When the storage buffer exceeds a certain amount, it is stored in the storage medium. On the other hand, valid packets are delivered to the system header processor 517 to extract and analyze a system header, extract metadata from the payload portion, and extract the metadata into a metadata buffer. Save it. The metadata stored in the metadata buffer is then stored in the storage medium in a certain amount.

Next, referring to a process of simultaneously storing and reproducing, in the storing process, Packet_ID for identifying program recognition information, Packet_ID for multi-view video stream, Packet_ID for metadata, etc. in the PID processor 516 determines whether validity is determined. If it is identified that the validity, and 3DSS stream packet including a multi-view video stream is delivered to the system header processor 517 and the timing recovery and storage buffer 519 at the same time, the storage and playback is performed at the same time.

As shown in FIG. 7, each of the compressed image streams is reconstructed through the multi-viewpoint image restoration unit 600. The multiview image reconstructor 600 preferably includes a DCT-based decoder 601 and a floating compensator 602. 7 illustrates a detailed configuration example of the multi-viewpoint image restoration unit 600 of the present invention. The compressed video stream is input to a DCT-based decoder 601 including an operation compensator, and the DCT-based decoder 601 restores the DV and the difference image, delivers the DV to the floating compensator 602, and then generates a compensation image. The image is reconstructed by combining the compensation image with the difference image.

The multi-view intermediate image synthesizing unit 700 performs a process of generating an intermediate image between the multi-view images restored by the multi-view image restoring unit 600. According to this, it is possible to obtain N-view images by generating new images (N-M) one by one between the obtained M-view images. For example, if the image reconstructed by the multi-view image restoring unit 600 is composed of nine viewpoints, the applied images Ov1, Ov3, Ov4, Adaptive point extraction for Ov6, Ov7, and Ov9) is performed by the adaptive point extractor 701, and then the occluded area manufacturer 702 extracts and generates an occluded area based on the adaptive point extraction data. Next, 17 points may be acquired by synthesizing the data corresponding to the adaptive point and the interpolated occlusion area and generating a new intermediate image in the final intermediate image generator 703.

The multi-view intermediate image synthesizer 700 may be used independently to generate an image that does not exist in an image editor or an image searcher or to restore a lost image.

The 2D image formed through the electrical steps is output by the 3D display controller 800 to the 3D display device.

In addition, the system of the present invention as described above can be applied to other multimedia related fields such as HDTV broadcasting equipment, set-top box, which is a two-dimensional video encoder by replacing any one means of each component. For example, the multiview image compression unit 200 may be a general two-dimensional video encoder, the multiview image restoration unit 600 may be replaced with a two-dimensional video decoder, and the multiview image intermediate image synthesis unit 700 may be removed. . Therefore, it will be apparent to those skilled in the art that a part of the constituent means can be replaced, deleted, and changed by the configuration of the present invention, and that the invention of equivalents having these modification means is not outside the scope of the present invention.

According to the present invention, unlike the existing technology that only a limited number of people can see the three-dimensional image, it is possible to secure more points of view than can be obtained from the camera has the advantage that many people can watch at different angles or at various points of time at the same time have. In addition to the three-dimensional multi-view video system, each means constituting the present invention can be independently applied to products such as digital broadcasting, medical field, game / entertainment, etc. In addition to unconditional reproduction, the present invention can be reproduced at various angles and viewpoints in three dimensions depending on the user's preferences and selections.

Claims (3)

In an image system apparatus using a multiview for realizing a 3D stereoscopic image, a transmitter acquires a 2D image from a plurality of cameras, and a multiview image preprocessor corrects a parameter due to a camera's characteristic difference. And a multi-view image compression unit for removing temporal and spatial redundancy between images output from each camera, a multi-view image retrieval system for extracting metadata from the 2D image obtained by the image acquisition unit, and the multi-view A multiview image multiplexer which multiplexes each image stream and the metadata compressed by the image compressor into one stream; The receiving end restores the metadata and each independent video stream from the multiplexed video stream and reproduces the multiplexed stream excluding the metadata for storage, and the metadata generated by the multi-view video demultiplexer. A multimedia storage and retrieval system for temporarily storing a multiplexed video stream that does not include data and metadata and retrieving it using metadata, and the multiplexed video stream retrieved by the multi-view image demultiplexer and the multimedia storage and retrieval system. A multi-view image restoring unit for decompressing and restoring each compressed image of the multi-view image, a multi-view intermediate image synthesizing unit for generating an intermediate image between the restored multi-view images, and a two-dimensional image formed in the previous step. 3D display control unit for outputting to The multi-image communication system for a three-dimensional image, including a search function The multi-view image multiplexer of claim 1, wherein the multi-view image multiplexer comprises a stream header analyzer, a video buffer, a video packetizer, a buffer controller, a system header maker, a DSS packetizer, and a metadata packetizer. In the stream header analyzer, the video stream is divided into AU units and sequentially delivered to the video buffer. The video stream divided into AU units is packetized by adding time information and stream information about each AU to the header in the video packetizer. The packetized video stream is delivered to the video buffer, and the video stream delivered to the video buffer is delivered to the system header maker under the control of the buffer controller to include the identifier and other information for each video stream. Create a header and resend the PES packet from the DSS packetizer into one DSS packet The metadata is sequentially stored in the metadata buffer and inserted into the payload of the packet of the multiplexed video stream by the metadata packetizer and inserted into the payload of the multiplexed video stream. Type of data for metadata in PSI information which is transmitted to a system header manufacturer and output along with the multiplexed video stream through a DSS packetizer and is distinguished using a unique identifier and exists in the multiplexed video stream. And multi-view video communication system for three-dimensional stereoscopic images including a search function characterized in that the identification information is inserted and distinguishable. 2. The multiview image demultiplexer 500 receives a multiplexed video stream including metadata and retrieves a synchronization byte of each packet, and an error detector for checking an error of the multiplexed video stream. A PCR extractor for receiving PCR_PID from the program information decoder and extracting PCR information from a corresponding DSS packet header, a system clock restorer for recovering an accurate system clock from the extracted PCR information, and an error-checked multiplexed video stream through a buffer PID processor for identifying Packet_ID to determine validity of multiplexed video stream packet, system header processor for extracting and analyzing system header of valid multiplexed video stream, and temporarily storing metadata extracted from system header processor. Metadata buffer and multiplexed image streams not valid in PID processor The multi-image communication system for a three-dimensional image, including a search function, characterized in that it comprises a timing recovery and storage buffer for storing and reconstructing the