WO2006129914A1 - Multi-view image system and method for compressing and decompressing applied to the same - Google Patents
Multi-view image system and method for compressing and decompressing applied to the same Download PDFInfo
- Publication number
- WO2006129914A1 WO2006129914A1 PCT/KR2006/001273 KR2006001273W WO2006129914A1 WO 2006129914 A1 WO2006129914 A1 WO 2006129914A1 KR 2006001273 W KR2006001273 W KR 2006001273W WO 2006129914 A1 WO2006129914 A1 WO 2006129914A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- images
- residual image
- residual
- threshold value
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000007906 compression Methods 0.000 claims abstract description 31
- 230000006835 compression Effects 0.000 claims abstract description 30
- 238000003384 imaging method Methods 0.000 claims abstract description 30
- 230000005540 biological transmission Effects 0.000 claims abstract description 26
- 230000006837 decompression Effects 0.000 claims description 17
- 230000002194 synthesizing effect Effects 0.000 claims description 12
- 230000015572 biosynthetic process Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 238000003786 synthesis reaction Methods 0.000 description 8
- 230000001965 increasing effect Effects 0.000 description 4
- 238000001308 synthesis method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 238000001093 holography Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/507—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction using conditional replenishment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Disclosed is a compression method applied to a multi-view stereo imaging system comprising steps of, out of first, second, and third images captured at the same time by first, second, and third cameras consecutively located, estimating a disparity vector corresponding to a motion vector between the first and third images; creating an intermediate image having an intermediate value of the estimated disparity vector; comparing the residual image with a threshold value by creating a residual image created based on a difference between the intermediate image and the second image; and encoding and transmitting the first and third images, encoding and transmitting data regarding non-transmission of the residual image when the residual image is lower than the threshold value, and encoding and transmitting the residual image when the residual image is higher than the threshold value. Thus, the present compression method can increase a compression rate since it does not transmit data compressed for the second image at the time when a difference between the intermediate image and the second image is lower than the threshold certain value.
Description
Description
MULTI-VIEW IMAGE SYSTEM AND METHOD FOR COMPRESSING AND DECOMPRESSING APPLIED TO THE
SAME
Technical Field
[1] The present invention relates to a multi-view stereo imaging system and a compression/decompression method applied thereto, and more particularly, to a multi- view stereo imaging system and a compression/decompression method applied thereto for transmitting, receiving, and decompressing compression-ratio-enhanced images. Background Art
[2] Studies are actively undertaken across the world on developing three-dimensional video technologies including multi-view video technologies, especially in USA, Europe, and Japan. In Europe, in order to develop a new three-dimensional TV to substitute for HDTV and two-dimensional image medium, the European Union carried out a joint project COST230 (1991 1996) for standardizations for three-dimensional TV-related devices, three-dimensional image signal encoding, and transmission technologies therefor. As a result, the European Union developed the three-dimensional image display and image transmission service technologies, and so on. Then, the Package for New Operational Autostereoscopic Multi-view systems (PANORAMA) (1991-2001) project of the Advisory Committee for Advanced Television Service (ACTS) is aiming at developing multi-view stereo imaging systems for the purpose of utilizing three-dimensional imaging remote display devices in communications, and is developing multi-view video-related technologies. Also, as another three-dimensional technology-related project, the Advanced Three-Dimensional Television System Technologies (ATTEST) project by eight European organizations such as Philips and HHI in 2002 is in progress for the purpose of developing three- dimensional TV systems. The ATTEST performs researches with a goal for developing systems compatible with the present two-dimensional digital TV while enabling users to enjoy three-dimensional images by additionally sending three-dimensional depth information.
[3] Japan is attempting to realize three-dimensional television sets familiar with human beings through the high definition three-dimensional project (1997-2002). In order to solve the problems such as unnatural distance feelings of existing stereo-view TV and fatigue coming from long-hour TV watching, Japan attempted to develop a three- dimensional TV by utilizing special display technologies using a multi-view imaging method or holography, and performed studies on three-dimensional display devices,
three-dimensional imaging systems, stereo visions, and the like. Disclosure of Invention
Technical Problem
[4] On the other hand, currently, one transmission channel is limited to 6 MHz in the
DTV standards adopting MPEG2, which corresponds to the compression capability of transmission of a sheet of HD-class image through a channel. In such a given channel environment, there exists a limitation to the bandwidth in transmitting an HD-class stereo image, i.e. two sheets of HD-class images. Also, the multi-view video compression requires a much broader bandwidth compared to the compression transmission of an existing single camera, so that a high-efficiency and high- compression-ratio compression method is required compared to a conventional method. Technical Solution
[5] An aspect of the present invention is to provide a multi-view stereo imaging system and a compression/decompression method applied thereto, capable of enhancing a compression ratio by preventing data for a substantial image from being transmitted, if a difference between an intermediate image and a substantial image is lower than a certain value, wherein the intermediate image is created based on the images captured by first and third cameras of consecutively neighboring first, second, and third cameras and the substantial image is captured by the second camera.
[6] The foregoing and other objects and advantages are substantially realized by providing a compression method applied to a multi-view stereo imaging system according to the present invention, comprising steps of, out of first, second, and third images captured by first, second, and third cameras consecutively located at the same time, estimating a disparity vector corresponding to a motion vector between the first and third images; creating an intermediate image having a intermediate value of the estimated disparity vector; creating a residual image created based on a difference between the intermediate image and the second image, and comparing the residual image with a threshold value; and encoding and transmitting the first and third image, encoding and transmitting data regarding non-transmission of the residual image if the residual image is lower than the threshold value, and encoding and transmitting the residual image if the residual image is higher than the threshold value.
[7] Preferably, the step of estimating the disparity vector estimates the disparity vector over the first and third images by the block unit.
[8] The step of comparing the residual image with the threshold value calculates any of mean square error (MSE), Sum of Absolute Difference (SAD), Median Absolute Deviation (MAD), and Peak signal to noise ratio (PSNR) based on a pixel value
difference between the intermediate image and the second image, and compares a calculation result with the threshold value.
[9] Meanwhile, a decompression method applied to a multi-view stereo imaging system according to the present invention, comprising steps of, out of first, second, and third images respectively captured by first, second, and third cameras consecutively located, receiving compressed data for the first and third images, and decompressing the first and third images; judging whether to receive a residual image created based on a difference between an intermediate image and the second image, the intermediate image being created based on a disparity vector corresponding to a motion vector estimated based on the first and third images; and decompressing the intermediate image as the second image by creating the intermediate image based on the decompressed first and third images when receiving the residual image, and decompressing the second image by synthesizing the intermediate image and the transmitted residual image when not receiving the residual image.
[10] A compression and decompression method applied to a multi-view stereo imaging system according to the present invention comprises steps of, out of first, second, and third images respectively captured at the same time by first, second, and third cameras consecutively located, estimating a disparity vector corresponding to a motion vector between the first and third images; creating an intermediate image having an intermediate value of the estimated disparity vector, creating a residual image based on a difference between the intermediate image and the second image, and comparing the residual image with a threshold value; encoding and transmitting the first and third images, and encoding and transmitting data regarding non-transmission of the residual image when the residual image is lower than the threshold value, or the residual image when the residual image is higher than the threshold value; decompressing the transmitted first and third images, and judging whether to receive the residual image; and decompressing the intermediate image as the second image by creating the intermediate image based on the decompressed first and third images upon receiving the residual image, and decompressing the second image by synthesizing the intermediate image and the transmitted residual image upon not receiving the residual image.
[11] A transmitter for a multi-view stereo imaging system according to the present invention comprises, out of first, second, and third images captured at the same time by first, second, and third cameras consecutively located, an estimator for estimating a disparity vector corresponding to a motion vector between the first and third images; an intermediate image creator for creating an intermediate image of the first and third images based on the estimated disparity vector; a comparator for detecting a residual image created based on a difference between the intermediate image and the second image, and judging whether to transmit the residual image by comparing the residual
image with a threshold value; and an encoder for encoding the first and third image, and encoding either the residual image or data regarding non-transmission of the residual image depending on a result of the comparison of the residual image to the threshold value.
[12] The comparator detects the residual image by calculating any of mean square error
(MSE), Sum of Absolute Difference (SAD), Median Absolute Deviation (MAD), and Peak signal to noise ratio (PSNR) based on a pixel value difference between the intermediate image and the second image.
[13] The encoder encodes data for the residual image when the residual image is higher than the threshold value, and encodes data regarding non-transmission of the residual image when the residual image is lower than the threshold value.
[14] The estimator estimates by the block unit the disparity vector over the first and third images.
[15] A receiver for the multi-view stereo imaging system according to the present invention comprises, in order to decompress first, second, and third images captured by cameras consecutively located, a judgment unit for judging whether to receive a residual image created based on a difference between an intermediate image and a second image, the intermediate image being synthesized based on the first image and the third image; an intermediate image synthesizer for decompressing the first and third image, estimating a disparity vector corresponding to a motion vector between the first and third images, and creating an intermediate image of the first and third images based on the estimated disparity vector; and a calculator for decompressing the intermediate image as the second image when not receiving the residual image, and decompressing the second image by synthesizing the intermediate image and the transmitted residual image when receiving the residual image.
[16] A multi-view stereo imaging system according to the present invention comprises, out of first, second, and third images captured by first, second, and third cameras consecutively located, an estimator for estimating a disparity vector corresponding to a motion vector between the first and third images, an intermediate image creator for creating an intermediate image of the first and third images based on the estimated disparity vector; a comparator for detecting a residual image created based on a difference between the intermediate image and the second image, and judging whether to transmit the residual image by comparing the residual image with a threshold value; an encoder for encoding the first and third images, and encoding either data for the residual image or data regarding non-transmission of the residual image depending on a result of the comparison of the residual image with the threshold value; a judgment unit for judging whether the residual image is received; an intermediate image synthesizer for decompressing the first and third images, estimating a disparity vector
between the first and third images, and synthesizing an intermediate image of the first and third images based on the estimated disparity vector; and a calculator for decompressing the intermediate image to the second image when the residual image is not received, and decompressing the second image by synthesizing the intermediate image and the transmitted data for the residual image when the residual image is received.
Advantageous Effects
[17] As described above, the present invention can increase a compression rate since it does not transmit data compressed for the second image at the time when a difference between an intermediate image and the second image is lower than a certain value, wherein the intermediate image is created through intermediate synthesis of first and third images upon compression of the first, second, and third images respectively captured by the first, second, and third cameras which are consecutively located. Brief Description of the Drawings
[18] FIG. 1 is a block diagram for showing a multi-view stereo imaging system according to an embodiment of the invention;
[19] FIG. 2 is a flow chart for explaining a compression method applied to the multi- view stereo imaging system of FIG. 1 ;
[20] FIGS. 3 to 5 are views for explaining in detail a compression method applied to a multi-view stereo imaging system;
[21] FIG. 6 is a flow chart for explaining a decompression method applied to the multi- view stereo imaging system of FIG. 1 ; and
[22] FIG. 7 is a view for explaining in detail a decompression method applied to a multi- view stereo imaging system. Best Mode for Carrying Out the Invention
[23] Hereafter, certain embodiments the present invention will be described in detail with reference to the accompanying drawings.
[24] FIG. 1 is a block diagram for showing a multi-view stereo imaging system according to an embodiment of the present invention.
[25] In FIG. 1, the multi-view stereo imaging system according to the present invention comprises a transmitter 100 and a receiver 200. In here, the receiver 200 can be a multi-view television set, a 3D-TV, or a monitor.
[26] First, the transmitter 100 includes a storage unit 110, an estimator 120, a compression unit 130, and a multiplexer 140. The transmitter 100 receives plural images captured by plural cameras located at certain intervals. The storage unit 110 stores the received images frame by frame.
[27] Out of first, second, and third images, out of stored input images, captured by first,
second, and third cameras consecutively located, the estimator 120 estimates a disparity vector corresponding to a motion vector between the first and third images. In here, the first, second, and third images are the images of the same object captured by the first, second, and third cameras at the same time. The first, second, and third images of the same object can be captured as different images by the first, second, and third cameras due to different locations of the first, second, and third cameras. Accordingly, the estimator 120 estimates the disparity vector corresponding to the motion vector between the first image and the third image created due to the location difference between the first and third cameras.
[28] In here, the disparity vector is estimated over the first and third images by the block unit. Preferably, the disparity vector is estimated by the 16X16 block unit, and also can be estimated by the block unit smaller than the 16X16 block unit.
[29] The compression unit 130 compresses information on the first, third, and second images based on a difference value between the substantial second image and the second image synthesized based on the first and third images, and has an intermediate image creator 131, a comparator 133, and an encoder 135.
[30] The intermediate image creator 131 creates an intermediate image based on the first and third images based on an intermediate image synthesis method. The intermediate image creator 131 creates an intermediate image by taking half the disparity vector value estimated by the estimator 120, and creates an intermediate image by the block unit used when the disparity vector is estimated. Accordingly, the disparity vector between the first image and the synthesized intermediate image is the same, by the block unit, as the estimated disparity vector between the third image and the synthesized intermediate image.
[31] The comparator 133 detects a residual image caused by the pixel value difference between the intermediate image synthesized by the intermediate image creator 131 and the substantial second image stored in the storage unit 110, and compares the residual image with a threshold value. The Mean Square Error (MSE), Sum of Absolute Difference (SAD), Median Absolute Deviation (MAD), Peak Signal to Noise Ratio (PSNR), or the like can be used for detection of the residual image. For example, when the MSE method is used, the sum of the pixel value difference between the synthesized intermediate image and the substantial second image is detected as a residual image.
[32] Further, the comparator 133 compares the detected residual image and the threshold value, and judges whether to transmit data regarding the residual image. In here, the threshold value is an experimental value, and refers to a critical value at which image distortion can be avoided when a synthesized intermediate image is decompressed to the second image since a pixel value difference between the synthesized intermediate image and the substantial second image is small.
[33] The encoder 135 encodes either data for the residual image or data regarding non- transmission of the residual image according to a result of the comparison of the residual image and the threshold value. If the residual image is higher than the threshold value as a result of the comparison by the comparator 133, data for the residual image is encoded when the image data is compressed, and, if the residual image is lower than the threshold value, data regarding the non-transmission of the residual image is encoded when the image data is compressed. Accordingly, if the residual image is lower than the threshold value, the encoder 135 does not encodes data for the entire residual image, but encodes only the data for the residual image not transmitted to the receiver 200, thereby increasing a compression ratio.
[34] The multiplexer 140 transmits individual bit streams, which are encoded, to the receiver 200 by performing multiplexing appropriate to a format of a bit stream transmission medium.
[35] The receiver 200 has a demultiplexer 210, a memory unit 220, judgment unit 230, and a decompression unit 240. The receiver 200 receives data for images captured by the consecutively located cameras. The demultiplexer 210 splits the received data into individual bit streams, and outputs the split bit streams, and the memory unit 220 stores the bit streams split by the demultiplexer 210.
[36] In order to decompress the first, second, and third images captured by the first, second, and third cameras consecutively located, the judgment unit 230 judges whether data for the residual image created by the difference between the intermediate image and the substantial second image is transmitted, wherein the first and third images are used for synthesis of the intermediate image.
[37] The decompression unit 240 decompresses the first, second, third images by using a result of the judgment on whether the data for the residual image is transmitted and by using data for the image stored in the memory unit 220, and has an intermediate image synthesizer 241 and an calculator 243. Further, the decompression unit 240 decompresses the first and third images based on the data stored in the memory unit 220.
[38] The intermediate image synthesizer 241 synthesizes the intermediate image by using the stored first and third images. The intermediate image synthesizer 241 estimates by the block unit a disparity vector corresponding to a motion vector between the first and third images over the decompressed image, and creates an intermediate image by taking half an estimated disparity vector value.
[39] As a result of the judgment of the judgment unit 230, if the data for the residual image is not transmitted, the calculator 243 decompresses the intermediate image, which is created by the intermediate image synthesizer 241, to a second image. However, if the data for the residual image is transmitted, the calculator 243 decompresses the second image by synthesizing the transmitted data for the residual
image and the intermediate image created by the intermediate image synthesizer 241.
[40] So far, description has been made on compressing information of the second image depending on whether a residual image is transmitted, wherein the residual image is created based on a difference value between a substantial second image and an intermediate image created based on the first and third images, and when compressing to a fourth image our of the third, fourth, and fifth images captured by third, fourth, and fifth cameras consecutively located, the method for compression into the second image is applied in the same manner to compress the third and fifth images into information on the fourth image. That is, for the even-numbered image captured by the even- numbered camera, the information on the even-numbered image is compressed and transmitted based on the neighboring odd-numbered images, so that a compression ratio can be increased.
[41] FIG. 2 is a flowchart for explaining a compression method applied to the multi- view stereo imaging system of FIG. 1. Further, FIGS. 3 to 5 are views for explaining FIG. 2 in detail.
[42] In FIGS. 2 and 3 A, when plural cameras located in a certain interval capture images, a step S 801 estimates a disparity vector corresponding to a motion vector between the first and third images out of the first, second, and third images captured by the first, second, and third cameras consecutively located. In here, the disparity vector estimation is to estimate the difference caused by the locations of the cameras between the images captured at the same time by the first and third cameras with respect to the same object.
[43] FIG. 3 is a view for showing the images captured by plural cameras located at certain intervals together with camera numbers and image-captured times. In here, reference numerals Cl, C2, and C3 denote the first, second, and third cameras consecutively located, respectively, and the first, second, and third images are denoted as image al, image bl, and image cl, respectively, wherein the first, second, and third images are images captured by the cameras Cl, C2, and C3 at the same time Tl. The image al as the first image and the image cl as the third image are used for disparity vector estimation.
[44] FIG. 4 is a view for showing the estimation for a disparity vector DV corresponding to a motion vector between the images al and cl. In FIG. 4, the disparity vector between the images al and cl is estimated block by block. In here, it is preferable to set a block size to 16 x 16, and a disparity vector can be estimated by the 8 x 8 block unit or by the unit of blocks smaller than 8 x 8 blocks depending on characteristics of an image.
[45] Next, a step S803 synthesizes the second image by using the estimated disparity vector. The step S 803 creates the second image by using the intermediate vector in-
terpolation (IVI) method based on a disparity vector between the images al and cl.
[46] FIG. 5 is a view for showing a method for creating an image /bl being the second image synthesized based on the disparity vector between the image al as the first image and the image cl as the third image. The method shown in FIG. 5 can synthesize an intermediate image corresponding to an image captured by the camera C2 by using a disparity vector. In here, a disparity vector between the images al and /bl and a disparity vector between the images /bl and cl have relations with a disparity vector between the images al and cl, which can be expressed in Equation 1 as below.
[47] [Equation 1]
[48]
[49] In here, DV denotes a disparity vector between the images al and /bl, and DV ab cb denotes a disparity vector between the images cl and /bl. Further, DV ac denotes a disparity vector between the images al and cl. As shown in Equation 1, the disparity vector between the images al and /bl is half the disparity vector between the images al and cl, and the disparity vector between the images cl and /bl is also half the disparity vector between the images al and cl.
[50] Accordingly, the method shown in FIG. 5 can synthesize the second image by taking half the disparity vector between the images al and cl that is an estimated disparity vector, and creates by the block unit the second image in which the disparity vector is estimated.
[51] Next, a step S805 detects a residual image created based on a difference between the synthesized second image and the substantial second image. The step S 805 can use the Mean Square Error (MSE), Sum of Absolute Difference (SAD), Median Absolute Deviation (MAD), Peak Signal to Noise Ratio (PSNR), and so on, to obtain the residual image caused by the difference between the synthesized second image /bl and the substantial second image bl captured by the camera C2. If the MSE is used to detect a residual image, the residual image can be expressed in Equation 2 as below.
[52] [Equation 2]
[53]
m = ∑∑b\{x,y) -lb\{x,>)
[54] In here, m denotes a residual image as a sum of pixel value differences between a
synthesized second image and a substantial second image, and bl denotes the substantial second image. Further, /bl denotes the second image synthesized with the image al as the first image and the image cl as the third image, and x and y denote horizontal and vertical coordinates of the pixels in the image. As shown in Equation 2, if the MSE is used, the residual image can be expressed as the sum of the pixel value differences between the synthesized second image and the substantial second image.
[55] Next, a step S807 compares the residual image with a threshold value. In here, the threshold value is determined by experiments, which refers to a threshold value at which the residual image does not have to be taken into account due to a small difference between the substantial second image and the second image synthesized based on the first and third images. Further, the threshold value depends on a method obtaining residual images, such as MSE, SAD, MAD, PSNR, and so on.
[56] When the residual image is higher than the threshold value, a step S809 transmits the residual image. When the residual image is higher than the threshold value, data for the residual image is compressed and transmitted together with data for the first and third images. Accordingly, the receiver can decompress the second image by using the first, third, and residual images.
[57] The residual image lower than a threshold value can occur when there exists big differences among first, second, and third images which are obtained with respect to the same object at the same time, since the cameras Cl, C2, and C3 capturing the first, second, and third images are quite apart from one another. Specifically, when an object placed relatively ahead forms a hidden area, the first, second, and third images can have a big difference therebetween. That is, when the object placed in the hidden area can or can not be captured depending on the locations of the cameras, the second image synthesized based on the first and third images can be quite different from the substantial second image.
[58] Alternatively, a step S811 transmits information on non-transmission of the residual image if the residual image is lower than the threshold value. The residual image is lower than the threshold value at the time the second image synthesized based on the first and third images has a little difference compared to the substantial second image captured by the camera C2. In here, image distortion problem may not occur even at the time when only the received first and third images are used for synthesis of the second image without the substantial second image transmitted to the receiver. Accordingly, when the residual image is lower than the threshold value, only the information on the non-transmission of the residual image is transmitted to the receiver rather than transmission of data for residual image, so that a compression ratio of data for the image can be increased.
[59] FIG. 6 is a flowchart for explaining a decompression method applied to the multi-
view stereo imaging system of FIG. 1, and FIG. 7 is a view for explaining FIG. 6 in de tail.
[60] In FIG. 6, first, a step S901 decompresses the first and third images by using the transmitted data.
[61] Next, a step S903 judges whether or not a residual image is transmitted. It is judged whether data regarding the non-transmission of the residual image is transmitted or the residual image is transmitted. If the residual image is lower than the threshold value, the transmitter does not transmit data regarding the residual image, but transmits data regarding the non-transmission of the residual image.
[62] If the residual image is transmitted, the first and third images decompressed in the step S901 are used for synthesis of an intermediate image (S905). The intermediate image synthesis method is used to create an intermediate image corresponding to the second image based on the decompressed first and third images. In the same manner as the residual image compression process, the step S905 estimates block by block a disparity vector corresponding to a motion vector between the decompressed first and third images, and creates the intermediate image by taking half the estimated disparity vector.
[63] Next, a step S907 synthesizes the synthesized intermediate image and the residual image, and decompresses the second image. If the residual image is transmitted, the step S907 decompresses the second image by synthesizing the transmitted residual image and the synthesized intermediate image again since image distortion can occur when the synthesized intermediate image is decompressed to the second image. That is, the step S907 uses the first and third images to synthesize the synthesized intermediate image and the residual image caused by a difference between the intermediate image and the substantial second image, and decompresses the second image.
[64] On the other hand, if the residual image is not transmitted in step S903, a step S909 decompresses the second image by synthesizing an intermediate image based on the first and third images. The residual image is not transmitted at the time when a difference between the substantial second image and an image synthesized by the intermediate image synthesis method using the first and third images is lower than the threshold value, so the step S909 can decompress the synthesized image as the second image.
[65] Meanwhile, FIG. 7 is a view for showing a sequence of images used for decompression of an image through intermediate image synthesis and decompressed through the intermediate image synthesis over time together with camera numbers.
[66] In FIG. 7, out of plural cameras located in predetermined intervals, the camera Cl captures an image al, the camera C3 captures an image cl, and the camera C5 captures
an image el, and data for the images al, cl, and el is received for decompressions from the transmission, wherein the cameras Cl, C3, and C5 are odd-numbered cameras. However, an image captured by an even-numbered camera C2 is decompressed through synthesis of the images al and cl of the neighboring cameras as an intermediate image, and an image captured by a camera C4 is decompressed through synthesis of the images cl and el of the neighboring cameras as an intermediate image. Accordingly, out of the images captured by the consecutively located cameras, the even-numbered images captured by the even-numbered cameras can be decompressed through the odd-numbered images captured by the odd-numbered cameras.
[67] Meanwhile, occasionally, out of the first, second, and third images respectively captured by first, second, and third cameras consecutively located, it can increase a compression rate of an input image to use images such as first and fourth images captured by non-consecutively located cameras for compression and decompression of the second image, rather than the first and third images being used for the compression and decompression of the second image.
[68] For example, when the fourth image is similar to the second image rather than to the third image, it is possible that a residual image between a synthesized image and the second image is higher than the threshold value, the synthesized image being created based on the disparity vector between the first image and the third image, but, it is possible that the residual image between the synthesized image and the second image is lower than the threshold value, the synthesized image being created based on the disparity vector between the first image and the fourth image. Accordingly, when the second image is compressed based on the first image and the third image, the residual image must be encoded, but when the second image is compressed based on the first image and the fourth image, data regarding the non-transmission of the residual image is transmitted, so that a compression ratio can be increased.
[69] While the invention has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Industrial Applicability
[70] The present invention applies to a multi-view stereo imaging system for transmitting, receiving, and decompressing compression-ratio-enhanced images.
Claims
[1] A compression method applied to a multi-view stereo imaging system, comprising steps of, out of first, second, and third images respectively captured by first, second, and third cameras consecutively located at the same time, estimating a disparity vector corresponding to a motion vector between the first and third images; creating an intermediate image having a intermediate value of the estimated disparity vector; creating a residual image created based on a difference between the intermediate image and the second image, and comparing the residual image with a threshold value; and encoding and transmitting the first and third image, encoding and transmitting data regarding non-transmission of the residual image if the residual image is lower than the threshold value, and encoding and transmitting the residual image if the residual image is higher than the threshold value.
[2] The method as claimed in claim 1, wherein the step of estimating the disparity vector estimates the disparity vector over the first and third images by the block unit.
[3] The method as claimed in claim 1, wherein the step of comparing the residual image with the threshold value calculates any of mean square error (MSE), Sum of Absolute Difference (SAD), Median Absolute Deviation (MAD), and Peak signal to noise ratio(PSNR) based on a pixel value difference between the intermediate image and the second image, and compares a calculation result with the threshold value.
[4] A decompression method applied to a multi-view stereo imaging system, comprising steps of, out of first, second, and third images respectively captured by first, second, and third cameras consecutively located, receiving compressed data for the first and third images, and decompressing the first and third images; judging whether to receive a residual image created based on a difference between an intermediate image and the second image, the intermediate image being created based on a disparity vector corresponding to a motion vector estimated based on the first and third images; and decompressing the intermediate image as the second image by creating the intermediate image based on the decompressed first and third images when receiving the residual image, and decompressing the second image by synthesizing the intermediate image and the transmitted residual image when not
receiving the residual image.
[5] A compression and decompression method applied to a multi-view stereo imaging system, comprising steps of, out of first, second, and third images respectively captured at the same time by first, second, and third cameras consecutively located, estimating a disparity vector corresponding to a motion vector between the first and third images; creating an intermediate image having an intermediate value of the estimated disparity vector, creating a residual image based on a difference between the intermediate image and the second image, and comparing the residual image with a threshold value; encoding and transmitting data regarding non-transmission of the residual image when the residual image is lower than the threshold value, or the residual image when the residual image is higher than the threshold value; decompressing the transmitted first and third images, and judging whether to receive the residual image; and decompressing the intermediate image as the second image by creating the intermediate image based on the decompressed first and third images upon receiving the residual image, and decompressing the second image by synthesizing the intermediate image and the transmitted residual image upon not receiving the residual image.
[6] A transmitter for a multi-view stereo imaging system, comprising, out of first, second, and third images captured by first, second, and third cameras consecutively located, an estimator for estimating a disparity vector corresponding to a motion vector between the first and third images; an intermediate image creator for creating an intermediate image of the first and third images based on the estimated disparity vector; a comparator for detecting a residual image created based on a difference between the intermediate image and the second image, and judging whether to transmit the residual image by comparing the residual image with a threshold value; and an encoder for encoding the first and third image, and encoding either the residual image or data regarding non-transmission of the residual image depending on a result of the comparison of the residual image to the threshold value.
[7] The transmitter as claimed in claim 6, wherein the comparator detects the residual image by calculating any of mean square error (MSE), Sum of Absolute
Difference (SAD), Median Absolute Deviation (MAD), and Peak signal to noise ratio (PSNR) based on a pixel value difference between the intermediate image and the second image.
[8] The transmitter as claimed in claim 6, wherein the encoder encodes data for the residual image when the residual image is higher than the threshold value, and encodes data regarding non-transmission of the residual image when the residual image is lower than the threshold value.
[9] The transmitter as claimed in claim 6, wherein the estimator estimates by the block unit the disparity vector over the first and third images.
[10] A receiver for the multi-view stereo imaging system, comprising, in order to decompress first, second, and third images captured by cameras consecutively located, a judgment unit for judging whether to receive a residual image created based on a difference between an intermediate image and a second image, the intermediate image being synthesized based on the first image and the third image; an intermediate image synthesizer for decompressing the first and third image, estimating a disparity vector corresponding to a motion vector between the first and third images, and creating an intermediate image of the first and third images based on the estimated disparity vector; and a calculator for decompressing the intermediate image as the second image when not receiving the residual image, and decompressing the second image by synthesizing the intermediate image and the transmitted residual image when receiving the residual image.
[11] A multi-view stereo imaging system, comprising, out of first, second, and third images captured by first, second, and third cameras consecutively located, an estimator for estimating a disparity vector corresponding to a motion vector between the first and third images, an intermediate image creator for creating an intermediate image of the first and third images based on the disparity vector; a comparator for detecting a residual image created based on a difference between the intermediate image and the second image, and judging whether to transmit the residual image by comparing the residual image with a threshold value; an encoder for encoding either data for the residual image or data regarding non- transmission of the residual image depending on a result of the comparison of the residual image with the threshold value; a judgment unit for judging whether the residual image is received; an intermediate image synthesizer for creating an intermediate image of the first
and third images based on the estimated disparity vector between the first and third images; and a calculator for decompressing the intermediate image to the second image when the residual image is not received, and decompressing the second image by synthesizing the intermediate image and the transmitted data for the residual image when the residual image is received.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP06757406.1A EP1889490A4 (en) | 2005-05-31 | 2006-04-07 | Multi-view image system and method for compressing and decompressing applied to the same |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR10-2005-0046431 | 2005-05-31 | ||
| KR1020050046431A KR100636785B1 (en) | 2005-05-31 | 2005-05-31 | Multi-view image system and method for compressing and decompressing applied to the same |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2006129914A1 true WO2006129914A1 (en) | 2006-12-07 |
Family
ID=37463347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/KR2006/001273 WO2006129914A1 (en) | 2005-05-31 | 2006-04-07 | Multi-view image system and method for compressing and decompressing applied to the same |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US8130836B2 (en) |
| EP (1) | EP1889490A4 (en) |
| KR (1) | KR100636785B1 (en) |
| WO (1) | WO2006129914A1 (en) |
Families Citing this family (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100720722B1 (en) * | 2005-06-21 | 2007-05-22 | 삼성전자주식회사 | Intermediate vector interpolation method and 3D display apparatus |
| EP2177036A2 (en) * | 2007-08-15 | 2010-04-21 | Thomson Licensing | Methods and apparatus for motion skip mode in multi-view coded video using regional disparity vectors |
| CN101415115B (en) * | 2007-10-15 | 2011-02-02 | 华为技术有限公司 | Method for encoding and decoding video based on movement dancing mode, and encoder and decoder thereof |
| CN101415114B (en) * | 2007-10-17 | 2010-08-25 | 华为终端有限公司 | Method and apparatus for encoding and decoding video, and video encoder and decoder |
| KR100950046B1 (en) * | 2008-04-10 | 2010-03-29 | 포항공과대학교 산학협력단 | Apparatus of multiview three-dimensional image synthesis for autostereoscopic 3d-tv displays and method thereof |
| KR101520620B1 (en) * | 2008-08-18 | 2015-05-18 | 삼성전자주식회사 | Method and apparatus for determining a two- or three-dimensional display mode of an image sequence |
| US8400570B2 (en) * | 2008-10-09 | 2013-03-19 | Manufacturing Resources International, Inc. | System and method for displaying multiple images/videos on a single display |
| TW201041392A (en) * | 2009-05-05 | 2010-11-16 | Unique Instr Co Ltd | Multi-view 3D video conference device |
| KR20100131061A (en) | 2009-06-05 | 2010-12-15 | 삼성전자주식회사 | Apparatus and method for stereo images processing |
| US9013560B2 (en) * | 2009-06-16 | 2015-04-21 | Lg Electronics Inc. | Viewing range notification method and TV receiver for implementing the same |
| KR20110011000A (en) * | 2009-07-27 | 2011-02-08 | 삼성전자주식회사 | Method and appratus for generating three-dimensional image datastream including additional information for displaying three-dimensional image, method and apparatus for receiving the same |
| KR101056306B1 (en) * | 2009-07-31 | 2011-08-11 | 삼성모바일디스플레이주식회사 | Digital display and its image arranging method |
| KR101645465B1 (en) * | 2010-07-23 | 2016-08-04 | 삼성전자주식회사 | Apparatus and method for generating a three-dimension image data in portable terminal |
| TW201205180A (en) * | 2010-07-27 | 2012-02-01 | Hon Hai Prec Ind Co Ltd | Camera |
| US8520080B2 (en) | 2011-01-31 | 2013-08-27 | Hand Held Products, Inc. | Apparatus, system, and method of use of imaging assembly on mobile terminal |
| US20120229595A1 (en) * | 2011-03-11 | 2012-09-13 | Miller Michael L | Synthesized spatial panoramic multi-view imaging |
| US9693033B2 (en) | 2011-11-11 | 2017-06-27 | Saturn Licensing Llc | Transmitting apparatus, transmitting method, receiving apparatus and receiving method for transmission and reception of image data for stereoscopic display using multiview configuration and container with predetermined format |
| JP6192902B2 (en) * | 2011-11-11 | 2017-09-06 | サターン ライセンシング エルエルシーＳａｔｕｒｎ Ｌｉｃｅｎｓｉｎｇ ＬＬＣ | Image data transmitting apparatus, image data transmitting method, image data receiving apparatus, and image data receiving method |
| KR20130057764A (en) * | 2011-11-24 | 2013-06-03 | 삼성전자주식회사 | Digital photographing apparatus and control method thereof |
| US20140009462A1 (en) * | 2012-04-17 | 2014-01-09 | 3Dmedia Corporation | Systems and methods for improving overall quality of three-dimensional content by altering parallax budget or compensating for moving objects |
| JP2014082540A (en) * | 2012-10-12 | 2014-05-08 | National Institute Of Information & Communication Technology | Method, program and apparatus for reducing data size of multiple images including information similar to each other, and data structure representing multiple images including information similar to each other |
| KR101918030B1 (en) | 2012-12-20 | 2018-11-14 | 삼성전자주식회사 | Method and apparatus for rendering hybrid multi-view |
| CN103813171B (en) * | 2014-01-17 | 2017-04-19 | 西安空间无线电技术研究所 | Method of improving compression ratio of existing data compression method |
| US10319408B2 (en) | 2015-03-30 | 2019-06-11 | Manufacturing Resources International, Inc. | Monolithic display with separately controllable sections |
| US10922736B2 (en) | 2015-05-15 | 2021-02-16 | Manufacturing Resources International, Inc. | Smart electronic display for restaurants |
| US10269156B2 (en) | 2015-06-05 | 2019-04-23 | Manufacturing Resources International, Inc. | System and method for blending order confirmation over menu board background |
| US10319271B2 (en) | 2016-03-22 | 2019-06-11 | Manufacturing Resources International, Inc. | Cyclic redundancy check for electronic displays |
| KR102204132B1 (en) | 2016-05-31 | 2021-01-18 | 매뉴팩처링 리소시스 인터내셔널 인코포레이티드 | Electronic display remote image verification system and method |
| US10510304B2 (en) | 2016-08-10 | 2019-12-17 | Manufacturing Resources International, Inc. | Dynamic dimming LED backlight for LCD array |
| KR102503743B1 (en) * | 2018-04-11 | 2023-02-28 | 삼성전자주식회사 | Apparatus and method for processing picture |
| RU2690757C1 (en) * | 2018-08-21 | 2019-06-05 | Самсунг Электроникс Ко., Лтд. | System for synthesis of intermediate types of light field and method of its operation |
| EP4074049A1 (en) * | 2019-12-11 | 2022-10-19 | InterDigital VC Holdings, Inc. | A method and apparatus for encoding and decoding of multiple-viewpoint 3dof+ content |
| US11895362B2 (en) | 2021-10-29 | 2024-02-06 | Manufacturing Resources International, Inc. | Proof of play for images displayed at electronic displays |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6289049B1 (en) * | 1997-07-30 | 2001-09-11 | Lg Electronics Inc. | Method for coding motion vector in moving picture |
| KR20020032954A (en) * | 2000-10-28 | 2002-05-04 | 김춘호 | 3D Stereosc opic Multiview Video System and Manufacturing Method |
| US20040131120A1 (en) * | 2003-01-02 | 2004-07-08 | Samsung Electronics Co., Ltd. | Motion estimation method for moving picture compression coding |
| KR20040065014A (en) * | 2003-01-13 | 2004-07-21 | 전자부품연구원 | Apparatus and method for compressing/decompressing multi-viewpoint image |
| KR20050038054A (en) * | 2003-10-21 | 2005-04-27 | 전자부품연구원 | Apparatus for adaptive multiplexing/demultiplexing for 3d multiview video processing and its method |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3733358B2 (en) | 1996-04-05 | 2006-01-11 | 松下電器産業株式会社 | Image transmission apparatus, transmission apparatus, reception apparatus, transmission method, and reception method |
| JP3769850B2 (en) | 1996-12-26 | 2006-04-26 | 松下電器産業株式会社 | Intermediate viewpoint image generation method, parallax estimation method, and image transmission method |
| JP2001502504A (en) * | 1996-10-11 | 2001-02-20 | サーノフ コーポレイション | Apparatus and method for encoding and decoding stereoscopic video |
| US6043838A (en) * | 1997-11-07 | 2000-03-28 | General Instrument Corporation | View offset estimation for stereoscopic video coding |
| US6055274A (en) * | 1997-12-30 | 2000-04-25 | Intel Corporation | Method and apparatus for compressing multi-view video |
| KR100424401B1 (en) | 2001-11-02 | 2004-03-24 | 전자부품연구원 | 3D Stereoscopic Multiview video system include Searching function |
| CN100452883C (en) * | 2001-12-17 | 2009-01-14 | 微软公司 | Skip macroblock coding |
| KR100481732B1 (en) * | 2002-04-20 | 2005-04-11 | 전자부품연구원 | Apparatus for encoding of multi view moving picture |
| KR100828353B1 (en) * | 2003-02-05 | 2008-05-08 | 삼성전자주식회사 | Method for dividing the image block and Apparatus thereof |
| US7672378B2 (en) * | 2005-01-21 | 2010-03-02 | Stmicroelectronics, Inc. | Spatio-temporal graph-segmentation encoding for multiple video streams |
-
2005
- 2005-05-31 KR KR1020050046431A patent/KR100636785B1/en not_active IP Right Cessation
-
2006
- 2006-04-07 WO PCT/KR2006/001273 patent/WO2006129914A1/en active Search and Examination
- 2006-04-07 EP EP06757406.1A patent/EP1889490A4/en not_active Withdrawn
- 2006-05-10 US US11/431,014 patent/US8130836B2/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6289049B1 (en) * | 1997-07-30 | 2001-09-11 | Lg Electronics Inc. | Method for coding motion vector in moving picture |
| KR20020032954A (en) * | 2000-10-28 | 2002-05-04 | 김춘호 | 3D Stereosc opic Multiview Video System and Manufacturing Method |
| US20040131120A1 (en) * | 2003-01-02 | 2004-07-08 | Samsung Electronics Co., Ltd. | Motion estimation method for moving picture compression coding |
| KR20040065014A (en) * | 2003-01-13 | 2004-07-21 | 전자부품연구원 | Apparatus and method for compressing/decompressing multi-viewpoint image |
| KR20050038054A (en) * | 2003-10-21 | 2005-04-27 | 전자부품연구원 | Apparatus for adaptive multiplexing/demultiplexing for 3d multiview video processing and its method |
Also Published As
| Publication number | Publication date |
|---|---|
| EP1889490A1 (en) | 2008-02-20 |
| KR100636785B1 (en) | 2006-10-20 |
| US20060268987A1 (en) | 2006-11-30 |
| EP1889490A4 (en) | 2013-11-20 |
| US8130836B2 (en) | 2012-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8130836B2 (en) | Multi-view stereo imaging system and compression/decompression method applied thereto | |
| US20120224025A1 (en) | Transport stream structure including image data and apparatus and method for transmitting and receiving image data | |
| US8031767B2 (en) | Encoding and decoding multi-view video while accommodating absent or unreliable camera parameters | |
| KR102343700B1 (en) | Video transmission based on independently encoded background updates | |
| US20100238264A1 (en) | Three dimensional video communication terminal, system, and method | |
| JP2007180981A (en) | Device, method, and program for encoding image | |
| US8810628B2 (en) | Image processing apparatus and image processing method | |
| KR20130053452A (en) | Calculating disparity for three-dimensional images | |
| WO2006080739A1 (en) | Method and apparatus for encoding and decoding multi-view video using image stitching | |
| WO2012111757A1 (en) | Image processing device and image processing method | |
| JP2005328299A (en) | Moving picture encoding device and method, and moving picture decoding device and method | |
| JP2007180982A (en) | Device, method, and program for decoding image | |
| US9143758B2 (en) | Method and apparatus for low-bandwidth content-preserving encoding of stereoscopic 3D images | |
| Yang et al. | An MPEG-4-compatible stereoscopic/multiview video coding scheme | |
| KR101832407B1 (en) | Method and system for communication of stereoscopic three dimensional video information | |
| US20140055561A1 (en) | Transmitting apparatus, transmitting method, receiving apparatus and receiving method | |
| CN104782127B (en) | Method and apparatus for processing video signal | |
| JP6114404B2 (en) | Video signal processing method and apparatus | |
| JP2004200814A (en) | Stereoscopic image forming method and stereoscopic image forming device | |
| US20110128355A1 (en) | Image processing apparatus and image processing method | |
| JP2009004942A (en) | Multi-viewpoint image transmitting method, multi-viewpoint image transmitting device, and multi-viewpoint image transmitting program | |
| KR101012760B1 (en) | System and Method for transmitting and receiving of Multi-view video | |
| WO2013141075A1 (en) | Image encoding device, image decoding device, image encoding method, image decoding method, and program | |
| KR101044952B1 (en) | Method and apparatus for transmitting and receiving of video, and transport stream structure thereof | |
| JP2008034893A (en) | Multi-viewpoint image decoder |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| DPE2 | Request for preliminary examination filed before expiration of 19th month from priority date (pct application filed from 20040101) | ||
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
| WWE | Wipo information: entry into national phase |
Ref document number: 2006757406 Country of ref document: EP |
|
| DPE2 | Request for preliminary examination filed before expiration of 19th month from priority date (pct application filed from 20040101) | ||
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| NENP | Non-entry into the national phase |
Ref country code: RU |
|
| WWP | Wipo information: published in national office |
Ref document number: 2006757406 Country of ref document: EP |