WO2014156648A1 - 複数の入力画像をエンコーディングする方法、プログラムを格納する記憶媒体および装置 - Google Patents
複数の入力画像をエンコーディングする方法、プログラムを格納する記憶媒体および装置 Download PDFInfo
- Publication number
- WO2014156648A1 WO2014156648A1 PCT/JP2014/056484 JP2014056484W WO2014156648A1 WO 2014156648 A1 WO2014156648 A1 WO 2014156648A1 JP 2014056484 W JP2014056484 W JP 2014056484W WO 2014156648 A1 WO2014156648 A1 WO 2014156648A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- unit
- residual
- input
- remainder
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/154—Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/187—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
- H04N19/196—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
- H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
Definitions
- the present invention relates to a method for encoding a plurality of different types of input images each containing different information on a subject, a storage medium for storing a program, and a device.
- Non-Patent Document 1 a video coding method that considers redundancy between frames for a moving image composed of a sequence of frames arranged in the time domain.
- a P frame predicted frame
- / or a B frame bi-directional predicted frame
- the P frame is a frame calculated by forward prediction
- the B frame is a frame calculated by any one of forward prediction, backward prediction, and bidirectional prediction.
- Non-Patent Document 2 discloses a method of applying such video encoding technology to the time domain and the spatial domain. That is, according to the teaching content of Non-Patent Document 2, P frames and / or B frames can be generated for a plurality of frames arranged in the time domain and the spatial domain.
- a frame sequence used in 3D video technology that provides high-definition stereoscopic video using multi-view video can be cited.
- Such a stereoscopic video is realized by a multi-view video obtained by imaging a subject from a very large number of viewpoints (for example, 200 viewpoints).
- viewpoint interpolation such as generating a P frame and / or a B frame using three-dimensional information such as a distance map It can also be applied to.
- Non-Patent Document 3 discloses a technique related to encoding of multi-view video.
- Non-Patent Document 3 discloses a method of generating P frames and / or B frames from three-dimensional information such as depth information by using viewpoint interpolation not only in the time domain but also in the spatial domain.
- encoding compressing (converting) data into a code according to the purpose
- decoding decoding
- coding shall mean encoding alone and both encoding and decoding.
- the generated P frame and B frame are transmitted in the form of a residual value.
- a data compression process is further performed on the residual information.
- image conversion typically, discrete cosine conversion
- quantization quantization
- entropy coding and the like are executed.
- the data compression rate is high, performing the quantization causes a significant loss of data due to the data size reduction. That is, residual information having a small value is lost by the data compression process.
- edge information and boundary information must be prevented from being lost even if the data compression rate is increased.
- Encoding technology that balances both compression efficiency and compression quality is required for a plurality of different types of input images each containing different information on a subject.
- a method for encoding a plurality of input images includes a plurality of first input images including first information of a subject and a plurality of second information different from the first information of the subject corresponding to the plurality of first input images, respectively.
- the step of converting into a residue includes performing a modulo operation on pixel values for a region to be defined by the residue, obtaining gradient information of a predicted image, a gradient strength and a modulo operation method, Determining a value that is a modulo arithmetic method based on the obtained gradient information with reference to a predetermined correspondence relationship with the value.
- the step of calculating the first predicted image uses the second input image corresponding to the first input image to be calculated and the one or more first input images preceding the first input image. Calculating a predicted image.
- the step of calculating the second predicted image includes a step of calculating the second predicted image using motion data regarding the first input image corresponding to the second input image to be calculated.
- the motion data for the first input image indicates a change component between the preceding first input image and the first input image to be calculated.
- the step of calculating the second predicted image includes the step of calculating the second predicted image using the second input image to be calculated and the one or more second input images in advance. Including.
- a storage medium for storing a program for encoding a plurality of input images stores a plurality of first input images including first information of a subject and second information different from the first information of the subject corresponding to the plurality of first input images, respectively.
- a step of specifying an area in which the pixel value is to be defined by a remainder among pixels constituting the difference image, and a pixel value included in the area that is to be defined by the remainder of the first residual image is converted to a remainder
- a step of generating a difference image a step of identifying a region in which the pixel value is to be defined by a remainder among pixels constituting the second residual image, based on the pixel value of the second residual image;
- an apparatus for encoding a plurality of input images includes a plurality of first input images including first information of a subject, and a plurality of second information different from the first information of the subject corresponding to the plurality of first input images, respectively.
- a first residual image based on a difference between the first input image and the corresponding first predicted image and a pixel value of the first residual image; Means for identifying an area in which the pixel value is to be defined by a remainder among the constituent pixels, means for converting a pixel value included in an area that is to be defined by the remainder of the first residual image, to a remainder, Regarding the second input image, there are few other second input images and first input images.
- FIG. 1 is a diagram showing a stereoscopic video reproduction system including an encoding / decoding system according to an embodiment of the present invention. It is a functional block diagram of the encoder which concerns on the related technique of this invention. It is a functional block diagram of the decoder which concerns on the related technique of this invention. It is a functional block diagram which shows the structure aimed at the encoding with respect to the multiview video which concerns on the related technique of this invention.
- FIG. 10 is a functional block diagram illustrating a configuration directed to decoding multi-view video according to a related technique of the present invention. It is a functional block diagram of the encoder group which concerns on embodiment of this invention.
- FIG. 1 is a diagram showing a stereoscopic video reproduction system 1 including an encoding / decoding system according to an embodiment of the present invention.
- a multi-viewpoint video is generated by imaging a subject 2 using a camera array including a plurality of cameras 10.
- a multi-view video corresponds to a group of images obtained by imaging the subject 2 from a plurality of viewpoints.
- This multi-view video is transmitted after being encoded by the information processing apparatus 100 functioning as a transmitter.
- the data generated by encoding in the information processing apparatus 200 functioning as a receiver is decoded, and the subject 2 is reproduced in the stereoscopic display apparatus 300. That is, the stereoscopic display device 300 displays a stereoscopic image of the subject 2.
- any medium can be used regardless of wired or wireless.
- encoding is performed on different types of image groups related to each other.
- a plurality of images and a plurality of depth information are generated from multi-view images acquired by the camera array, and encoding is performed on each of the images.
- the video includes luminance information or color information (that is, shading information of each color component) at each viewpoint, and the depth information includes information on the distance (depth) from the captured viewpoint to each point in the image.
- the video includes a gray image (a gray value map) defined separately for each color component, and the depth information is a gray image (a gray value map) in which the distance at each pixel position is defined as a pixel value. )including.
- the information processing apparatus 100 that functions as a transmitter includes a preprocessor 110 that performs preprocessing on an input image, an encoder 120 that performs encoding on video, and an encoder 140 that performs encoding on depth information.
- the preprocessing executed by the preprocessor 110 includes processing for generating depth information from the video signal.
- the encoder 120 and the encoder 140 perform encoding by sharing information with each other. By adopting such a mechanism capable of sharing information between encoders, more efficient compression processing is realized by utilizing the relationship (that is, redundancy) between images.
- Each encoding executed in the information processing apparatus 100 includes data format conversion and data compression processing, as will be described later. That is, the encoder according to the embodiment of the present invention performs data format conversion and data compression in parallel.
- the information processing apparatus 200 functioning as a receiver includes decoders 210 and 230 that perform decoding on received data, and a post processor 240 that performs post-processing.
- the decoder 210 performs decoding on the data regarding the video included in the received data
- the decoder 230 performs decoding on the data regarding the depth information included in the received data.
- the decoder 210 and the decoder 230 share information with each other and perform decoding.
- the post processor 240 generates a signal for the stereoscopic display device 300 to reproduce the subject 2 separately from the projector array 302 by executing predetermined processing on the decoding results obtained by the decoders 210 and 230.
- Each decoding executed in the information processing apparatus 200 includes data format reverse conversion and data restoration processing, as will be described later. That is, the decoder according to the embodiment of the present invention performs data format inverse conversion and data restoration in parallel.
- the stereoscopic display device 300 includes a display screen 310 mainly including a diffusion film 312 and a condenser lens 314, and a projector array 302 that projects a multi-viewpoint video on the display screen 310.
- Each projector constituting the projector array 302 projects a corresponding viewpoint image of the multi-viewpoint video output from the information processing apparatus 200 onto the display screen 310.
- a reproduced stereoscopic image of the subject 2 is provided to an observer in front of the display screen 310.
- the viewpoint image entering the visual field of the observer changes according to the relative position between the display screen 310 and the observer, and the observer is as if in front of the subject 2.
- Such a stereoscopic video reproduction system 1 is used in a movie theater or an amusement facility as a general use, and is used as an electronic advertisement system such as a telemedicine system, an industrial design design system, or public viewing as an industrial use. It is expected that
- FIG. 2 is a functional block diagram of the encoder 820 according to the related technology of the present invention.
- FIG. 3 is a functional block diagram of the decoder 910 according to the related art of the present invention.
- each frame of the video signal that is a moving image (that is, a frame sequence arranged in the time domain) from the input source is divided into a plurality of macroblocks (Macroblocks).
- Intraframe prediction is a method of interpolating a target macroblock from other macroblocks in the same frame.
- interframe prediction is a method of interpolating a target macroblock from information of another frame using any one of forward prediction, backward prediction, and bidirectional prediction.
- the encoder 820 performs data compression by paying attention to the relationship (that is, redundancy) with the information of the same or similar frames.
- the encoder 820 includes an input buffer 8202, a division unit 8204, a subtraction unit 8206, an orthogonal transformation / quantization unit 8208, a local decoder 8210, a control unit 8230, a motion estimation unit 8240, An output buffer 8242 and an entropy encoding unit 8250 are included.
- the input buffer 8202 temporarily stores the video signal from the input source.
- the dividing unit 8204 divides the video signal stored in the input buffer 8202 into a plurality of macro blocks (N ⁇ N pixels). The output from the dividing unit 8204 is given to the subtracting unit 8206, the control unit 8230, and the motion estimation unit 8240.
- the subtraction unit 8206 calculates residual information by subtracting the previously calculated interpolation information (intraframe prediction or interframe prediction) for each macroblock from the division unit 8204. That is, the subtraction unit 8206 generates a residual image by subtracting the predicted image from the original image. This residual image generation processing is typically executed in units of macroblocks.
- the orthogonal transform / quantization unit 8208 performs orthogonal transform (typically, discrete Fourier transform) and quantization on the residual image from the subtraction unit 8206.
- the orthogonal transform / quantization unit 8208 also performs scaling.
- the quantized transform coefficient from orthogonal transform / quantization unit 8208 is output to local decoder 8210 and entropy coding unit 8250.
- the local decoder 8210 calculates interpolation information for the subsequent frame (the macroblock). More specifically, local decoder 8210 includes an inverse orthogonal transform / scaling unit 8212, an adding unit 8214, a deblocking filter 8216, an intra frame prediction unit 8218, a motion compensation unit 8220, and a switching unit 8222. .
- the inverse orthogonal transform / scaling unit 8212 performs inverse orthogonal transform and scaling on the transform coefficient after quantization from the orthogonal transform / quantization unit 8208. That is, the inverse orthogonal transform / scaling unit 8212 restores the residual image output from the subtracting unit 8206.
- the adding unit 8214 adds the residual image from the inverse orthogonal transform / scaling unit 8212 and the predicted image (interpolation information) calculated in advance.
- the deblocking filter 8216 smoothes the block boundary with respect to the addition result from the adding unit 8214 in order to suppress the occurrence of block noise.
- the original image given from the input buffer 8202 is restored by the inverse orthogonal transform / scaling unit 8212, the adding unit 8214, and the deblocking filter 8216. Then, the information of the restored original image is given to the intra frame prediction unit 8218 and the motion compensation unit 8220.
- the intra frame prediction unit 8218 generates a prediction image based on adjacent macroblocks.
- the motion compensation unit 8220 generates a predicted image using inter-frame prediction (inter-frame prediction). More specifically, the motion compensation unit 8220 generates a predicted image based on the restored original image and the motion data from the motion estimation unit 8240.
- One of the predicted images generated by each of the intra frame prediction unit 8218 and the motion compensation unit 8220 is appropriately selected by the switching unit 8222, and is supplied to the subtraction unit 8206.
- the motion estimation unit 8240 calculates motion data (typically a motion vector) based on each macro block from the division unit 8204 and information on the restored original image one frame before.
- the calculated motion data is output to the motion compensation unit 8220 and the entropy encoding unit 8250.
- the control unit 8230 controls processing in the orthogonal transform / quantization unit 8208, the inverse orthogonal transform / scaling unit 8212, the switching unit 8222, and the motion estimation unit 8240. In addition, the control unit 8230 instructs, as control data, parameters related to encoding, an encoding order for each component, and the like.
- the entropy encoding unit 8250 performs entropy encoding on the transform coefficient after quantization from the orthogonal transform / quantization unit 8208, the motion data from the motion estimation unit 8240, and the control data from the control unit 8230. As a result, a bit stream is output. This output bit stream is the encoding result for the input video signal.
- the output buffer 8242 is not an essential component, but temporarily stores the original image (video) restored from the deblock filter 8216.
- the decoder 910 shown in FIG. 3 the original image is restored from the bit stream from the encoder 820 shown in FIG.
- the encoding is inversely converted by the encoder 820 shown in FIG.
- the decoder 910 includes an input buffer 9102, an entropy decoding unit 9104, an inverse orthogonal transform / scaling unit 9112, an adding unit 9114, a deblocking filter 9116, an intra frame prediction unit 9118, and motion compensation.
- the input buffer 9102 temporarily stores the bit stream from the encoder 820.
- the entropy decoding unit 9104 performs entropy decoding on the bit stream from the input buffer 9102, and outputs motion data, quantized transform coefficients, and control data as a result.
- the inverse orthogonal transform / scaling unit 9112 performs inverse orthogonal transform (typically, discrete Fourier inverse transform) and scaling for the quantized transform coefficient decoded by the entropy decoding unit 9104. By these processes, the residual image is restored.
- inverse orthogonal transform typically, discrete Fourier inverse transform
- the addition unit 9114 adds the residual image from the inverse orthogonal transform / scaling unit 9112 and the previously calculated prediction image (interpolation information).
- the deblocking filter 9116 smoothes the block boundary with respect to the addition result from the adding unit 9114 in order to suppress the occurrence of block noise.
- the intra frame prediction unit 9118 generates a prediction image based on adjacent macroblocks.
- the motion compensation unit 9120 generates a predicted image using inter-frame prediction (inter-frame prediction). More specifically, the motion compensation unit 9120 generates a prediction image based on the restored original image and the motion data decoded by the entropy decoding unit 9104.
- One of the predicted images generated by each of the intra frame prediction unit 9118 and the motion compensation unit 9120 is appropriately selected by the switching unit 9122 and is supplied to the addition unit 9114.
- Control unit 9130 controls processing in inverse orthogonal transform / scaling unit 9112 and switching unit 9122 based on the control data decoded by entropy decoding unit 9104.
- the output buffer 9142 temporarily stores the restored original image (video) from the deblock filter 9116.
- MPEG-4 AVC which is one of the compression standards for moving images
- moving images are transmitted in a compressed state by the encoding / decoding system as described above.
- FIG. 4 is a functional block diagram showing a configuration directed to encoding for a multi-view video according to the related art of the present invention.
- FIG. 5 is a functional block diagram illustrating a configuration directed to decoding multi-view video according to the related art of the present invention.
- FIG. 4 shows a configuration in which multi-view video and multi-view depth information are encoded in an integrated manner. According to this scheme, it is possible to share information between encoders, thereby further improving the encoding efficiency.
- the encoder 820 has substantially the same configuration as the encoder 820 shown in FIG. However, the encoder 820 shown in FIG. 4 encodes the multi-view video.
- the encoder 840 has the same configuration as the encoder 820. However, the encoder 840 encodes the multi-view depth information.
- the encoder 840 includes an input buffer 8402, a division unit 8404, a subtraction unit 8406, an orthogonal transformation / quantization unit 8408, a local decoder 8410, a control unit 8430, a motion estimation unit 8440, and an output.
- a buffer 8442 and an entropy encoding unit 8450 are included.
- Local decoder 8410 includes an inverse orthogonal transform / scaling unit 8412, an adding unit 8414, a deblocking filter 8416, an intra frame prediction unit 8418, a motion compensation unit 8420, and a switching unit 8422.
- FIG. 5 when restoring MVD from a bitstream obtained by encoding MVD, a configuration as shown in FIG. 5 is used.
- the configuration shown in FIG. 5 includes two decoders 910 and 930 corresponding to the two encoders 820 and 840 shown in FIG. Also in decoding, the decoder 910 and the decoder 930 are linked to each other.
- Non-Patent Document 3 a technique for sharing motion estimation information (typically, a motion vector) as disclosed in Non-Patent Document 3 has been proposed between encoders. This is because the video and depth information are acquired by imaging the same subject, and thus both are synchronized.
- the encoding / decoding system typically includes a plurality of first input images (multi-view video / multi-point) including first information (luminance information) of a subject such as MVD. Viewpoint images) and a plurality of second input images (multi-view depth information / multiple) including second information (depth information) different from the first information of the subject respectively corresponding to the plurality of first input images.
- a plurality of input images such as a viewpoint depth map).
- the present invention is not limited to MVD, and encoding and decoding can be performed on a plurality of types of input image groups (typically a pair of input image groups) that can share the information with each other. ing. Therefore, the present invention is not limited to a combination of video and depth information, and can be applied to an image group including a combination of other information.
- data format conversion is performed in each of the encoder for video and the encoder for depth information. Is incorporated.
- information on the data format type is transmitted from the encoder group to the decoder group.
- the encoding / decoding system according to the embodiment of the present invention includes a data format conversion process that can be incorporated into the existing standard as described above.
- the concept of remainder is introduced to further increase the data compression efficiency.
- each pixel value is defined by a residual corresponding to the difference between the original image and the predicted image.
- a data format in which each pixel value is defined by “remainder” is adopted.
- This remainder is defined as a remainder (integer value) obtained by dividing a calculated value by a predetermined integer value. At this time, the quotient is also an integer. More specifically, the remainder is calculated by a modulo operation. The remainder calculation procedure will be described in detail later.
- a data format defined only by a residue instead of a residual or a data format defined by combining a residue and a residual can be adopted.
- the embodiment of the present invention not only the residual used in the existing standard but also the remainder can be used to increase the efficiency of data compression and improve the quality.
- the encoding / decoding system can further improve the data compression efficiency by sharing the motion data and depth information about the video.
- FIG. 6 is a functional block diagram of the encoder group according to the embodiment of the present invention.
- multi-view images constituting MVD and corresponding multi-view depth information are given as input data.
- the encoder 120 illustrated in FIG. 1 encodes a multi-view video
- the encoder 140 encodes multi-view depth information.
- the encoders 120 and 140 perform encoding while sharing information with each other.
- the basic configuration is common between the encoder 120 and the encoder 140.
- the encoder 120 for encoding multi-view video includes an input buffer 1202, a division unit 1204, a data format conversion unit 1206, an orthogonal transformation / quantization unit 1208, a local decoder 1210, a control unit 1230, and motion estimation.
- the local decoder 1210 includes an inverse orthogonal transform / scaling unit 1212, a data format inverse transform unit 1214, a deblock filter 1216, an intra frame prediction unit 1218, a motion compensation unit 1220, and a switching unit 1222.
- the encoder 140 for encoding the multi-view depth information includes an input buffer 1402, a division unit 1404, a data format conversion unit 1406, an orthogonal transformation / quantization unit 1408, a local decoder 1410, and a control unit 1430.
- the local decoder 1410 includes an inverse orthogonal transform / scaling unit 1412, a data format inverse transform unit 1414, a deblock filter 1416, an intra frame prediction unit 1418, a motion compensation unit 1420, and a switching unit 1422.
- the encoder 120 is provided with a data format conversion unit 1206 instead of the subtraction unit 8206 for generating a residual image, as compared with the encoder 820 shown in FIGS. 2 and 4, and performs addition for restoring the original image.
- a data format reverse conversion unit 1214 is provided instead of the unit 8214.
- the encoder 140 is provided with a data format conversion unit 1406 instead of the subtraction unit 8406 that generates a residual image, compared to the encoder 840 shown in FIG. 4, and an addition unit 8414 for restoring the original image.
- the data format inverse conversion unit 1414 is mainly different.
- the operations of the control units 1230 and 1430 are also different from those of the control units 8230 and 8430, respectively.
- the processing in the motion estimation units 1240 and 1440 is also different from that of the motion estimation units 8240 and 8440, respectively.
- the functions of the input buffers 1202 and 1402, the division units 1204 and 1404, the orthogonal transform / quantization units 1208 and 1408, the motion estimation units 1240 and 1240, the output buffers 1242 and 1442, and the entropy encoding units 1250 and 1450 are as follows:
- the functions of the inverse orthogonal transform / scaling units 1212 and 1412, the deblock filters 1216 and 1416, the intra frame prediction units 1218 and 1418, and the switching units 1222 and 1422 of the local decoders 1210 and 1410 are the same as those of the local decoder shown in FIG. 8210 and 8410 are inverse orthogonal transform / scaling units 8212 and 8412, deblock filters 8216 and 8416, intra-frame prediction units 8218 and 8418, and switching units 8222 and 8422, respectively.
- the video signal from the input source is applied to input buffer 1202, and the corresponding depth information is applied to input buffer 1402.
- multi-view images captured by a plurality of cameras 10 are input as images, and corresponding multi-view depth information is input as depth information.
- MVD multi-view images captured by a plurality of cameras 10
- corresponding multi-view depth information is input as depth information.
- it is not limited to such an MVD, and may be a single viewpoint video imaged by a single camera 10 and corresponding depth information.
- the input buffer 1202 temporarily stores these video signals, and all or a part of them is given to the dividing unit 1204 as input data.
- the depth information is temporarily stored in the input buffer 1402, and all or part of the depth information is provided to the dividing unit 1404 as input data.
- the dividing unit 1204 divides each frame included in the video signal output from the input buffer 1202 into a plurality of macro blocks (N ⁇ N pixels). Similarly, the dividing unit 1404 divides each frame included in the depth information output from the input buffer 1402 into a plurality of macro blocks (N ⁇ N pixels). This speeds up the prediction process by using an appropriate image size as a processing unit. However, one frame may be processed as it is without being divided into macroblocks in consideration of the computing capability of the information processing apparatus and the required processing time. Each divided macroblock is supplied to data format conversion units 1206 and 1406, respectively.
- the data format conversion unit 1206 performs data format conversion using the macroblock from the division unit 1204 and the motion compensation macroblock from the intra frame prediction unit 1218 or the motion compensation unit 1220.
- the data format conversion unit 1406 performs data format conversion using the macroblock from the division unit 1404 and the motion compensation macroblock from the intra frame prediction unit 1418 or the motion compensation unit 1420.
- the motion compensation macroblock corresponds to a motion image indicating a change component included in the subsequent input image from one or more previous input images
- the intra frame prediction unit 1218 or the motion compensation unit 1220 includes: This motion image is estimated for the video.
- the data format conversion unit 1206 generates a residual image from the difference between the subsequent input image and the estimated motion image. Then, based on the pixel value of the residual image, the data format conversion unit 1206 identifies an area in which the pixel value is to be defined with a remainder among the pixels constituting the residual image.
- the data format conversion unit 1206 converts the pixel value for the area to be defined by the specified remainder into a remainder. By such a procedure, the converted residual image is output as an image after data format conversion.
- the data format conversion unit 1406 performs the same process on the depth information.
- the corresponding motion compensation macroblock provided from the intra frame prediction unit 1218 or the motion compensation unit 1220 is used as side information for reconstructing the original macroblock from the macroblock generated by the data format conversion unit 1206. Is done.
- the corresponding motion compensation macroblock given from the intra frame prediction unit 1418 or the motion compensation unit 1420 is used as side information for reconstructing the original macroblock from the macroblock generated by the data format conversion unit 1406.
- the macroblock after the data format conversion for the video is given to the orthogonal transform / quantization unit 1208.
- the orthogonal transform / quantization unit 1208 further optimizes the input macroblock after data format conversion by executing orthogonal transform, quantization, and scaling. Typically, discrete Fourier transform is adopted as the orthogonal transform.
- the quantization table used in quantization and the scaling coefficient used in scaling may be optimized according to the data format type (type) indicating the type of data format conversion in the data format conversion unit 1206.
- the data format conversion unit 1206 can perform several data format conversions, and an example of the types of these data format conversions will be described in detail later.
- the transformed transform coefficient from the orthogonal transform / quantization unit 1208 is output to the local decoder 1210 (inverse orthogonal transform / scaling unit 1212) and the entropy coding unit 1250.
- the orthogonal transform / quantization unit 1408 performs orthogonal transform, quantization, and scaling on the macroblock after the data format conversion for the depth information.
- the inverse orthogonal transform / scaling unit 1212 performs inverse orthogonal transform and scaling on the transform coefficient after quantization of the video from the orthogonal transform / quantization unit 1208. That is, the inverse orthogonal transform / scaling unit 1212 performs a process reverse to the transform process in the orthogonal transform / quantization unit 1208 to restore the macroblock after the data format conversion. Further, the data format inverse conversion unit 1214 performs data format inverse conversion on the restored macro block after the data format conversion, and restores each divided macro block. Similarly, the inverse orthogonal transform / scaling unit 1412 performs inverse orthogonal transform and scaling on the quantized transform coefficient for the depth information from the orthogonal transform / quantization unit 1408.
- Deblock filters 1216 and 1416 smooth the block boundaries in order to suppress the occurrence of block noise for the restored macroblocks from the data format inverse conversion units 1214 and 1414, respectively.
- the original video is restored by the inverse orthogonal transform / scaling unit 1212, the data format inverse transform unit 1214, and the deblock filter 1216. Then, the restored original image is provided to the intra frame prediction unit 1218 and the motion compensation unit 1220. Similarly, the original depth information is restored by the inverse orthogonal transform / scaling unit 1412, the data format inverse transform unit 1414, and the deblock filter 1416.
- the intra frame prediction unit 1218 generates a prediction image (hereinafter also referred to as “intra macro block”) based on adjacent macro blocks.
- the motion compensation unit 1220 generates a predicted image (hereinafter also referred to as “inter macroblock”) using interframe prediction (interframe prediction). These predicted images become motion compensation macroblocks.
- a connection is provided between the encoder 120 and the encoder 140 that indicates what type of information is shared. Parameters for notifying each other of such shared information will be described later.
- the motion estimator 1240 estimates motion data for the video, and the motion estimator 1440 estimates motion data for the depth information. As these motion data, typically, motion vectors are used.
- the motion estimation unit 1240 basically estimates motion data for a video based on the original video divided into the macroblocks from the division unit 1204 and the restored original video one frame before. In order to improve the estimation accuracy of the motion data for this video, the corresponding depth information is used. More specifically, the motion estimation unit 1240 uses the depth information from the encoder 140 in the same frame as the processing target frame for estimation of motion data (typically in the spatial direction).
- the motion estimation unit 1440 basically performs motion data on depth information based on the depth information divided into the macroblocks from the division unit 1404 and the restored depth information one frame before. Is estimated. In order to improve the estimation accuracy of the motion data (in the spatial direction and / or the temporal direction) for the depth information in the motion estimation unit 1440, the motion data estimated in the encoder 120 is used.
- the control unit 1230 controls processing in the data format conversion unit 1206, the orthogonal transformation / quantization unit 1208, the inverse orthogonal transformation / scaling unit 1212, the data format inverse transformation unit 1214, the switching unit 1222, and the motion estimation unit 1240. Also, the control unit 1230 outputs, as control data, parameters related to encoding, the encoding order for each component, and the like. Further, the control unit 1230 outputs additional information (data format type (type), threshold value, flag, etc.) related to the data format conversion to the entropy encoding unit 1250.
- data format type type
- threshold value threshold value
- flag flag
- control unit 1430 performs processing in the data format conversion unit 1406, the orthogonal transformation / quantization unit 1408, the inverse orthogonal transformation / scaling unit 1412, the data format inverse transformation unit 1414, the switching unit 1422, and the motion estimation unit 1440. Control. Also, the control unit 1430 outputs, as control data, parameters related to encoding, the encoding order for each component, and the like. Further, the control unit 1430 outputs additional information (data format type (type), threshold value, flag, etc.) related to the data format conversion to the entropy encoding unit 1450.
- data format type type
- threshold value threshold value
- flag flag
- control unit 1230 and the control unit 1430 exchange some control data in order to share information as described above. As a result, integrated encoding for MVD can be realized.
- the entropy encoding unit 1250 codes the converted residual image and additional information for specifying an area to be defined by the remainder. More specifically, the entropy encoding unit 1250 converts the quantized transform coefficient from the orthogonal transform / quantization unit 1208, the motion data from the motion estimation unit 1240, and the control data and additional information from the control unit 1230. On the other hand, entropy coding is performed, and as a result, a bit stream for the video is generated. This generated bit stream is the encoding result for the input video signal.
- the entropy encoding unit 1450 performs the transform coefficient after quantization from the orthogonal transform / quantization unit 1408, the motion data from the motion estimation unit 1240, and the control data and additional information from the control unit 1430. Entropy encoding is performed, and as a result, a bit stream of depth information is output.
- the output buffer 1242 is not an essential component, the restored original video from the deblock filter 1216 is temporarily stored. Similarly, the output buffer 1442 temporarily stores the restored original depth information from the deblock filter 1416.
- the multi-view depth information is used in the estimation of the motion data for the video (multi-view video) in the motion estimation unit 1240.
- this depth information the following methods are typically considered.
- the difference between frames acquired at the same time depends on the difference in the corresponding viewpoint (that is, the difference in the spatial domain).
- This difference in the spatial region is based on the knowledge that the corresponding depth information can be used as it is as the motion data for the spatial direction of the video because it has a strong relationship with the corresponding depth information. By using such depth information, processing efficiency and accuracy can be improved.
- the motion data for depth information itself may be used as motion data for video (multi-viewpoint video).
- the motion data for the corresponding depth information may be handled as an initial value of the estimated motion data of the video, and further adjusted using the video, and then output as final motion data.
- a multi-view video is used in motion data estimation for depth information (multi-view depth information) in the motion estimation unit 1440.
- the following two methods are conceivable as usage forms of the multi-view video.
- the motion data itself for the corresponding video is output from the motion estimation unit 1440 as motion data.
- the motion data for the corresponding video is treated as the initial value of the motion information of the estimated depth information, and further adjusted using the depth information, and then output as final motion data.
- motion data for the corresponding video is used for encoding in the encoder 140.
- the motion data about the video may not be used for encoding the depth information.
- the following two methods are conceivable.
- the encoder 140 generates motion data from the depth information without using motion data about the video, and uses the generated motion data for coding and data compression.
- the encoder 140 treats the depth information itself as motion data without using motion data for the video.
- G3 Prediction image generation example
- FIG. 7 is a diagram showing an example of a procedure for generating a predicted image by encoding according to the embodiment of the present invention.
- FIG. 7A shows an example of a predicted image generation procedure for multi-view video
- FIG. 7B shows an example of a predicted image generation procedure for multi-view depth information.
- FIG. 7 shows an example in which each of the plurality of cameras 10 arranged at the arrangement positions S0, S1, S2, S3,... Sequentially outputs frames at times T0, T1, T2, T3, T4, T5,. .
- the predicted image for each frame is generated using intra-frame prediction or inter-frame prediction.
- FIG. 7A shows an example of a predicted image generation procedure for multi-view video
- FIG. 7B shows an example of a predicted image generation procedure for multi-view depth information.
- FIG. 7 shows an example in which each of the plurality of cameras 10 arranged at the arrangement positions S0, S1, S2, S3,... Sequentially outputs frames at times T0, T1, T2, T3, T4,
- FIG. 7 illustrates the generation procedure in units of frames. However, as described above, a predicted image may be generated in units of macroblocks.
- a predicted image (I frame) is generated using intra-frame prediction instead of inter-frame prediction (inter-frame prediction). Generated. Using this I frame, predicted images are sequentially generated according to a predetermined generation order.
- motion data of the corresponding frame is generated. Reflected in the predicted image.
- the motion data (time T0 and placement position S2) for the corresponding video is reflected on the frame (frame at time T0 of the camera 10 at the placement position S2) that is the target of interframe prediction.
- the depth information itself may be used as motion data for each corresponding macroblock of the video.
- the amount of texture of a video macroblock may be determined by applying a threshold to the pseudo-gradient macroblock.
- the information of the lacking area can be generated by estimating by using the depth information or the motion data of the depth information as an initial value. Or you may produce
- FIG. 8 is a diagram for explaining a method of combining the residue and the residual according to the embodiment of the present invention.
- FIG. 8A shows a method of combining a residue and a residual in units of pixels
- FIG. 8B shows a method of combining a residue and a residual in units of macroblocks.
- “Rem” indicates a remainder
- “Res” indicates a residual.
- each frame is divided into a plurality of macro blocks and processed.
- a predetermined determination criterion typically a threshold value TH1 as will be described later
- TH1 a threshold value
- a predetermined judgment criterion (typically thresholds TH1 and TH2 as will be described later) is applied to each of a plurality of macroblocks constituting a frame, and a remainder is obtained. It is determined whether to use (residual macroblock) or residual (residual macroblock). For a pixel or macroblock that is determined to be defined by the remainder, the pixel value is calculated using a modulo operation as described later.
- the data format conversion unit 1206 calculates the difference between the original macroblock and the motion compensation macroblock (the intra macroblock generated by the intraframe prediction unit 1218 or the inter macroblock generated by the motion compensation unit 1220) in the same frame. Data format conversion is executed for (that is, the residual image). For the area defined by the remainder, the motion compensation macroblock is also used as side information.
- the pseudo gradient macroblock about a motion compensation macroblock (intra macroblock or an inter macroblock), or the macroblock which has the information similar to it Is generated.
- the gradient information may be calculated in units of frames.
- first data format a data format that combines the residue and the residue in pixel units
- second data a data format
- first data format a data format that combines the residue and the residue in pixel units
- second data a data format that combines the residue and the residue in macroblock units.
- FIG. 9 is a functional block diagram of the data format conversion unit 1206 according to the embodiment of the present invention.
- the data format conversion unit 1206 includes a subtraction unit 1260, a comparison unit 1262, a mask generation unit 1264, a process selection unit 1266, a gradient image generation unit 1270, a coefficient selection unit 1272, a Lookup.
- a table 1274, a modulo operation unit 1278, and a synthesis unit 1280 are included.
- the subtracting unit 1260 performs a motion compensation macro block (intra macro block or inter macro block) from an original macro block (indicated as “Original MB” in FIG. 9) input from the dividing unit 1204 (FIG. 6). ) (Indicated as “Inter / Intra MB” in FIG. 9) is subtracted to generate a residual macroblock (indicated as “Res MB” in FIG. 9).
- the comparison unit 1262 and the mask generation unit 1264 identify the pixels defined by the residual in the target macroblock. That is, the comparison unit 1262 determines an area to be defined by the remainder on a pixel basis based on the size of the pixel value of each pixel constituting the residual image (residual macroblock).
- the mask generation unit 1264 outputs information for specifying each pixel defined by the remainder among the pixels constituting the residual image as additional information (typically, flag flag1 described later).
- the comparison unit 1262 compares the pixel value of each pixel constituting the target macroblock with the threshold value TH1 that is part of the side information.
- the mask generation unit 1264 determines that a pixel whose pixel value is less than the threshold value TH1 should be defined as a remainder, and determines that other pixels should be defined as a residual. That is, in the residual macroblock, information of an area having a small pixel value may be largely lost, so that data compression is performed after conversion to a data format defined by a remainder instead of a residual.
- the mask generation unit 1264 generates a mask (map) in which the value of the flag flag1 for each pixel is expanded in the target frame, outputs the mask (map) to the process selection unit 1266, and outputs it to the control unit 1230. Based on the value of the flag flag1 from the mask generation unit 1264, a procedure to be applied to each pixel is determined in encoding and decoding.
- the process selection unit 1266 selects a process for each pixel constituting the target macroblock based on the value of the flag flag1. Specifically, the process selection unit 1266 outputs the pixel value of the pixel determined to be defined by the residual (indicated as “Residual” in FIG. 9) to the synthesis unit 1280 as it is. On the other hand, for the pixel determined to be defined by the remainder (indicated as “Remainder” in FIG. 9), the pixel value is output to the modulo arithmetic unit 1278.
- the modulo operation unit 1278 performs a modulo operation on the pixel value for the area to be defined by the remainder. More specifically, the modulo calculation unit 1278 performs a modulo calculation using the coefficient D (integer) set by the coefficient selection unit 1272 as a denominator to calculate a remainder. The calculated remainder is output to the synthesis unit 1280. The combining unit 1280 combines the remainder or residual input for each pixel, and outputs a macroblock after data format conversion (indicated as “Converted MB” in FIG. 9).
- the data format conversion unit 1206 may dynamically change the coefficient (denominator) D used for the modulo calculation in the modulo calculation unit 1278 based on the motion compensation macroblock.
- a region having a large pixel value in a motion compensation macroblock means a region having relatively low redundancy between frames, and the information contained in such a region is maintained even after data format conversion. It is preferable. Therefore, an appropriate coefficient D is selected according to the degree of redundancy between frames.
- FIG. 9 illustrates a processing example in which gradient information of a motion compensation macroblock (motion image) is acquired and a value that is a modulo arithmetic method is determined based on the acquired gradient information. More specifically, a pseudo-gradient macroblock (gradient-like macro-block) is generated for the motion compensation macroblock, and a modulus that is modulo depending on the size of the pixel value in each pixel of the pseudogradient macroblock. D is determined.
- a pseudo-gradient macroblock gradient-like macro-block
- the gradient image generation unit 1270 generates a pseudo gradient macroblock for the motion compensation macroblock. Then, a value that becomes the modulus of the modulo calculation may be determined with reference to a predetermined correspondence relationship between the gradient strength and the value that becomes the modulus of the modulo calculation. More specifically, the coefficient selection unit 1272 determines the coefficient D for each pixel by referring to the Lookup table 1274 based on the pixel value (gradient strength) of each pixel of the generated pseudo gradient macroblock. To do. By using the Lookup table 1274, the coefficient D can be determined nonlinearly with respect to the pseudo gradient macroblock. Thus, by determining the coefficient D nonlinearly, the image quality after decoding can be improved.
- FIG. 10 is a diagram showing an example of the Lookup table 1274 for determining the coefficient D used for calculating the remainder according to the embodiment of the present invention. As shown in FIG. 10, it is discretized into a plurality of stages (Gradient Range) according to the gradient strength, and a coefficient D for each stage is defined.
- the gradient image generation unit 1270 refers to the Lookup table 1274 and selects a coefficient D corresponding to each pixel of the target macroblock. Here, the coefficient D is determined for each pixel of each color component included in the target macroblock.
- the Lookup table 1274 shown in FIG. 10 is designed such that a value (coefficient D) used as a modulo arithmetic method is a power of two. By assigning the coefficient D in this way, the modulo operation can be speeded up. Since the Lookup table 1274 can be designed arbitrarily, a Lookup table with a smaller number of stages or a larger number of stages may be adopted.
- the Lookup table is not necessarily used, and the coefficient D may be determined using a predetermined function or the like.
- the pixel value in each pixel of the pseudo gradient macroblock may be used as the coefficient D as it is.
- q is a quotient and m is a remainder.
- the gradient image generation unit 1270 generates a pseudo gradient macroblock indicating the degree of change in the image space from a motion compensation macroblock (intra macroblock or inter macroblock) as side information.
- the pseudo gradient macroblock means an image in which a region having a larger texture change has a higher luminance in the motion compensation macroblock.
- Arbitrary filtering processing can be used as generation processing of the pseudo gradient macroblock.
- Each pixel value constituting the pseudo gradient macroblock is normalized so as to take any integer value within a predetermined range (for example, 0 to 255).
- a pseudo gradient macroblock is generated by the following processing procedure.
- the processing procedure shown here is an example, and the processing content and processing procedure of Gaussian smoothing processing and morphological processing can be designed as appropriate.
- any method may be adopted as long as a macroblock that assigns a larger pixel value (luminance) to a region where a larger luminance change occurs in the motion compensation macroblock can be generated.
- a sobel filter may be applied to each of the x direction and the y direction, and the average value of the application results may be a macroblock.
- FIG. 11 is another functional block diagram of data format conversion unit 1206 according to the embodiment of the present invention.
- data format conversion unit 1206 is different from data format conversion unit 1206 shown in FIG. 9 in place of mask generation unit 1264, process selection unit 1266, and synthesis unit 1280, integrating unit 1265, An evaluation unit 1267 and a switching unit 1269 are provided. Since the details of the other components have been described above, their contents will not be repeated.
- the comparison unit 1262, the integration unit 1265, and the evaluation unit 1267 determine which of the residual and the remainder should be defined for the target macroblock. That is, the comparison unit 1262, the integration unit 1265, and the evaluation unit 1267 evaluate the pixel value of each pixel constituting the block for each block obtained by dividing the residual image (residual macroblock) by a predetermined size. Based on the combined result, the area to be defined by the remainder is determined in block units.
- the evaluation unit 1267 outputs, as additional information, information for specifying a block defined by a remainder among blocks included in the residual image.
- the comparison unit 1262 compares the pixel value of each pixel constituting the residual macroblock with the threshold value TH1 that is part of the side information. Then, for the pixel whose pixel value exceeds the threshold value TH1, the comparison unit 1262 outputs the difference between the pixel value and the threshold value TH1 to the integrating unit 1265. That is, for each residual macroblock, the accumulating unit 1265 is the sum of “pixel value ⁇ threshold value TH1” ( ⁇ (pixel value ⁇ threshold value TH1)) for pixels whose pixel value exceeds the threshold value TH1. Is calculated.
- the evaluation unit 1267 compares the calculated sum with the threshold value TH2, and determines whether the target residual macroblock should be defined as a residual or a remainder. Specifically, if the calculated sum is equal to or greater than threshold value TH2, evaluator 1267 determines to output the target residual macroblock as it is. On the other hand, if the calculated sum is less than the threshold value TH2, the evaluation unit 1267 determines to output the target residual macroblock after converting it into a residual macroblock. In other words, when it is determined that the residual macroblock is composed of pixels having relatively small pixel values, the information of the macroblock may be largely lost, so not the residual but the remainder. Converted to the defined data format.
- the evaluation unit 1267 gives a command to the switching unit 1269 based on this determination. More specifically, when it is determined to output the target residual macroblock as it is, the switching unit 1269 activates a path that bypasses the modulo arithmetic unit 1278. On the other hand, when it is determined that the target residual macroblock is converted into a remainder macroblock and then output, the switching unit 1269 validates the path for supplying the residual macroblock to the modulo arithmetic unit 1278. .
- Additional information on whether the macro block is defined as a residue or a residual is included in the side information as a flag flag2. Based on the value of the flag flag2 from the mask generation unit 1264, a procedure to be applied to each macroblock is determined in encoding and decoding.
- a deblock filter 1216 (FIG. 4) is used. ) May be bypassed. Thereby, the generation of noise can be reduced.
- Orthogonal transformation / quantization units 1208 and 1408 perform orthogonal transformation, quantization, and scaling on the macroblocks after data format conversion from data format conversion units 1206 and 1406, respectively.
- the orthogonal transform and quantization type may be dynamically changed according to the data format type of the macroblock output from the data format conversion units 1206 and 1406. For example, for the region defined by the residual, the same method as that used in the related art is applied, while for the region defined by the remainder, parameters related to orthogonal transformation, quantization, and scaling are further added. You may adjust.
- the data format inverse transform unit 1214 generates a motion compensation macroblock (an intra macroblock generated by the intraframe prediction unit 1218 or a motion compensation unit 1220 in the same frame for the region defined by the residual.
- the original macroblock is restored by adding the (inter macroblock).
- the motion compensation macroblock is also used as side information. More specifically, in order to determine a coefficient (denominator) used for inverse modulo operation for estimating the original pixel value from the remainder, a macro gradient macroblock for the motion compensation macroblock or a macro having information similar thereto A block is generated.
- the macroblock after the data format conversion includes the first data format in which the residue and the residue are combined in units of pixels, and the second data format in which the residue and the residue are combined in units of macroblocks.
- the same data format inverse conversion (restoration process) is applied to any macroblock.
- the data format reverse conversion (restoration process) for the macroblock after the data format conversion defined only by the remainder can be realized by excluding the process related to the calculation of the residual.
- FIG. 12 is a functional block diagram of the data format reverse conversion unit 1214 according to the embodiment of the present invention.
- data format inverse conversion unit 1214 includes processing selection unit 1290, addition unit 1292, gradient image generation unit 1270, coefficient selection unit 1272, Lookup table 1274, and inverse modulo calculation unit 1298. And a synthesis unit 1294. Components that execute the same processing as the components that constitute the data format conversion unit 1206 shown in FIG. 9 are denoted by the same reference numerals.
- the process selection unit 1290 determines the data format type for the macroblock after the data format conversion (restored by the inverse orthogonal transform / scaling unit 1212). At the same time, an area (pixel / macroblock) defined by the remainder and the residual is specified. Then, the process selection unit 1290 outputs the pixel value included in the region defined by the residual to the addition unit 1292 and outputs the pixel value included in the region defined by the remainder to the inverse modulo calculation unit 1298. To do.
- the addition unit 1292 adds the pixel value in the motion compensation macroblock corresponding to the pixel position of the pixel from which the pixel value is output from the process selection unit 1290 to the output pixel value. By this addition process, the corresponding pixel value of the original macroblock is restored.
- the adding unit 1292 outputs the calculation result to the synthesizing unit 1294.
- the inverse modulo operation unit 1298 inversely modulo the corresponding pixel value of the original macroblock from the pixel value (residue) output from the processing selection unit 1290 and the coefficient D used when calculating the residue. Estimate by calculation.
- the coefficient D necessary for the inverse modulo operation is determined according to the same process as the remainder calculation process in the data format conversion unit 1206. That is, the gradient image generation unit 1270 generates a pseudo gradient macroblock for the motion compensation macroblock, and the coefficient selection unit 1272 generates a Lookup table based on the pixel value (gradient strength) of each pixel of the generated pseudogradient macroblock. By referring to 1274, the coefficient D for each pixel is determined. Since the processing related to gradient image generation unit 1270, coefficient selection unit 1272, and Lookup table 1274 has been described with reference to FIG. 9, detailed description will not be repeated.
- the candidate value C (q ′) is as follows.
- the candidate value C (1) having the smallest difference from the corresponding pixel value SI of the motion compensation macroblock is selected, and the corresponding pixel value of the original macroblock is “11”. Determined. In this way, the pixel value of each pixel of the original macroblock is determined for each color component.
- the calculated pixel value is output to the synthesis unit 1294.
- the synthesizing unit 1294 combines the remainder or residual input for each pixel and outputs an original macroblock (Original MB).
- FIG. 13 is a functional block diagram of a decoder group according to the embodiment of the present invention.
- the decoding according to the embodiment of the present invention the multi-view video and the bit stream encoded with the corresponding multi-view depth information are processed, so that the decoder 210 decodes the multi-view video.
- the decoder 230 decodes the multi-view depth information. Decoders 210 and 230 perform decoding while sharing information with each other.
- the basic configuration is common between the decoder 210 and the decoder 230.
- FIG. 3 is a functional block diagram of a decoder 210 for decoding a multi-view video.
- the decoder 210 includes an input buffer 2102, an entropy decoding unit 2104, an inverse orthogonal transform / scaling unit 2112, a data format inverse transform unit 2114, a deblock filter 2116, and an intra frame prediction unit 2118.
- a decoder 230 for decoding multi-view depth information includes an input buffer 2302, an entropy decoding unit 2304, an inverse orthogonal transform / scaling unit 2312, a data format inverse transform unit 2314, a deblock filter 2316, Intraframe prediction unit 2318, motion compensation unit 2320, switching unit 2322, control unit 2330, and output buffer 2342 are included.
- the decoder 210 replaces the decoder 910 shown in FIG. 5 with a data format inverse conversion unit 2114 instead of the addition unit 9114 that adds the residual image and the previously calculated prediction image (interpolation information).
- the decoder 230 replaces the adder 9314 that adds the residual image and the previously calculated prediction image (interpolation information), instead of the data format inverse converter 2314.
- the main difference is that.
- the operation of the control unit 2130 is also different from that of the control unit 9130.
- the operations of the control units 2130 and 2330 are also different from the control units 9130 and 9330, respectively.
- the processing in the motion compensation units 2120 and 2320 is also different from that of the motion compensation units 9120 and 9320, respectively.
- input buffers 2102 and 2302, entropy decoding units 2104 and 2304, inverse orthogonal transform / scaling units 2112 and 2312, deblock filters 2116 and 2316, intra-frame prediction units 2118 and 2318, motion compensation units 2120 and 2320, and switching unit 2122 And 2322, and output buffers 2142 and 2342 are the functions of the input buffers 9102 and 9302, entropy decoding units 9104 and 9304, inverse orthogonal transform / scaling units 9112 and 9312, deblock filters 9116 and 9316, and intra frames shown in FIG. Prediction units 9118 and 9318, motion compensation units 9120 and 9320, switching units 9122 and 9322, and output buffer 9142 It has become something similar respectively to the pre-9342.
- a bit stream in which video is encoded is provided to input buffer 2102, and a bit stream in which corresponding depth information is encoded is provided to input buffer 2302.
- the embodiment of the present invention is suitable for a bitstream obtained by encoding an MVD composed of a multi-view video and corresponding multi-view depth information, but a single-view video captured by a single camera 10 and corresponding depth information. It can also be applied to a bitstream that encodes.
- the input buffer 2102 temporarily stores a bit stream encoded with video. Similarly, the input buffer 2302 temporarily stores a bit stream obtained by encoding the depth signal.
- the entropy decoding unit 2104 performs entropy decoding on the bit stream from the input buffer 2102 and, as a result, outputs motion data, a transformed transform coefficient, control data, and additional information.
- the motion data is given to the motion compensation unit 2120.
- the entropy decoding unit 2304 performs entropy decoding on the bit stream from the input buffer 2302, and as a result, outputs motion data, a transformed transform coefficient, control data, and additional information.
- the motion data is given to the motion compensation unit 2320.
- the inverse orthogonal transform / scaling units 2112 and 2312 respectively perform inverse orthogonal transform (typically, discrete Fourier inverse transform) and scaling on the quantized transform coefficients restored by the entropy decoding units 2104 and 2304, respectively. Execute. By these processes, the macroblock after the data format conversion is restored.
- inverse orthogonal transform typically, discrete Fourier inverse transform
- the data format reverse conversion unit 2114 executes the data format reverse conversion on the macroblock after the data format conversion, and the deblock filter 2116 blocks the result in order to suppress the occurrence of block noise. Smooth the border. By these processes, the original video is restored. Similarly, the data format reverse conversion unit 2314 performs data format reverse conversion on the macroblock after the data format conversion, and the deblock filter 2316 suppresses the occurrence of block noise on the result. To smooth the block boundaries. By these processes, the original depth information is restored.
- the intra frame prediction units 2118 and 2318 generate a prediction image based on adjacent macroblocks.
- connection that indicates what type of information is shared.
- the motion compensation unit 2120 of the decoder 210 shares motion data about the video decoded from the bitstream with the motion compensation unit 2320 of the decoder 230.
- the motion compensation unit 2320 of the decoder 230 shares motion data regarding the video decoded from the bitstream with the motion compensation unit 2120 of the decoder 210.
- the motion data from these other decoders is used to calculate motion data in each of the motion compensation units 2120 and 2320. More specifically, the motion compensation unit 2120 generates a predicted image using inter-frame prediction (inter-frame prediction). More specifically, each of the motion compensation units 2120 and 2320 generates a predicted image based on the restored original macroblock and the motion data for each of the restored video and depth information.
- One of the predicted images generated by each of the intra-frame prediction unit 2118 and the motion compensation unit 2120 is appropriately selected by the switching unit 2122 and given to the data format inverse conversion unit 2114.
- one of the prediction images generated by the intra frame prediction unit 2318 and the motion compensation unit 2320 is appropriately selected by the switching unit 2322 and is supplied to the data format inverse conversion unit 2314.
- control unit 2130 controls processing in the inverse orthogonal transform / scaling unit 2112, the data format inverse transform unit 2114, and the switching unit 2122 based on the control data and parameters restored by the entropy decoding unit 2104.
- control unit 2330 controls processing in inverse orthogonal transform / scaling unit 2312, data format inverse transform unit 2314, and switching unit 2322 based on the control data and parameters restored by entropy decoding unit 2304.
- control unit 2130 and the control unit 2330 exchange some control data in order to share the information as described above. As a result, integrated encoding for MVD can be realized.
- the output buffer 2142 temporarily stores the restored original video from the deblock filter 2116, and the output buffer 2342 temporarily stores the restored original depth information from the deblock filter 2316.
- the information defining the pixel value is switched between the residue and the residual in the encoding. Therefore, as additional information necessary for such control, parameters relating to data format conversion such as flags flag1 and flag2, type type, threshold values TH1 and TH2, and a remainder calculation parameter a are used.
- the encoder 120 that processes multi-view video and the encoder 140 that processes multi-view depth information perform encoding while sharing information.
- Parameters such as flags flag3 and flag4 are used as additional information necessary for the above.
- the flag flag1 and / or the flag flag2 are used to specify an area defined by a remainder in the macroblock after data format conversion. In other words, by invalidating both the flag flag1 and the flag flag2, it is specified that all regions are defined by residuals. When all such regions are defined by residuals, that is, when data format conversion is not performed, encoder 120 (more specifically, control unit 1230) and decoder 210 (more specifically, control unit) 2130) performs an operation according to a standard such as MPEG-4 AVC.
- the type type, threshold values TH1 and TH2, and the remainder calculation parameter a are used. It is done.
- Type type is a first data format (FIG. 8 (a)) that combines the remainder and residual in pixel units, and a second data format (FIG. 8 (b)) that combines the remainder and residual in macroblock units. ) And a parameter indicating which one is selected.
- the type type only needs to be able to specify which data format is selected, so it is sufficient that information of a single bit (1 bit) is assigned. The following parameters are used according to the selected data format.
- Threshold TH1 is used as a determination criterion for determining which of the plurality of pixels constituting each macroblock should be defined as a residue or a residual. That is, the threshold value TH1 is a criterion for specifying an area in which the pixel value is to be defined by the remainder among the pixels constituting the residual image (residual macroblock). It is transmitted to the decoder side as additional information.
- the remainder calculation parameter a is a parameter for determining the coefficient D used in the modulo calculation unit 1278 (FIG. 9).
- a threshold value for the pseudo gradient macroblock generated in the gradient image generation unit 1270 (FIG. 9) may be used as the remainder calculation parameter a. That is, the threshold value for determining each gradation in the Lookup table 1274 as shown in FIG.
- a plurality of Lookup tables as shown in FIG. 10 may be prepared, and an identifier indicating which Lookup table is selected may be used as the remainder calculation parameter a.
- the threshold value TH2 is used as a determination criterion for determining whether to define each of the macroblocks as a residue or a residual. In this determination, the threshold value TH1 is also used.
- ⁇ Remainder operation parameter a Similar to the remainder calculation parameter a used for the first data format described above, the threshold value for the pseudo gradient macroblock or an identifier indicating whether the Lookup table to be used is selected is included.
- the encoder 120 may perform optimization (rate-distortion optimization) according to speed distortion.
- optimization rate-distortion optimization
- the threshold value TH1 and / or the threshold value TH2 for determining whether to define the remainder or the residual should be the object of this optimization. This optimization can further improve performance.
- Flag 3 (having different meanings for encoders 120 and 140)
- Each of the encoders 120 and 140 uses a flag flag3 indicating the content of the processing related to the own device.
- the flag flag3 in the encoder 120 that performs encoding on the video is output from the corresponding depth information (dividing unit 1404 in FIG. 6) in motion data estimation for the video (multi-view video) in the motion estimation unit 1240 (FIG. 6). Is used.
- the flag flag3 in the encoder 140 that performs encoding for the depth information is the motion data (FIG. 6) for the corresponding video in the motion data estimation for the depth information (multi-view depth information) in the motion estimation unit 1440 (FIG. 6). Whether or not (output from the motion estimation unit 1240) is used.
- Flag flag4 (used in the encoder 120)
- the flag flag4 in the encoder 120 that performs encoding on the video is which flag information (indicated by the flag flag3 described above) is used in motion data estimation for the video in the motion estimation unit 1240 (FIG. 6). How it is used.
- the flag flag 4 (i) outputs the corresponding depth information itself as estimated motion data, and (ii) handles the corresponding depth information as an initial value of estimated motion data. It shows which process is executed among the output of final motion data after adjustment using the above.
- flags flag3 and flag4 are used, and in the decoder 230, flag flag3 is used.
- a flag flag3 handled by the decoders 210 and 230 indicates whether or not motion data is shared.
- the flag flag4 handled by the decoder 210 indicates whether or not corresponding depth information is used in motion data estimation for a video (multi-view video).
- FIG. 14 is a schematic diagram illustrating a hardware configuration of the information processing apparatus 100 that functions as a transmitter.
- FIG. 15 is a schematic diagram illustrating a hardware configuration of the information processing apparatus 200 that functions as a receiver.
- the information processing apparatus 100 includes a processor 104, a memory 106, a camera interface 108, a communication interface 112, a hard disk 114, an input unit 116, and a display unit 118. Each of these components is configured to be able to perform data communication with each other via a bus 122.
- the processor 104 implements the encoding process according to the embodiment of the present invention by reading a program stored in the hard disk 114 or the like, developing it in the memory 106 and executing it.
- the memory 106 functions as a working memory for the processor 104 to execute processing.
- the camera interface 108 is connected to a plurality of cameras 10 and acquires images captured by the respective cameras 10.
- the acquired image may be stored in the hard disk 114 or the memory 106.
- the hard disk 114 holds an encoding program 114a for realizing the above-described encoding process, multi-view video data 114b input from the camera interface 108, and the like in a nonvolatile manner.
- the input unit 116 typically includes a mouse, a keyboard, etc., and accepts an operation from the user.
- the display unit 118 notifies the user of processing results and the like.
- the communication interface 112 is connected to the wireless transmission device 102 and the like, and outputs data output as a result of processing by the processor 104 to the wireless transmission device 102.
- information processing apparatus 200 includes a processor 204, a memory 206, a projector interface 208, a communication interface 212, a hard disk 214, an input unit 216, and a display unit 218. Each of these components is configured to be able to perform data communication with each other via a bus 222.
- the processor 204, the memory 206, the input unit 216, and the display unit 218 are the same as the processor 104, the memory 106, the input unit 116, and the display unit 118 shown in FIG. Absent.
- the projector interface 208 is connected to the stereoscopic display device 300, and outputs the multi-viewpoint video restored by the processor 204 to the stereoscopic display device 300.
- the communication interface 212 is connected to the wireless transmission device 202 or the like, receives a bit stream transmitted from the information processing device 100, and outputs the bit stream to the bus 222.
- the hard disk 214 holds a decoding program 214a for realizing decoding and image data 214b including the restored original image in a nonvolatile manner.
- the hardware itself and the operation principle of the information processing apparatuses 100 and 200 shown in FIGS. 14 and 15 are general, and are essential for realizing encoding / decoding according to the embodiment of the present invention.
- the part is software (instruction code) such as an encoding program 114a and a decoding program 214a stored in a storage medium such as a hard disk.
- Such encoding program 114a and decoding program 214a are stored and distributed in storage media such as optical storage media, magnetic storage media, and semiconductor storage media.
- a storage medium for storing such a program can also be included in the scope of the present invention.
- the encoding program 114a and / or the decoding program 214a may be configured to execute processing using a module provided by an OS (Operating System).
- OS Operating System
- the encoding program 114a and / or the decoding program 214a does not include some modules, but even such a case is included in the technical scope of the present invention.
- All or part of the functions of the information processing apparatus 100 and / or the information processing apparatus 200 may be realized by using a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Or programmable hardware such as DSP (Digital Signal Processor).
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- DSP Digital Signal Processor
- a threshold is applied to a residual macroblock obtained by subtracting a motion compensation macroblock (intra macroblock or inter macroblock) from an original macroblock, whereby a remainder is obtained. And an area defined by each of the residuals is determined.
- This threshold and other parameters required for data format conversion may be optimized dynamically or statically using a speed optimization loop.
- a modulo operation is performed to calculate the remainder.
- the coefficient D that becomes the denominator (modulus) in this modulo calculation is determined based on the gradient image of the same motion compensation macroblock (or motion compensation frame) as the target macroblock.
- This gradient image ((pseudo) gradient macroblock or (pseudo) gradient frame) is generated from an intra macroblock (or intra frame) or an inter macroblock (or inter frame).
- the gradient may be calculated between macroblocks over a plurality of frames. That is, the gradient image may be calculated over the time domain and / or the spatial domain.
- the coefficient D used in the modulo calculation is determined according to the gradient image calculated in this way.
- the coefficient D used in the modulo operation is applied to a (pseudo) gradient macroblock (or gradient frame) for determining whether each region should be defined as a residue or a residual. It may be set to the same threshold value.
- a data format for a macroblock or a frame (1) a data format in which each area is defined only by a remainder, and (2) each area is defined by a combination of a remainder and a residual.
- a macroblock or frame is all zeros, a combination of residuals and zeros, all residuals, a combination of residuals and zeros, all residuals, a combination of residuals and residuals, a combination of residuals, residuals and zeros , And so on.
- the configuration example applied to MPEG-4 AVC which is one of the compression standards for moving images, has been shown.
- the data compression process after the data format conversion is executed according to a procedure in accordance with the standard.
- the data format conversion process is optimized in accordance with parameters related to data compression. Any data compression tool for images / video / multi-view video can be applied at the final stage of encoding.
- a decoder corresponding to the data format according to the embodiment of the present invention is used.
- information on the data format type (type) is transmitted from the encoder to the decoder. By adding such information, compatibility with conventional devices and existing standards can be ensured.
- the bit stream includes parameters related to coding and parameters related to the data format in addition to parameters necessary for the standard.
- an area defined by a residual may be further compensated based on a motion compensation macroblock / frame or a synthesized macroblock / frame.
- the corresponding value of the motion compensation macroblock / frame may be assigned to a region where zero is set.
- the area defined by the remainder is restored by the inverse modulo operation as described above.
- an image data format conversion method and an integrated encoding method of an image and a depth image which are used in image data compression processing.
- This method is a coding tool (an improvement to an image / video / multi-view video) that improves an existing standard for a multi-view video / image sequence captured by a camera row or Depth camera, or image data of an arbitrary format.
- a data compression tool an improvement to an image / video / multi-view video
- the data format conversion is executed in units of blocks (macroblocks) composed of a plurality of pixels.
- the data format conversion process includes the following steps.
- a difference block, that is, a residual block is generated based on an inter block, an intra block, and an original block.
- the data format inverse conversion processing includes the following steps.
- the encoding / decoding system according to the embodiment of the present invention can maintain compatibility with existing compression standards, it is possible to easily incorporate new data format conversion (encoding) according to the embodiment of the present invention. . As described above, in the encoding / decoding system according to the embodiment of the present invention, if the remainder information is not used, the same processing as that in the existing standard can be realized. Therefore, compatibility can be maintained.
- the encoding / decoding system includes various types such as distributed source coding, distributed video coding, and data compression for images / videos / multi-view video. Applicable to imaging systems.
- the data compression efficiency is improved by using a new data format within the framework of the existing standard related to data compression for images / videos / multi-viewpoint videos. Can be increased.
- the change to the data compression tool for the image / video / multi-view video in accordance with the existing standard is minimal.
- the data compression tool for the image / video / multi-view video that implements the encoding / decoding system according to the embodiment of the present invention invalidates the processing according to the embodiment of the present invention, The compatibility with the standard can still be maintained.
- the encoding / decoding system integrally encodes multi-view video and multi-view depth information.
- the encoding / decoding system integrally encodes multi-view video and multi-view depth information.
- by sharing motion data and depth information about the video between the encoder 120 and the encoder 140 it is possible to realize an integrated encoding for the MVD, thereby reducing the total data size after compressing the MVD.
- the size can be reduced to the same level as the shared information.
- 1 stereoscopic video playback system 2 subject, 10 camera, 100, 200 information processing device, 102, 202 wireless transmission device, 104, 204 processor, 106, 206 memory, 108 camera interface, 110 preprocessor, 112, 212 communication interface, 114, 214 hard disk, 114a encoding program, 116, 216 input unit, 118, 218 display unit, 120, 140, 820, 840 encoder, 122, 222 bus, 208 projector interface, 210, 230, 910, 930 decoder, 214a de Coding program, 214b image data, 240 post processor, 300 stereoscopic display, 302 projector array 310 display screen, 312 diffusion film, 314 condenser lens, 1202, 1402, 2102, 2302, 8202, 8402, 9102 input buffer, 1204, 1404, 8204, 8404 division unit, 1206, 1406 data format conversion unit, 1208, 1408 , 8208, 8408 Orthogonal transformation / quantization unit, 1210, 1410, 8
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Description
まず、本発明の実施の形態に係るエンコーディング/デコーディングシステムについての理解を容易化するために、典型的な応用例について説明する。なお、当然のことながら、本発明の実施の形態に係るエンコーディング/デコーディングシステムの応用範囲は、以下に示す構成に限定されるものではなく、任意の構成に応用できる。また、エンコーディングおよびデコーディングのいずれか一方のみを実行する方法、装置、プログラム、そのプログラムを格納する記憶媒体などについても、本願発明の範囲に含まれ得る。
まず、本発明の実施の形態に係るエンコーディング/デコーディングシステムに関連する技術について説明する。動画像の圧縮規格の一つである、MPEG-4 AVC(ITU-T Recommendation H.264|ISO/IEC 14496-10 Advanced Video Coding)のエンコーディングおよびデコーディングについて説明する。
次に、奥行情報付加の多視点映像(Multiview Video Plus Depth:以下「MVD」とも称す。)に向けられたエンコーディング/デコーディングシステムについて説明する。図4は、本発明の関連技術に係る多視点映像に対するエンコーディングに向けられた構成を示す機能ブロック図である。図5は、本発明の関連技術に係る多視点映像に対するデコーディングに向けられた構成を示す機能ブロック図である。
本発明の実施の形態に係るエンコーディング/デコーディングシステムは、典型的には、MVDのような、被写体の第1の情報(輝度情報)を含む複数の第1の入力画像(多視点映像/多視点画像)と、複数の第1の入力画像にそれぞれ対応する、被写体の第1の情報とは異なる第2の情報(奥行情報)を含む複数の第2の入力画像(多視点奥行情報/多視点Depthマップ)といった、複数の入力画像をエンコーディングする。但し、MVDに限定されることなく、互いにその情報を共有することが可能な複数種類の入力画像群(典型的には、一対の入力画像群)に対して、エンコーディングおよびデコーディングが可能になっている。そのため、映像と奥行情報との組み合わせに限らず、他の情報の組み合わせを含む画像群に対して適用可能である。
まず、本発明の実施の形態に係るエンコーディング/デコーディングシステムを構成するエンコーダー群の機能構成について説明する。図6は、本発明の実施の形態に係るエンコーダー群の機能ブロック図である。図6を参照して、本発明の実施の形態に係るエンコーディングでは、MVDを構成する多視点映像および対応する多視点奥行情報が入力データとして与えられる。図1に示すエンコーダー120は、多視点映像をエンコーディングし、エンコーダー140は、多視点奥行情報をエンコーディングする。エンコーダー120および140は、互いに情報を共有しつつエンコーディングを行なう。基本的な構成は、エンコーダー120とエンコーダー140との間で共通している。
次に、エンコーダー120および140における処理について説明する。図6を参照して、入力ソースからの映像信号が入力バッファ1202へ与えられ、対応する奥行情報が入力バッファ1402へ与えられる。典型的に、映像として、複数のカメラ10(カメラアレイ)によって撮像された多視点映像が入力され、奥行情報として、対応する多視点奥行情報が入力される。但し、このようなMVDに限られず、単一のカメラ10によって撮像された単視点映像および対応する奥行情報であってもよい。
制御部1230は、データフォーマット変換部1206、直交変換・量子化部1208、逆直交変換・スケーリング部1212、データフォーマット逆変換部1214、切換部1222、および、動き推定部1240における処理を制御する。また、制御部1230は、制御データとして、符号化に係るパラメータや各コンポーネントに対する符号化の順序などを出力する。さらに、制御部1230は、データフォーマット変換に係る付加情報(データフォーマットタイプ(type)、しきい値、フラグなど)をエントロピー符号化部1250へ出力する。
[G.動き推定部1240および1440における処理]
まず、図1に示すエンコーダー120とエンコーダー140との間での情報共有の一形態として、動き推定部1240および1440における動きデータの推定に係る処理について説明する。なお、映像に係るエンコーダーと奥行情報に係るエンコーダーとの間で情報を共有する方法としては、以下の方法に限られるものではない。
動き推定部1240における映像(多視点映像)についての動きデータの推定において、多視点奥行情報が利用される。この奥行情報の利用形態としては、典型的には、以下のような方法が考えられる。
(ii)対応する奥行情報を推定される動きデータの初期値として取り扱い、さらに、映像の情報などを用いて調整をした上で、最終的な動きデータとして出力する。
動き推定部1440における奥行情報(多視点奥行情報)についての動きデータの推定において、多視点映像が利用される。この多視点映像の利用形態としては、典型的には、以下のような2つの方法が考えられる。
上述のような情報共有によって生成される、エンコーディングに用いられる予測画像(補間情報)について説明する。
映像のマクロブロックが十分なテクスチャーを有していない場合には、奥行情報そのものを映像の対応する各マクロブロックの動きデータとして用いてもよい。なお、映像のマクロブロックのテクスチャーの量は、その疑似勾配マクロブロックにしきい値を適用することで、決定してもよい。この場合、映像の動きデータを生成するためには、不足している領域の情報を生成する必要がある。この不足している領域の情報は、奥行情報または奥行情報の動きデータを初期値として用いることで推定することで生成できる。あるいは、この不足している領域の情報を映像の情報からのみ推定することで生成してもよい。また、奥行情報についても同様の手法を適用できる。
次に、本発明の実施の形態に係るデータフォーマット変換部1206および1406における処理について詳述する。なお、データフォーマット変換部1206および1406における処理は実質的に同一であるので、説明の便宜上、以下では、データフォーマット変換部1206について説明する。
上述したように、本発明の実施の形態においては、剰余のみで定義する構成、および剰余と残差とを組み合せて定義する構成の両方を採用し得る。後者の場合には、さらに、(1)画素単位での剰余と残差との組み合わせ、および、(2)マクロブロック単位での剰余と残差(あるいは、すべてゼロ)との組み合わせの両方を採用し得る。
上述したように、データフォーマット変換部1206から出力されるデータフォーマット変換後のマクロブロックには複数のタイプがあるので、サイド情報の一部として、このデータフォーマット変換の手順を示す情報(データフォーマットタイプ(type))が用いられる。但し、残差で定義される領域については、サイド情報を含めないようにしてもよい。つまり、対応するサイド情報が存在する領域(画素またはマクロブロック)について剰余で定義されていることが暗示される。
図9は、本発明の実施の形態に係るデータフォーマット変換部1206の機能ブロック図である。図9を参照して、データフォーマット変換部1206は、減算部1260と、比較部1262と、マスク生成部1264と、処理選択部1266と、勾配画像生成部1270と、係数選択部1272と、Lookupテーブル1274と、モジュロ演算部1278と、合成部1280とを含む。
(iii-2)(1回以上の)ガウシアンスムージング処理(あるいは、メディアンフィルタ処理)
(iii-3)一連のモルフォロジカル処理(例えば、(1回以上の)膨脹処理、(1回以上の)収縮処理、(1回以上の)膨脹処理)
(iii-4)(1回以上の)ガウシアンスムージング処理
以上のような処理によって、動き補償マクロブロックを構成するカラーコンポーネント別に疑似勾配マクロブロックが生成される。
図11は、本発明の実施の形態に係るデータフォーマット変換部1206の別の機能ブロック図である。図11を参照して、データフォーマット変換部1206は、図9に示すデータフォーマット変換部1206に比較して、マスク生成部1264、処理選択部1266、および合成部1280に代えて、積算部1265、評価部1267、および切換部1269を設けたものである。その他のコンポーネントの詳細については、上述したのでその内容は繰り返さない。
次に、本発明の実施の形態に係る直交変換・量子化部1208および1408における処理について詳述する。
次に、本発明の実施の形態に係るデータフォーマット逆変換部1214および1414における処理について詳述する。なお、データフォーマット逆変換部1214および1414における処理は実質的に同一であるので、説明の便宜上、以下では、データフォーマット逆変換部1214について説明する。
上述したように、データフォーマット変換部1206から出力されるデータフォーマット変換後のマクロブロックには複数のタイプがあるので、サイド情報に含まれるデータフォーマットタイプに基づいて、データフォーマット逆変換の手順が選択される。
図12は、本発明の実施の形態に係るデータフォーマット逆変換部1214の機能ブロック図である。図12を参照して、データフォーマット逆変換部1214は、処理選択部1290と、加算部1292と、勾配画像生成部1270と、係数選択部1272と、Lookupテーブル1274と、逆モジュロ演算部1298と、合成部1294とを含む。なお、図9に示すデータフォーマット変換部1206を構成するコンポーネントと同様の処理を実行するコンポーネントについては、同じ参照符号を付している。
候補値C(1)=1×8+3=11 (SIとの差=3)
候補値C(2)=2×8+3=19 (SIとの差=11)
…
これらの候補値C(q’)のうち、動き補償マクロブロックの対応する画素値SIとの差が最も小さくなる候補値C(1)が選択され、オリジナルマクロブロックの対応する画素値は「11」に決定される。このようにして、オリジナルマクロブロックの各画素の画素値がカラーコンポーネント別にそれぞれ決定される。この算出された画素値が合成部1294へ出力される。合成部1294は、画素毎に入力される剰余または残差を結合して、オリジナルマクロブロック(Original MB)を出力する。
次に、本発明の実施の形態に係るエンコーディング/デコーディングシステムを構成するデコーダー群の機能構成について説明する。図13は、本発明の実施の形態に係るデコーダー群の機能ブロック図である。図13を参照して、本発明の実施の形態に係るデコーディングにおいては、多視点映像および対応する多視点奥行情報がエンコーディングされたビットストリームを処理するので、デコーダー210は、多視点映像をデコーディングし、デコーダー230は、多視点奥行情報をデコーディングする。デコーダー210および230は、互いに情報を共有しつつデコーディングを行なう。基本的な構成は、デコーダー210とデコーダー230との間で共通している。
次に、デコーダー210および230における処理手順について説明する。図13を参照して、映像をエンコーディングしたビットストリームが入力バッファ2102へ与えられ、対応する奥行情報をエンコーディングしたビットストリームが入力バッファ2302へ与えられる。本発明の実施の形態は、多視点映像と対応する多視点奥行情報とからなるMVDをエンコーディングしたビットストリームに適しているが、単一のカメラ10によって撮像された単視点映像および対応する奥行情報をエンコーディングしたビットストリームに対しても適用可能である。
次に、本発明の実施の形態に係るエンコーディング/デコーディングシステムにおいて用いられるパラメータおよびサイド情報などを含む付加情報について詳述する。
フラグflag1および/またはフラグflag2は、データフォーマット変換後のマクロブロックにおいて剰余で定義される領域を特定するために用いられる。言い換えれば、フラグflag1およびフラグflag2をいずれも無効化することで、すべての領域が残差で定義されることが特定される。このようなすべての領域が残差で定義される、つまりデータフォーマット変換が実施されない場合には、エンコーダー120(より具体的には、制御部1230)およびデコーダー210(より具体的には、制御部2130)は、例えば、MPEG-4 AVCのような規格に従った動作を行なう。
《フラグflag1》
マクロブロックを構成する画素毎にフラグflag1が割り当てられ、フラグflag1の各々は、対応する画素が剰余および残差のいずれで定義されるのかを示す。代替の構成として、剰余および残差の一方にのみフラグflag1を割当て、他方にはフラグflag1を割り当てないようにすることで、各画素について、剰余および残差のいずれで定義されるのかを特定できる。
しきい値TH1は、各マクロブロックを構成する複数の画素の各々について、剰余および残差のいずれで定義すべきかを判断するための判断基準として用いられる。つまり、しきい値TH1は、残差画像(残差マクロブロック)を構成する画素のうちその画素値を剰余で定義すべき領域を特定するための判断基準であり、このしきい値TH1は、付加情報としてデコーダー側へ送信される。
剰余演算用パラメータaは、モジュロ演算部1278(図9)において用いられる係数Dを決定するためのパラメータである。一例として、勾配画像生成部1270(図9)において生成される疑似勾配マクロブロックに対するしきい値を、剰余演算用パラメータaとしてもよい。つまり、図10に示すようなLookupテーブル1274における各階調を決定するしきい値が剰余演算用パラメータaになる。
《フラグflag2》
マクロブロック毎にフラグflag2が割り当てられ、フラグflag2の各々は、対応するマクロブロックが剰余および残差のいずれで定義されるのかを示す。代替の構成として、剰余および残差の一方にのみフラグflag2を割当て、他方にはフラグflag2を割り当てないようにすることで、各マクロブロックについて、剰余および残差のいずれで定義されるのかを特定できる。
しきい値TH2は、マクロブロックの各々について、剰余および残差のいずれで定義すべきかを判断するための判断基準として用いられる。この判断においては、しきい値TH1も用いられる。
上述の第1データフォーマットに対して用いられる剰余演算用パラメータaと同様に、疑似勾配マクロブロックに対するしきい値、あるいは、使用するLookupテーブルを選択するのかを示す識別子を含む。
《フラグflag3(エンコーダー120および140で異なる意味を持つ)》
エンコーダー120および140の各々は、それぞれ自装置に関する処理の内容を示すフラグflag3を用いる。
映像に対するエンコーディングを実行するエンコーダー120におけるフラグflag4は、動き推定部1240(図6)における映像についての動きデータの推定において、対応する奥行情報が用いられる場合(上述のフラグflag3によって示される)、どのように用いられるのかを示す。つまり、フラグflag4は、(i)対応する奥行情報そのものを推定された動きデータとして出力する、および、(ii)対応する奥行情報を推定される動きデータの初期値として取り扱い、さらに、映像の情報などを用いて調整をした上で、最終的な動きデータとして出力する、のうちいずれの処理が実行されるのかを示す。
デコーダー210では、フラグflag3およびflag4が用いられ、デコーダー230では、フラグflag3が用いられる。
次に、上述の送信機および受信機を実現するためのハードウェア構成の一例について説明する。図14は、送信機として機能する情報処理装置100のハードウェア構成を示す模式図である。図15は、受信機として機能する情報処理装置200のハードウェア構成を示す模式図である。
本発明の実施の形態においては、オリジナルマクロブロックから動き補償マクロブロック(イントラマクロブロック、または、インターマクロブロック)を減じた残差マクロブロックに対して、しきい値が適用されることで、剰余および残差のそれぞれで定義される領域が決定される。このしきい値およびデータフォーマット変換に必要な他のパラメータは、速度最適化ループを用いて動的または静的に最適化されてもよい。
(1a)インターブロック(前方向予測、後方向予測、および両方向予測のうちいずれかを用いてコーディングされたブロック)、または、イントラブロック、ならびに対応する位置のオリジナルブロックに基づいて、各画素のブロックデータフォーマットを所定のパラメータの組に従って、剰余、残差、ゼロのいずれかに変換する工程
(1b)インターブロック、または、イントラブロック、ならびにオリジナルブロックに基づいて、差分ブロック、すなわち残差ブロックを生成する工程
(1c)所定のパラメータの組および残差ブロックに基づいて、その値を剰余に変換すべきと判断された画素についてフラグを有効化する工程
(1d)所定のパラメータの組および残差マクロブロックに基づいて、その値をゼロにすべきと判断された画素について、その値をゼロに変換する工程(これらの画素は、残差がゼロの画素として取り扱われる)
(1e)インターマクロブロック、または、イントラマクロブロックに基づいて、疑似勾配画像を生成する工程
(1f)当該疑似勾配画像に基づいて、モジュロ演算によって剰余を決定するためのパラメータを設定する工程
(1g)オリジナルブロックおよびモジュロ演算について設定されたパラメータの組に基づいて、その値を剰余に変更すべきと判断された画素について、剰余に変換する工程
(1h)画像/映像/多視点映像に対するデータ圧縮に係る既存の規格に類似したデータ圧縮に係る処理を、新たなデータフォーマット変換後のブロックに対して実行する工程
(1i)データフォーマット変換のためのパラメータを規格に従う最適化処理に追加して、当該パラメータのデータ圧縮を実行する工程
(1j)新たなデータフォーマットの規格に従うデータ圧縮パラメータおよびデータフォーマット変換のためのパラメータに対して、既存の最適化処理を用いて、その圧縮効率および圧縮品質をより高くするための最適化処理を実行する工程
次に、統合的なデータ圧縮の処理は、以下の工程を含む。
(2b)奥行情報が映像エンコーダーにおける動き推定部における推定結果の初期値として用いられる場合に、動きデータをさらに調整し改善する工程
(2c)映像エンコーダーにおける動きデータに代えて、奥行情報エンコーダーにおける動き推定結果を用いる工程
(2d)奥行情報エンコーダーにおける動きデータに代えて、映像エンコーダーにおける動き推定結果を用いる工程
(2e)共有される情報のタイプを示す対応するパラメータをデコーダーへ送信する工程
次に、データフォーマット逆変換の処理は、以下の工程を含む。
(3b)インターマクロブロック、または、イントラマクロブロック、ならびに残差画素およびゼロ画素に基づいて、フラグが有効化されていない画像について、残差からオリジナルの画素値を復元する工程
(3c)インターマクロブロック、または、イントラマクロブロック、ならびにフラグによって示される剰余の画素に基づいて、逆モジュロ演算を実行する工程(逆モジュロ演算の実行にあたって、受信したビットストリームから抽出されたモジュロ演算のための対応するパラメータが利用される)
最後に、統合的なデータ復元の処理は、以下の工程を含む。
(4b)ビットストリームに含まれる、デコーディングのための制御データを用いて共有情報のタイプを識別する工程
上述の説明においては、データフォーマット変換およびデータフォーマット逆変換をマクロブロック単位で実行する場合について説明したが、画像全体に適用できることはいうまでもない。具体的には、オリジナル画像全体についての残差画像を先に生成し、当該生成した残差画像に対して、上述した処理を画像単位で実行してもよい。
本発明の実施の形態に係るエンコーディング/デコーディングシステムは、既存の圧縮規格との互換性を維持できるので、本発明の実施の形態に係る新たなデータフォーマット変換(エンコーディング)の組み込みが容易化できる。上述したように、本発明の実施の形態に係るエンコーディング/デコーディングシステムにおいては、剰余の情報を用いなければ、既存の規格での処理と同一の処理を実現することもできる。そのため、互換性を維持できる。
Claims (7)
- 複数の入力画像をエンコーディングする方法であって、
被写体の第1の情報を含む複数の第1の入力画像と、前記複数の第1の入力画像にそれぞれ対応する、前記被写体の前記第1の情報とは異なる第2の情報を含む複数の第2の入力画像とを取得するステップと、
前記第1の入力画像について、他の第1の入力画像および前記第2の入力画像の少なくとも一方に含まれる情報から第1の予測画像を算出するステップと、
前記第1の入力画像と対応する前記第1の予測画像との差分から第1の残差画像を生成するステップと、
前記第1の残差画像の画素値に基づいて、前記第1の残差画像を構成する画素のうちその画素値を剰余で定義すべき領域を特定するステップと、
前記第1の残差画像の前記剰余で定義すべきとされた領域に含まれる画素値を剰余に変換するステップと、
前記第2の入力画像について、他の第2の入力画像および前記第1の入力画像の少なくとも一方に含まれる情報から第2の予測画像を算出するステップと、
前記第2の入力画像と対応する前記第2の予測画像との差分から第2の残差画像を生成するステップと、
前記第2の残差画像の画素値に基づいて、前記第2の残差画像を構成する画素のうちその画素値を剰余で定義すべき領域を特定するステップと、
前記第2の残差画像の前記剰余で定義すべきとされた領域に含まれる画素値を剰余に変換するステップと、
変換後の第1の残差画像と、変換後の第2の残差画像と、前記第1の残差画像および前記第2の残差画像のそれぞれにおける前記剰余で定義すべき領域を特定する付加情報とをエンコーディングするステップとを含む、方法。 - 前記剰余に変換するステップは、
前記剰余で定義すべき領域についての画素値に対してモジュロ演算を実行するステップと、
前記予測画像の勾配情報を取得するステップと、
勾配強度とモジュロ演算の法となる値との予め定められた対応関係を参照して、取得した前記勾配情報に基づいて前記モジュロ演算の法となる値を決定するステップとを含む、請求項1に記載の方法。 - 前記第1の予測画像を算出するステップは、算出対象の第1の入力画像に対応する第2の入力画像と、先行の1つ以上の第1の入力画像とを用いて、前記第1の予測画像を算出するステップを含む、請求項1または2に記載の方法。
- 前記第2の予測画像を算出するステップは、算出対象の第2の入力画像に対応する第1の入力画像についての動きデータを用いて、前記第2の予測画像を算出するステップを含み、
前記第1の入力画像についての動きデータは、先行の第1の入力画像と算出対象の第1の入力画像との間の変化成分を示す、請求項1~3のいずれか1項に記載の方法。 - 前記第2の予測画像を算出するステップは、算出対象の第2の入力画像と、先行の1つ以上の第2の入力画像とを用いて、前記第2の予測画像を算出するステップを含む、請求項1~3のいずれか1項に記載の方法。
- 複数の入力画像をエンコーディングするプログラムを格納する記憶媒体であって、当該プログラムは、コンピュータに、
被写体の第1の情報を含む複数の第1の入力画像と、前記複数の第1の入力画像にそれぞれ対応する、前記被写体の前記第1の情報とは異なる第2の情報を含む複数の第2の入力画像とを取得するステップと、
前記第1の入力画像について、他の第1の入力画像および前記第2の入力画像の少なくとも一方に含まれる情報から第1の予測画像を算出するステップと、
前記第1の入力画像と対応する前記第1の予測画像との差分から第1の残差画像を生成するステップと、
前記第1の残差画像の画素値に基づいて、前記第1の残差画像を構成する画素のうちその画素値を剰余で定義すべき領域を特定するステップと、
前記第1の残差画像の前記剰余で定義すべきとされた領域に含まれる画素値を剰余に変換するステップと、
前記第2の入力画像について、他の第2の入力画像および前記第1の入力画像の少なくとも一方に含まれる情報から第2の予測画像を算出するステップと、
前記第2の入力画像と対応する前記第2の予測画像との差分から第2の残差画像を生成するステップと、
前記第2の残差画像の画素値に基づいて、前記第2の残差画像を構成する画素のうちその画素値を剰余で定義すべき領域を特定するステップと、
前記第2の残差画像の前記剰余で定義すべきとされた領域に含まれる画素値を剰余に変換するステップと、
変換後の第1の残差画像と、変換後の第2の残差画像と、前記第1の残差画像および前記第2の残差画像のそれぞれにおける前記剰余で定義すべき領域を特定する付加情報とをエンコーディングするステップとを実行させる、記憶媒体。 - 複数の入力画像をエンコーディングする装置であって、
被写体の第1の情報を含む複数の第1の入力画像と、前記複数の第1の入力画像にそれぞれ対応する、前記被写体の前記第1の情報とは異なる第2の情報を含む複数の第2の入力画像とを取得する手段と、
前記第1の入力画像について、他の第1の入力画像および前記第2の入力画像の少なくとも一方に含まれる情報から第1の予測画像を算出する手段と、
前記第1の入力画像と対応する前記第1の予測画像との差分から第1の残差画像を生成する手段と、
前記第1の残差画像の画素値に基づいて、前記第1の残差画像を構成する画素のうちその画素値を剰余で定義すべき領域を特定する手段と、
前記第1の残差画像の前記剰余で定義すべきとされた領域に含まれる画素値を剰余に変換する手段と、
前記第2の入力画像について、他の第2の入力画像および前記第1の入力画像の少なくとも一方に含まれる情報から第2の予測画像を算出する手段と、
前記第2の入力画像と対応する前記第2の予測画像との差分から第2の残差画像を生成する手段と、
前記第2の残差画像の画素値に基づいて、前記第2の残差画像を構成する画素のうちその画素値を剰余で定義すべき領域を特定する手段と、
前記第2の残差画像の前記剰余で定義すべきとされた領域に含まれる画素値を剰余に変換する手段と、
変換後の第1の残差画像と、変換後の第2の残差画像と、前記第1の残差画像および前記第2の残差画像のそれぞれにおける前記剰余で定義すべき領域を特定する付加情報とをエンコーディングする手段とを実行させる、装置。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201480018695.4A CN105103546A (zh) | 2013-03-27 | 2014-03-12 | 编码多个输入图像的方法及装置、存放程序的存储介质 |
KR1020157030610A KR20150135457A (ko) | 2013-03-27 | 2014-03-12 | 복수의 입력 화상을 인코딩하는 방법, 프로그램을 격납하는 기억 매체 및 장치 |
EP14775632.4A EP2981083A1 (en) | 2013-03-27 | 2014-03-12 | Method for encoding a plurality of input images and storage medium and device for storing program |
US14/780,963 US20160065958A1 (en) | 2013-03-27 | 2014-03-12 | Method for encoding a plurality of input images, and storage medium having program stored thereon and apparatus |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013-066314 | 2013-03-27 | ||
JP2013066314A JP2014192702A (ja) | 2013-03-27 | 2013-03-27 | 複数の入力画像をエンコーディングする方法、プログラムおよび装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014156648A1 true WO2014156648A1 (ja) | 2014-10-02 |
Family
ID=51623631
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2014/056484 WO2014156648A1 (ja) | 2013-03-27 | 2014-03-12 | 複数の入力画像をエンコーディングする方法、プログラムを格納する記憶媒体および装置 |
Country Status (6)
Country | Link |
---|---|
US (1) | US20160065958A1 (ja) |
EP (1) | EP2981083A1 (ja) |
JP (1) | JP2014192702A (ja) |
KR (1) | KR20150135457A (ja) |
CN (1) | CN105103546A (ja) |
WO (1) | WO2014156648A1 (ja) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2014082541A (ja) * | 2012-10-12 | 2014-05-08 | National Institute Of Information & Communication Technology | 互いに類似した情報を含む複数画像のデータサイズを低減する方法、プログラムおよび装置 |
CN117097891A (zh) * | 2017-06-05 | 2023-11-21 | 松下电器(美国)知识产权公司 | 解码装置、编码装置、存储介质、解码方法和编码方法 |
JP7401309B2 (ja) * | 2018-01-30 | 2023-12-19 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | 符号化装置、復号装置、符号化方法及び復号方法 |
JP7315480B2 (ja) * | 2018-01-30 | 2023-07-26 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | 符号化装置、復号装置、符号化方法及び復号方法 |
CN110493602A (zh) * | 2019-08-19 | 2019-11-22 | 张紫薇 | 视频编码快速运动搜索方法及系统 |
US20210127125A1 (en) * | 2019-10-23 | 2021-04-29 | Facebook Technologies, Llc | Reducing size and power consumption for frame buffers using lossy compression |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH04343576A (ja) * | 1991-05-21 | 1992-11-30 | Matsushita Electric Ind Co Ltd | 高能率符号化方法と高能率符号の復号方法 |
JP2011514024A (ja) * | 2008-01-21 | 2011-04-28 | テレフオンアクチーボラゲット エル エム エリクソン(パブル) | 予測に基づく画像処理 |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100365555B1 (ko) * | 1994-10-19 | 2003-08-27 | 마츠시타 덴끼 산교 가부시키가이샤 | 화상부호화/복호화장치 |
CN1258925C (zh) * | 2003-06-27 | 2006-06-07 | 中国科学院计算技术研究所 | 多视角视频编解码预测补偿方法及装置 |
JP5059766B2 (ja) * | 2005-09-22 | 2012-10-31 | サムスン エレクトロニクス カンパニー リミテッド | 視差ベクトルの予測方法、その方法を利用して多視点動画を符号化及び復号化する方法及び装置 |
US7706619B2 (en) * | 2005-11-30 | 2010-04-27 | Brother Kogyo Kabushiki Kaisha | Image processing device that produces high-quality reduced image at fast processing speed |
CN101563930B (zh) * | 2006-12-28 | 2011-05-25 | 日本电信电话株式会社 | 影像编码方法和解码方法、其装置和程序和记录有程序的存储介质 |
KR100801968B1 (ko) * | 2007-02-06 | 2008-02-12 | 광주과학기술원 | 변위를 측정하는 방법, 중간화면 합성방법과 이를 이용한다시점 비디오 인코딩 방법, 디코딩 방법, 및 인코더와디코더 |
CN101919250B (zh) * | 2008-01-21 | 2012-11-21 | 艾利森电话股份有限公司 | 压缩多个像素块的方法和压缩器、以及对压缩像素块进行解压的方法和解压器 |
EP2327059B1 (en) * | 2008-10-02 | 2014-08-27 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Intermediate view synthesis and multi-view data signal extraction |
US8694573B2 (en) * | 2009-10-26 | 2014-04-08 | Jadavpur University | Method and system for determining a quotient value |
JP5660361B2 (ja) * | 2010-03-26 | 2015-01-28 | ソニー株式会社 | 画像処理装置および方法、並びにプログラム |
AU2011201336B2 (en) * | 2011-03-23 | 2013-09-05 | Canon Kabushiki Kaisha | Modulo embedding of video parameters |
US9723315B2 (en) * | 2011-07-01 | 2017-08-01 | Apple Inc. | Frame encoding selection based on frame similarities and visual quality and interests |
-
2013
- 2013-03-27 JP JP2013066314A patent/JP2014192702A/ja not_active Withdrawn
-
2014
- 2014-03-12 EP EP14775632.4A patent/EP2981083A1/en not_active Withdrawn
- 2014-03-12 KR KR1020157030610A patent/KR20150135457A/ko not_active Application Discontinuation
- 2014-03-12 US US14/780,963 patent/US20160065958A1/en not_active Abandoned
- 2014-03-12 CN CN201480018695.4A patent/CN105103546A/zh active Pending
- 2014-03-12 WO PCT/JP2014/056484 patent/WO2014156648A1/ja active Application Filing
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH04343576A (ja) * | 1991-05-21 | 1992-11-30 | Matsushita Electric Ind Co Ltd | 高能率符号化方法と高能率符号の復号方法 |
JP2011514024A (ja) * | 2008-01-21 | 2011-04-28 | テレフオンアクチーボラゲット エル エム エリクソン(パブル) | 予測に基づく画像処理 |
Non-Patent Citations (8)
Title |
---|
HISAYOSHI FURIHATA ET AL.: "Residual Prediction for Free Viewpoint Image Generation", IEICE TECHNICAL REPORT, vol. 109, no. 63, 21 May 2009 (2009-05-21), pages 199 - 203, XP008179072 * |
MEHRDAD PANAHPOUR TEHRANI ET AL.: "A hybrid representation for multi-view images, 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video", 3DTV- CON, 15 October 2012 (2012-10-15), pages 1 - 4, XP032275888, DOI: 10.1109/3DTV.2012.6365445 * |
MEHRDAD PANAHPOUR TEHRANI ET AL.: "A New Data Format for Multiview Video", IPSJ SIG TECHNICAL REPORT, vol. 2013, no. 6, 22 February 2013 (2013-02-22), pages 1 - 5, XP055280613 * |
MEHRDAD PANAHPOUR TEHRANI ET AL.: "The Adaptive Distributed Source Coding of Multi-View Images in Camera Sensor Networks", IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS, COMMUNICATIONS AND COMPUTER SCIENCES, vol. E88-A, no. 10, 1 October 2005 (2005-10-01), pages 2835 - 2843, XP001234495, DOI: 10.1093/IETFEC/E88-A.10.2835 * |
P. MERKLE; K. MULLER; A. SMOLIC; T. WIEGAND: "Efficient Compression of Multi-view Video Exploiting inter-view dependencies based on H.264/MPEG4-AVC", PROC. ICME 2006, pages 1717 - 1720 |
S. YEA; A. VETRO: "View synthesis prediction for multiview video coding", SIGNAL PROCESSING: IMAGE COMMUNICATION, vol. 24, 2009, pages 89 - 100, XP025884347, DOI: doi:10.1016/j.image.2008.10.007 |
THOMAS WIEGAND; GARY J. SULLIVAN; GISLE BJONTEGAARD; AJAY LUTHRA: "Overview of the H.264/AVC Video Coding Standard", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, vol. 13, no. 7, July 2003 (2003-07-01), pages 560 - 576, XP008129745, DOI: doi:10.1109/TCSVT.2003.815165 |
YUTA HIGUCHI ET AL.: "N-View N-depth Coding for Free View Generation", IEICE TECHNICAL REPORT, vol. 111, no. 349, 8 December 2011 (2011-12-08), pages 1 - 6, XP008179069 * |
Also Published As
Publication number | Publication date |
---|---|
CN105103546A (zh) | 2015-11-25 |
JP2014192702A (ja) | 2014-10-06 |
US20160065958A1 (en) | 2016-03-03 |
KR20150135457A (ko) | 2015-12-02 |
EP2981083A1 (en) | 2016-02-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6633694B2 (ja) | 多視点信号コーデック | |
JP6178017B2 (ja) | ステレオビデオのための深度認識向上 | |
JP2021168479A (ja) | デプスマップの推定および更新を用いる効率的なマルチビュー符号化 | |
CA2692250C (en) | Video encoding and decoding methods using residual prediction, and corresponding apparatuses | |
WO2014156648A1 (ja) | 複数の入力画像をエンコーディングする方法、プログラムを格納する記憶媒体および装置 | |
US20150245062A1 (en) | Picture encoding method, picture decoding method, picture encoding apparatus, picture decoding apparatus, picture encoding program, picture decoding program and recording medium | |
KR20120000485A (ko) | 예측 모드를 이용한 깊이 영상 부호화 장치 및 방법 | |
JPWO2008081810A1 (ja) | 映像符号化方法及び復号方法、それらの装置、それらのプログラム並びにプログラムを記録した記憶媒体 | |
CN112789851B (zh) | 视频解码方法、装置、电子设备及可读存储介质 | |
JP6039178B2 (ja) | 画像符号化装置、画像復号装置、並びにそれらの方法及びプログラム | |
CN117880499A (zh) | 去块效应滤波自适应的编码器、解码器及对应方法 | |
WO2011114755A1 (ja) | 多視点画像符号化装置 | |
JP5894301B2 (ja) | 映像符号化装置および方法、映像復号装置および方法、及びそれらのプログラム | |
CN113597769A (zh) | 基于光流的视频帧间预测 | |
KR102105323B1 (ko) | 객체 기반 적응적 밝기 보상 방법 및 장치 | |
WO2011070730A1 (ja) | 映像符号化装置および映像復号装置 | |
WO2015056712A1 (ja) | 動画像符号化方法、動画像復号方法、動画像符号化装置、動画像復号装置、動画像符号化プログラム、及び動画像復号プログラム | |
WO2014156647A1 (ja) | 複数の入力画像をエンコーディングする方法、プログラムを格納する記憶媒体および装置 | |
CN112262574A (zh) | 一种用于帧内预测的设备和方法 | |
JP6232117B2 (ja) | 画像符号化方法、画像復号方法、及び記録媒体 | |
WO2011142221A1 (ja) | 符号化装置、および、復号装置 | |
KR20140124045A (ko) | 객체 기반 적응적 밝기 보상 방법 및 장치 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201480018695.4 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14775632 Country of ref document: EP Kind code of ref document: A1 |
|
REEP | Request for entry into the european phase |
Ref document number: 2014775632 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2014775632 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14780963 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 20157030610 Country of ref document: KR Kind code of ref document: A |