KR101386651B1 - Multi-View video encoding and decoding method and apparatus thereof - Google Patents

Abstract

Disclosed is a technique for synthesizing an intermediate view image using a plurality of multi-view images and depth image information of each view image. According to an aspect of the present invention, there is provided a method of encoding a multiview video, acquiring multiview image information and depth information of each view image, image information and depth information of any first view of the obtained multiview image, and Generating virtual information on the intermediate view of the first and second views using image information and depth information of the second view, which is the view closest to the first view, and encoding the virtual information.

Description

Multi-view video encoding and decoding method and apparatus for encoding and decoding using same

The present invention relates to a method and apparatus for providing free view video content in a multiview video codec, and more particularly, to synthesize an intermediate view image using a plurality of multiview images and depth image information of each view image. It is about a method.

Recently, due to the rapid development of multimedia processing technology and hardware technology, high-definition broadcasting service of higher than HD (High-Definition) level has become widespread. In addition, interest in 3D TV broadcasting using 3D stereoscopic contents is increasing to provide consumers with a better sense of reality and realism. Such 3D TV broadcasting is expected to be most noticed in the next generation broadcasting service market following HDTV. In this research, JVT (Joint Video Team of ISO / IEC JTC1 / SC29 / WG11 MPEG and ITU-T SG16 Q.6 VCEG) uses multi-view video encoding to efficiently encode images from two or more cameras. Multi-view Video Coding (H.264 / AVC) standard has been completed, and ISO / IEC MPEG, a video standardization organization, has implemented 3D video standard to provide users with various viewpoints and high realism. In the process. The 3D video standard is a standard for supporting a user to generate various virtual viewpoint images in addition to the viewpoint images received. A virtual-view image may be generated indefinitely through a view-interpolation method using a depth-map, which is a distance between a camera and a real object in the real world. Can be. Therefore, in the 3D video standard, only a small number of view image data and its depth map need to be transmitted, thereby saving bandwidth and storage space compared to the conventional multiview video encoding method.

In this method, the multiview image encoder encodes the depth map for synthesizing the intermediate view image together with the input multiview image, and the decoder synthesizes the intermediate view image of the desired view using the decoded image and the depth map. . In this case, there is a difference between the decoded image and the image of the viewpoint to be synthesized. Since the decoder cannot know the information about the difference, there are many difficulties in synthesizing the intermediate view image and the synthesized intermediate view image. There was also a problem with the quality.

In order to solve the above problem, the encoder generates and transmits information about an occlusion area and a non-occlusion area that are generated during the synthesis of an intermediate view image in advance, thereby reducing the amount of computation of the decoder. The purpose of this paper is to provide a method for synthesizing high-quality mid-point images with high reliability.

Another object of the present invention is to improve encoding efficiency by reducing the encoding bitrate of information about the occluded and non-occluded regions generated by the encoder.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a multi-view video encoding method, the method comprising: acquiring multi-view image information and depth information of each view image, and image information of any first view from the obtained multi-view image And generating virtual information on the intermediate view between the first and second views using depth information, image information and depth information of a second view, which is a view closest to the first view, and encoding the virtual information. It includes a step.

The generating of the virtual information may include projecting an image of the first view onto an image of the second view based on depth information of the first view. Generating information about an occlusion region and a non-occluded region of the image, and projecting the image of the second viewpoint to the image of the first viewpoint based on depth information of the second viewpoint; Generating information about the occluded area and the non-occluded area for the image of the second view is performed.

The generating of the information about the occlusion area and the non-occlusion area may include generating the occlusion area and the non-occlusion area by retrieving the image of the first or second view in an arbitrary coding unit. And generating information related to whether or not as a binary map.

In another embodiment of the present invention, the generating of the information about the occlusion area and the non-occlusion area may be performed by retrieving the image of the first or the second view point in an arbitrary coding unit to generate the occlusion area or the non-obstruction area. And generating information related to whether or not to map using 2 bits.

Also, in the embodiment of the present invention, the encoding of the virtual information may include continuously encoding the number of the same number in the raster scan order in the binary map.

In another embodiment of the present invention, the encoding of the virtual information may include sequentially encoding a number of the same number in the map using the two bits; And

Encoding an identifier for state translation.

In addition, according to an embodiment of the present invention, the encoded virtual information may be selectively added to an additional information area or an essential information area of a transport bitstream according to the type of an application that reproduces the multiview video information in an H.264 or HEVC codec. It is characterized in that the transmission.

Multi-view video decoding method according to another aspect of the present invention for achieving the above object is decoding the received multi-view image information, decoding the virtual information occlusion region (noncclusion) and non-obstruction (deisocclusion) for each viewpoint image Extracting information about an area, and generating image information of an intermediate view for each view by referring to the decoded image information and the information about the extracted occlusion area and the non-blocking area.

According to another aspect of the present invention, a multi-view video encoding apparatus according to another aspect of the present invention is a receiver for receiving multi-view image information and depth information for each view image from a plurality of cameras, and any arbitrary one of the received multi-view images. Virtual information for generating virtual information on the intermediate view between the first and second views using the image information and the depth information of the first view and the image information and the depth information of the second view which is the closest view to the first view. A generation unit and a virtual information encoding unit for encoding the virtual information.

In this example, the virtual information generator may project the image of the first view onto the image of the second view based on the depth information of the first view, and thus the image of the first view. Generate information about an occlusion area and a non-occlude area for the second view, and project the image of the second view onto the image of the first view based on the depth information of the second view. Generates information about occluded and non-occluded regions of the image.

Also, in another embodiment of the present invention, the virtual information generation unit may search for the image of the first or second view in an arbitrary coding unit to obtain information related to whether the occluded area and the non-occluded area occur. Binary map)

Also, in another embodiment of the present invention, the virtual information generator may search for the image of the first or second view in an arbitrary coding unit to obtain 2 bits of information related to whether the occluded area or the non-occluded area is generated. Create using map.

In this case, according to an embodiment of the present invention, the virtual information encoder continuously encodes the number of the same number in the raster scan order in the binary map.

In another embodiment of the present invention, the virtual information encoder continuously encodes the number of the same number in the map using the two bits, and encodes an identifier for state transformation.

According to another aspect of the present invention, an apparatus for decoding a multiview video according to another aspect of the present invention includes an image information decoder for decoding received multiview image information, an occlusion region for each viewpoint image by decoding virtual information; A virtual information decoder which extracts information about a non-occluded region, an intermediate to generate image information of an intermediate viewpoint for each viewpoint by referring to the decoded image information and the information about the extracted occluded region and the non-occluded region It includes a viewpoint image information generation unit.

Meanwhile, the multi-view video encoding method and the decoding method according to an embodiment of the present invention may be recorded as a program for execution by a computer and stored in a computer-readable recording medium.

As described above, according to the present invention, by receiving information about the occlusion area and the non-occlusion area generated by the encoder, the decoder can solve the problem of image overlap and empty space due to the occlusion area and the non-occlusion area when synthesizing the intermediate view image. This can improve the quality and reliability of the output mid-view image.

1 is a block diagram illustrating a configuration of an encoder of a multiview image according to an embodiment of the present invention.
2 is a flowchart illustrating a multiview image encoding method according to an embodiment of the present invention.
3 is an exemplary diagram illustrating an example of a method of encoding identifier information for state transformation in virtual information encoding in the multi-view image encoding method according to an embodiment of the present invention.
4 is a block diagram illustrating a configuration of a decoder of a multiview image according to an embodiment of the present invention.
5 is a flowchart illustrating a multi-view image decoding method according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, like reference numerals refer to like elements throughout. In the drawings, like reference numerals are used to denote like elements, and in the description of the present invention, In the following description, a detailed description of the present invention will be omitted.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It is to be understood that the invention may be embodied in other specific forms. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. It is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Hereinafter, a multiview video encoder according to an embodiment of the present invention will be described with reference to FIGS. 1 and 2. 1 is a block diagram illustrating a configuration of an encoder of a multiview image according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating a multiview image encoding method according to an embodiment of the present invention.

The present invention provides a video content providing method for supporting a multi-view 3D TV or a free-view TV service, wherein a service can be provided by image information of one view and depth information of one view associated therewith, and the multiview of two or more views. The service may be provided by the image information and the multi-view depth information.

That is, the present invention supports information provided by various video content providing services such as binocular 3D TV, multiview 3D TV, and free view TV service using the image information and depth information. The present invention generates virtual image information (closed region information and non-closed region information) in each viewpoint image by using a multiview image acquired from a multiview camera and the depth information of each viewpoint image, and transmits it to the decoder. The present invention provides a method for synthesizing a reliable and high quality image by reducing the amount of computation when the decoder synthesizes an intermediate view image by using the decoded multi-view image. A detailed configuration of a multi-view encoder according to an embodiment of the present invention for providing such a method will be described below.

Referring to FIG. 1, a multi-view video encoder according to an embodiment of the present invention includes a receiver 110, a depth information encoder 120, an image information encoder 130, a virtual information generator 140, and virtual information. The encoder 145 includes a multiplexer 150, a bitstream generator 160, and a transmitter 170.

The receiver 110 receives image information photographed by at least one multiview video camera, a stereo camera, a depth information camera, or the like, that is, depth information of a multiview image and each view image (S210).

The multi-view image can be taken while moving the camera by using one camera, or can be photographed by placing several cameras at various positions. In addition, instead of a general camera having a limited viewpoint, images of various directions may be acquired at a time through a special camera for acquiring a multiview image.

Depth information is a representation of the distance between the camera and the real object in the real world with a certain number of bits corresponding to each pixel (pixel) at the same resolution as the real image, and generally a map having the same size as the corresponding real image (depth Information map). In general, the depth information map is generated for all multi-view images and displayed in pixels, so the depth resolution is the same as the image resolution.

The virtual information generating unit 140 uses the image information and depth information of an arbitrary first viewpoint among the acquired multi-view images, and the image information and depth information of a second viewpoint which is the closest to the first viewpoint. Virtual information about an intermediate time point between the first and second time points is generated (S220).

For example, the receiver 110 acquires depth information of the V0, V2, and V4 viewpoint images and each viewpoint image photographed at different viewpoints at the same time, and the virtual information generation unit 140 uses the intermediate viewpoint image to obtain depth information. Assuming that the virtual information for the V1 and V3 view images is generated, the virtual information generator 140 generates the virtual information for the V1 view, the V0 view image, the depth image information, and the V2 view image. And depth information thereof, and in the same manner, the V2 view image and the depth information thereof, and the V4 view image and the depth information thereof are used for the V3 view.

In detail, when the virtual information generator 140 generates virtual information about the V1 viewpoint, the virtual information generator 140 projects the image of the V0 viewpoint to the image of the V2 viewpoint based on the depth information of the V0 viewpoint. In this case, the projection of the image may be performed by a basic coding unit, and since the displacements projected by each coding unit vary depending on the depth information, they do not all have the same value. Therefore, because images of all coding units are not uniformly moved with the same displacement but are moved with different displacements, in some cases, regions where different coding unit images are moved to the same position (hereinafter, occlusion region). This may occur, or an area (disocclusion area) in which no coding unit image is moved may occur.

In this case, the coding unit serving as the basic unit of the occlusion area and the non-occlusion area search may be a pixel unit, or more accurate virtual information may be collected through a precise search of a half pixel and a quarter pixel unit.

For example, the virtual information generator 140 searches in units of pixels and determines a closed region and a non-closed region based on the distance traveled by the pixel B closest to the arbitrary pixel A. If the A pixel and the B pixel move to the same position or move to a pixel within the septic tank, the virtual information generating unit 140 belongs to the occlusion area. On the contrary, the A pixel and the B pixel move. If the displacement has a distance greater than or equal to the septic tank, it can be determined that this region belongs to the non-occluded region.

The present invention is characterized in that the decoder generates and transmits reference information in advance so as to improve the efficiency in synthesizing the intermediate view image, wherein the reference information is the projection of the adjacent view for the virtual intermediate view synthesis as described above. Includes information about occlusion and non-occlusion areas that occur during

The virtual information generated by the virtual information generating unit 140 is transmitted to be decoded. In this case, the location of the virtual information is defined so that the encoder and the decoder can recognize each other through an appointment or an identifier. Here, the virtual information may be generated as a binary map related to whether the occluded area and the non-occluded area are generated.

In one embodiment, the virtual information generating unit 140 does not distinguish between the occluded region and the non-occluded region as a result of the search, and the region is determined to be a region (occluded region, non-occluded region) where problems may occur when synthesizing the intermediate image. A binary map is generated by marking '1', and marking '0' in a region that is not abnormal.

In addition, the virtual information may be generated as a map using two bits related to whether a closed region or a non-closed region is generated. For example, a map using two bits of '0', '1', and '2' is configured to distinguish between the occlusion area and the non-occlusion area. In this case, '0' is no abnormality, '1' is the occlusion area, ' 2 'is divided into non-occluded areas.

The virtual information encoder 145 encodes the generated virtual information (S230). In this case, the amount of bits transmitted through various methods can be minimized. As an example, when the virtual information is generated as a binary map relating to whether a closed region and a non-closed region are generated, the number '0' is generated, '1' Only numbers with 'are consecutively encoded in the raster scan order.

Example) Binary Map 0000000000000000011111111111111110000000000011000000000

Coding 17-16-11-2-....

In another embodiment, when the virtual information is generated as a map using two bits related to whether an occluded area or a non-occluded area is generated, the conversion information of three states, i. ) Is encoded.

3 is an exemplary diagram illustrating an example of a method of encoding identifier information for state transformation in virtual information encoding in the multi-view image encoding method according to an embodiment of the present invention.

Referring to FIG. 3, a method in which the virtual information encoder 145 encodes virtual information will be described.

First, the virtual information encoder 145 encodes consecutive numbers. As in the previous example, if 17 consecutive 0s are found, 1 is used to encode 17, and the remaining 1 bits are used to encode identification information of whether the number after 0 is 1 or 2. .

The identification information is a promised rule between the encoder and the decoder, and may include information capable of identifying a state change in a clockwise or counterclockwise direction by arranging 0 / 1/2 in a circle as shown in FIG. Can be. For example, 0 may indicate counterclockwise and 1 may indicate clockwise state change information.

 The depth information encoder 120 encodes depth information of each viewpoint image acquired by the receiver 110 (S230).

The encoding of the depth information may be H.264 (or MPEG-4 Part 10 Advanced Video Coding), which is a DCT-based video encoding method used for encoding a real picture. The DCT-based video encoding method is a block-based transform method, which is efficient in a low spatial frequency image (high correlation image between pixels), whereas a high spatial frequency image such as an object boundary part (image with large change between neighboring pixels). ), In particular, when encoding is performed using a DCT-based video encoding method in a low bit rate environment, an object boundary cannot be accurately represented due to loss of high frequency components due to quantization, and the image is crushed. There is a problem. To this end, in the embodiment of the present invention, when the depth information is encoded, the depth information may be hierarchically encoded according to the characteristics of the depth information, that is, the importance of the depth information. In this case, the present invention may encode the depth information using a spatial scalable or signal to noise ratio (SNR) layer algorithm.

The image information encoder 130 encodes a multiview image acquired by the receiver 110 (S230).

For example, in the present invention, when the multiview image is encoded, the multiview image may be encoded using correlation information of a time axis and a space axis (view).

That is, the motion information on the time axis and the disparity on the spatial axis are predicted to perform motion and disparity compensation to be encoded. At this time, in consideration of compatibility with the existing mono image information encoding technology, one view is encoded using only motion information without using information of another view on a spatial axis as a base view, and the base view is encoded. Only image information of the remaining viewpoints is encoded using information on the spatial axis.

For example, encoding for the base view may apply H.264 encoding as a DCT-based encoding method. In H.264 encoding, an intra mode for performing only intra prediction and inter mode encoding for performing both intra prediction and inter prediction. In general, H.264 encoding is performed in macroblock units, which are basic units of encoding. First, a prediction block for an input macroblock is generated, and then a difference between the input macroblock and a prediction block is obtained and the difference information is obtained. Encode

The encoded depth information, the encoded image information, and the encoded virtual information are subjected to a multiplexing process through the multiplexer 150 (S240). As described above, according to the present invention, the coded depth information, the coded image information, and the coded virtual information which have been subjected to the multiplexing process are generated as respective bitstreams by the bitstream generator 160 (S250). The transmission unit 170 is transmitted to the decoder side (S260).

Here, the virtual information about the occlusion area and the non-occlusion area is included in an additional information area such as Supplemental Enhanced Information (SEI) in order to be compatible with a codec such as H.264 or HEVC. In addition, the virtual information may be included in an essential information area such as a sequence parameter, a picture parameter, a slice header, and transmitted according to a characteristic of an application program playing the multiview image.

Hereinafter, a multiview video decoding method and apparatus according to another embodiment of the present invention will be described with reference to FIGS. 4 and 5. 4 is a block diagram illustrating a decoder of a multiview image decoder according to an embodiment of the present invention, and FIG. 5 is a flowchart illustrating a multiview image decoding method according to an embodiment of the present invention.

As shown in FIG. 4, the multi-view image decoder according to another embodiment of the present invention includes a demultiplexer 410, an image information decoder 430, a depth information decoder 420, and a virtual information decoder 440. The intermediate view image generator 450 is included.

First, the demultiplexer 410 receives decoded depth information, image information, and virtual information to demultiplex (S510).

The depth information decoder 420 decodes the demultiplexed depth information.

The image information decoder 430 decodes the demultiplexed image information. In detail, the image information decoder 430 may decode the encoded bitstream using the correlation information on the space-time axis. In this case, the image information decoder 430 may use the decoded depth information of the corresponding viewpoint and the image information of another viewpoint already decoded.

The virtual information decoder 440 decodes the demultiplexed virtual information. In detail, the virtual information decoding unit 440 decodes the virtual information and extracts information about an occlusion region and a non-occluded region for each viewpoint image (S520).

The mid-view image generator 450 generates image information of the mid-view for each of the viewpoints by referring to the decoded image information and the information about the extracted occlusion region and the non-occluding region (S530).

Meanwhile, the encoding and decoding method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Examples of program instructions, such as magneto-optical and ROM, RAM, flash memory and the like, can be executed by a computer using an interpreter or the like, as well as machine code, Includes a high-level language code. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It is to be understood that the invention may be embodied in other specific forms. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

Claims (16)

delete Obtaining multi-view image information and depth information on each view image;
An intermediate viewpoint between the first and second viewpoints by using image information and depth information of an arbitrary first viewpoint among the acquired multiview images, and image information and depth information of a second viewpoint, which is a view closest to the first viewpoint. Generating virtual information about the network; And
Encoding the virtual information;
Generating the virtual information,
The image of the first view is projected onto the image of the second view based on the depth information of the first view, and the occlusion area and the non-occluded area of the image of the first view. Generating information; And
Projecting the image of the second view onto the image of the first view based on the depth information of the second view to generate information about the occluded and non-occluded areas of the image of the second view.
Multiview video encoding method. The method of claim 2, wherein the generating of the information about the occluded area and the non-occluded area comprises:
Retrieving the image of the first or second view in an arbitrary coding unit and generating information related to whether the occluded region and the non-occluded region are generated as a binary map.
Multiview video encoding method. The method of claim 2, wherein the generating of the information about the occluded area and the non-occluded area comprises:
Retrieving the image of the first or second view in an arbitrary coding unit and generating information related to whether the occluded region or the non-occluded region is generated as a map using 2 bits.
Multiview video encoding method. The method of claim 3, wherein the encoding of the virtual information comprises:
Continuously encoding the number of the same number in the raster scan order in the binary map.
Multiview video encoding method. The method of claim 4, wherein the encoding of the virtual information comprises:
Continuously encoding the number of the same number in the map using the 2 bits; And
Encoding an identifier for state translation
Multiview video encoding method. The method of claim 2, wherein the encoded virtual information,
Depending on the type of the application that reproduces the multi-view video information in the H.264 or HEVC codec, it is optionally included in the additional information area or the essential information area of the transport bitstream and transmitted.
Multiview video encoding method. Decoding the received multiview image information;
Extracting information about an occlusion area and a non-occlusion area for each viewpoint image by decoding the virtual information of claim 2;
Generating image information of an intermediate view for each of the viewpoints by referring to the decoded image information and the information about the extracted occlusion area and the non-obstruction area.
Multi-view video decoding method comprising a. delete A receiver configured to receive multi-view image information and depth information of each view image from a plurality of cameras;
An intermediate viewpoint between the first and second viewpoints using image information and depth information of an arbitrary first viewpoint among received multiview images, and image information and depth information of a second viewpoint, which is a view closest to the first viewpoint. A virtual information generator for generating virtual information on the virtual machine; And
Including a virtual information encoder for encoding the virtual information,
The virtual information generation unit,
The image of the first view is projected onto the image of the second view based on the depth information of the first view, and the occlusion area and the non-occluded area of the image of the first view. Generate information,
Projecting the image of the second view onto the image of the first view based on the depth information of the second view to generate information about the occluded and non-occluded areas of the image of the second view
Multiview video encoding device. The method of claim 10, wherein the virtual information generating unit,
Retrieving the image of the first or second view in an arbitrary coding unit and generating information related to whether the occluded area and the non-occluded area are generated as a binary map.
Multiview video encoding device. The method of claim 10, wherein the virtual information generating unit,
Retrieving the image of the first or second view in an arbitrary coding unit and generating information related to whether the occluded area or the non-occluded area is generated as a map using 2 bits;
Multiview video encoding device. The method of claim 11, wherein the virtual information encoder,
Successively encoding the number of the same number in the raster scan order in the binary map
Multiview video encoding device. The method of claim 12, wherein the virtual information encoder,
Continuously encoding the number of the same number in the map using the 2 bits, and encoding an identifier for state transformation.
Multiview video encoding device. The method of claim 12, wherein the virtual information encoder,
Continuously encoding the number of the same number in the map using the 2 bits, and encoding an identifier for state transformation.
Multiview video encoding device. A computer-readable recording medium having recorded thereon a program for executing the method of claim 2.

Images