KR20110132704A - Encoding apparatus and method for 3d image - Google Patents

Abstract

PURPOSE: A 3D image encoding apparatus and method thereof are provided to offer a real 3D image to a user by extracting the hiding area of the image. CONSTITUTION: A difference estimation module(210) includes an edge extraction unit(211), an object identification unit(212), and a depth information estimation unit(213). The edge extraction unit extracts the edge of the image. The object identification unit distinguishes an object. The object identification unit uses a modeling method. The depth information estimation unit estimates depth information. A radio wave graph model is used for estimating the depth information.

Description

Encoding Apparatus and Method for 3D Image}

The present application relates to the encoding of stereoscopic images, and more particularly, to an apparatus and method for stereoscopic image encoding using stereoscopic technology.

The development of 2D display technology has already been completed, and consumers have not noticed a big difference in display quality or thinning of the display. In comparison, 3D display technology has not yet reached the level of completion compared to 2D flat panel displays.

3D display technology is a technology that displays 3D images that are realistically formed instead of a planar image formed by a conventional 2D flat panel display, and is a technology that expresses natural image information closest to a natural landscape or an object that we see and feel everyday. 3D displays are used in various fields such as broadcasting, medical, aviation, military, animation, and film, and are next generation high value-added technologies.

There are numerous types of 3D displays currently in commercialization or research. There are many classification schemes in 3D displays, and no classification foundation has yet been created that describes all areas. However, according to the implementation method, stereoscopic image is formed in space by using stereoscopic copying, variable focusing mirror or rotating screen, which gives the viewer a sense of depth by providing images with parallax between left and right eyes. It can be classified into a volumetric display method and a holographic display method using interference of light, and the binocular parallax method can be divided into two categories according to the use of auxiliary glasses.

Among these, in the case of the stereoscope technology first introduced in 1838, two images (Stereoscopic Image) corresponding to the input of the left and right eyes are acquired, and the technology for allowing the respective images to be viewed through the left and right eyes. to be. 3D technology using stereoscopes has been developed in many countries, and technology improvements and developments have been made.

In relation to this, stereo matching technology is a vision technology for restoring three-dimensional depth information of a camera looking at image information obtained through cameras of different positions. The depth of a scene can be obtained by calculating a binocular difference between two pixels by finding a matching pixel between two images acquired at different positions.

There are two methods of stereo matching: local matching and global matching. Stereo Propagation (BP) -based stereo matching technology is a global matching technology that determines the binocular difference of pixels using the information of the whole image, and shows robustness to the obscuration phenomenon that appears in stereo images. Even less results show excellent results.

Despite these many advantages, reliable propagation-based stereo matching effectively responds to lighting differences caused by shadows caused by partial obstruction, the relative position of the lights and the camera, and the physical differences between the apertures of the two cameras. There are problems such as not being able to.

Accordingly, the present invention has been proposed to solve this problem, and is a method for 3D image processing using a stereoscope, and further extracts and adds edge information of an image to a blue band before performing image matching using trust propagation. It is an object of the present invention to provide an encoding apparatus and method for providing realistic stereoscopic images.

In the encoding method of the stereoscopic stereoscopic image according to an embodiment of the present invention, generating a depth map by estimating a difference between a first image and a second image, compressing the first image, the generated depth map Compressing the; and performing channel encoding on the compressed first image and the compressed depth map.

The generating of the depth map may include extracting an edge of the input image, performing object discrimination on the image from which the edge is extracted, and estimating depth information on the image classified by the object.

The extracting of the edge of the input image may include applying an edge to the blue band of the input image.

The first image is a right image, and the second image is a left image.

The encoding method also compares the brightness of the first image and the brightness of the second image and estimates the difference between the first image and the second image and generates the depth map, based on the first image. And adjusting the brightness of the second image.

According to another aspect of the present invention, there is provided a stereoscopic stereoscopic image system comprising: a difference estimating module for generating a depth map by estimating a difference between a first image and a second image; a first compression module compressing the first image; And an encoder including a second compression module to compress a depth map and a channel encoder to perform channel encoding on the compressed first image and the compressed depth map.

The stereoscopic imaging system may include a channel decoder configured to receive data transmitted from the encoder and to decode a transmission channel, a first decoder to decode a first image output from the channel decoder, and a difference map output from the channel decoder. And a decoder including a second decoder to decode the stereoscopic image reproducing unit to reproduce a stereoscopic image using data output from the first decoder and the second decoder.

According to another aspect of the present invention, a method of generating a depth map includes extracting an edge of an input image, performing object discrimination on the image from which the edge is extracted, and estimating depth information on the image classified by the object. Steps.

The extracting of an edge of the input image may include applying an edge to a blue band of the input image.

According to another aspect of the present invention, an apparatus for generating a depth map extracts an edge of an input image, performs object classification on the image from which the edge is extracted, and extracts depth information on the image classified by the object. Extraction of the edges is characterized by applying the edges to the blue band.

According to the encoding apparatus and the method according to the preferred embodiments of the present invention, in particular, the boundary of the image registration can be improved in the extraction of unclear or hidden regions.

1 shows an overall system of an encoder and a decoder for a three-dimensional image in accordance with an embodiment of the present invention.
2 is a block diagram of a difference estimation module according to an embodiment of the present invention;
3 is a diagram illustrating a standard depth information image that is depth information of an original image.
4 is a diagram illustrating a resultant image when depth information estimation using an object discrimination method and confidence spreading using MRF is performed according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a result image when edge extraction is additionally performed before performing depth information estimation using an object discrimination method and confidence spreading using MRF according to an embodiment of the present invention. FIG.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is said to be "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that another component may exist in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same elements in the drawings, and redundant description of the same elements is omitted.

1 shows an overall system of an encoder and a decoder for three-dimensional images according to an embodiment of the present invention.

As shown in FIG. 1, a right image and a left image are input from two cameras for 3D image encoding and decoding according to an embodiment of the present invention. Our two eyes have an average distance of about 65mm in the horizontal direction, and the stereoscopic technique uses the binocular parallax effect. The left and right eyes are fused with different two-dimensional images to reproduce the three-dimensional, depth and realism of the original three-dimensional stereoscopic image. The right and left images input from the two cameras correspond to the images input to the human eyes.

Image data transmitted through the brightness control module 110, the compression module 120, and the channel encoder 100 only through the channel decoder 300 and the image decoders 310 and 320 on the decoder side is reproduced only for the right image. You get a 2D image.

The left image is added, and stereoscopic related blocks such as the difference estimation module 210, the depth map 220, and the difference map decoder 320 for processing the difference between the right image and the left image are added to obtain a 3D stereoscopic image. Done.

The encoder for the 3D image according to an embodiment of the present invention, the brightness control module 110, the compression module 120, the channel encoder 100, and the difference estimation module 210, the depth map 220, and And a compression module 113 for the depth map. In addition, the decoder for a 3D image according to an embodiment of the present invention includes a channel decoder 300, a right image decoder 310, and a difference map decoder 320.

First, a detailed function of an encoder according to an embodiment of the present invention will be described.

The right image and the left image shown in FIG. 1 are commonly input to the brightness adjusting module 110. The brightness control module 110 adjusts and outputs brightness that varies on the right image and the left image according to shadows, lighting, and the like. The brightness control module 110 adjusts the brightness based on the right image. If the brightness is different from each other compared with the left image based on the right image, an algorithm for correcting the left image is operated to correct the left image. Therefore, the corrected left image and right image The difference estimation module 210 is input, and the right image is input to the compression module 120.

The compression module 120 for the right image compresses the image using several techniques used for data compression. The compression module 120 may be, for example, a compression module of a type such as JPEG2000, JPEG, JPEG XR.

The JPEG compression module uses DCT conversion and is a commonly used compression method. As a feature, the compression may be performed in units of 4 * 4 blocks. In other words, compression is performed using 4 * 4 matrix or 8 * 8 pixel value. The disadvantage is that the pixels in the middle cause a blocking phenomenon.

The compression module according to JPEG2000 has a feature of using global compression. In this case, the compression module performs a compression process including discrete wavelet transform, quantization, and entropy encoding. Discrete wavelet transform is applied to a tile component generated by decomposing an image by each component, and then again decomposing it into square tiles. Coefficients transformed by wavelet transform, which describe horizontal and vertical spatial frequency characteristics of tile components, are subjected to entropy encoding before quantization and generation of code sequences.

One of the preferred embodiments of the compression module according to the invention is a compression module using lapped transformation. Currently used in JPEG XR. Wrapped transform is a method of transform coding among the techniques for reducing image information, and it divides into partitions of a certain size and converts them into partitions. Therefore, a blocking effect caused by discontinuity of the edge portion of a partition is generated when the data rate is low. That's how it can be. The wrapped transform uses a block transform but reduces the blocking phenomenon in an overlapping manner of blocks.

The difference estimation module 210 is a block for estimating the difference between the right image and the right image, that is, the binocular difference. The difference estimation module 210 according to an embodiment of the present invention is responsible for generating and extracting a depth map 220 by extracting edges of an image to distinguish objects and estimating depth information. That is, the difference estimation module 210 according to an embodiment of the present invention finally outputs the depth map 220.

In particular, the difference estimation module 210 according to an embodiment of the present invention extracts and adds edge information of an image before performing image registration using confidence propagation (BP), thereby extracting an area where the boundary of the image registration is unclear or hidden. Improved effect can be obtained. The difference estimating module according to the preferred embodiment of the present invention is characterized by applying a dpt sheet to a single band, particularly B band, for edge information extraction. A detailed description will be made with reference to FIG. 2.

The depth map 220 generated by the difference estimation module 210 is subjected to a compression process through the compression module 113. That is, the compression module 113 is a compression module that receives and processes the depth map as an input value. The compression module 113 for the depth map may use one of the transforms, the wavelet, the DCT, the lap transform, or other forms of compression as described above in the compression module 120 for the right image. In addition, the compression scheme used in the compression module 113 for the depth map and the compression module 120 for the right image may be the same or different depending on whether the compression scheme is used.

Finally, the compressed right image and the depth map are prepared for transmission through the channel encoder 100 and are transmitted to the decoder side. The data received at the decoder side is separated into the right image data and the difference map data of the uncompressed form through the channel decoder 300, the right image is decoded by the right image decoder 310, and the difference map is the difference map. It is decoded by the map decoder 320. The image data decoded by the right image decoder 310 becomes a 2D image, that is, a 2D image, and a 3D stereoscopic image may be obtained by adding the difference map decoded by the difference map decoder.

2 illustrates a detailed block of a difference estimation module according to an embodiment of the present invention.

The difference estimation module 210 according to an embodiment of the present invention may include an edge extractor 211, an object discriminator 212, and a depth information estimator 213.

Depth estimation in stereoscopic systems can estimate depth information using a confidence propagation (BP) graph model. The confidence propagation method repeatedly estimates optimized state values by repeatedly exchanging probabilistic information with each neighboring node on a two-dimensional graph model. In this case, the function for the stochastic information is defined as a message function and is composed of a data value, a smoothness value, and a neighbor node.

In general, the order of image matching using MRF (Markov Random Fields) to find the maximum value and match the image based on the message format is a trust propagation method. According to an exemplary embodiment of the present invention, before performing the trust propagation method, The edge extracting unit 211 is added as a preprocessing process. In this preprocessing process, edge information is added to create an advantageous condition for extracting an unclear or hidden region. Edge extraction may be applied to, for example, a Prewitt or Sobel edge.

Here, the digital image typically uses a RGB color model. In the abstract mathematical model, when a particular color is represented as an ordered pair with several components, the color space is represented by an ordered pair of values (usually three or four). The most common form is the color space. RGB color space. Image input devices such as cameras and scanners store light intensity as RGB data, and image output devices such as displays and printers display RGB data in color. However, the same object is stored as different RGB data for each input device, and different colors are reproduced for each output device for the same RGB data.

RGB images have red, green, and blue samples for each pixel. Digital images combine these three primary colors to represent the color of each pixel. A specific set of samples is called a band or channel. In this case, the blue band refers to the blue band for all pixels in the image, which is often referred to as a 0 band or a B band.

On the other hand, an object with a small pattern in the image has a small edge inside the object and exhibits strong edge characteristics only at the interface. If edges of B, G, and R are all edged due to the characteristics of bands, many segments are generated in MRF, which is an adverse condition in image registration. Therefore, in an exemplary embodiment of the present invention, the edge is applied to a single channel band without applying the edge to all the bands of B, G, and R. As a result of experiments applying edges to each of B, G, and R bands, it can be seen that the most reliable edge components can be obtained by applying edges to B bands.

Therefore, in the edge extraction according to the preferred embodiment of the present invention, the edge is applied to a single band, inter alia, the B band. After that, object discrimination and depth information estimation are sequentially applied.

The object discrimination unit 212 according to an embodiment of the present invention uses a modeling technique such as, for example, MRF.

Understanding of image consists of segmentation and labeling. Image segmentation is the process of segmenting an image into regions with uniform properties such as contrast, color, texture, and distance information. It is a process of recognizing each divided region by using knowledge of the model and scene of the object, the properties of the divided regions and the spatial relationship between them. In other words, image labeling can be defined as an optimization process that distinguishes all divided regions as the most suitable objects using constraints derived from knowledge of the scene, and MRF is one of the methods of effectively naming images.

The MRF model is defined on the region adjacency graph composed of the segmented region of the image as each node and based on the spatial proximity between them. By separating the basic regions from a given image and forming adjacent graphs according to their spatial adjacencies, the interpretation of the image is achieved by labeling each node in the graph with the correct object. According to MRF modeling according to the present invention, each object is classified, that is, labeled according to the color of the object.

The image distinguished by the object discriminating unit 212 is input to the depth information estimating unit 213. An example of a preferable estimation method used in the depth information estimating unit may be a confidence propagation (BP).

Here, the trust propagation technique is a method of estimating an optimized state of each node on a two-dimensional graph or a two-dimensional Markov network composed of nodes and edges, and estimates an optimized state of each node. Trust propagation assumes that the state values of neighboring nodes change smoothly and spatially on the graph, and performs iteration for exchanging information with neighboring nodes based on this assumption. A value that models the probabilistic effect between nodes in an iteration process is called a message, and trust propagation exchanges this message with neighbors through iteration. Through this iterative node, nodes transmit their probability information to other nodes and reflect the information of other nodes in their state estimation.

As described above with reference to FIG. 1, the depth information estimator 213 finally outputs the depth map 220.

Hereinafter, the results of the difference estimation according to the preferred embodiment of the present invention will be described with reference to FIGS. 3 to 5. In order to facilitate the result comparison, the images used in the processing according to FIGS. 3, 4, and 5 all use the same image.

3 illustrates a standard depth information image that is depth information of an original image.

The resultant image of FIG. 3 illustrates a resultant image output when the input image is processed according to a depth information extraction method using a conventional 3D modeling tool, and may be referred to as standard depth information. The original result depth map used as a reference is typically generated manually by graphic experts using 3D modeling tools.

As can be seen from the result image representing the standard depth information, the depth information related to the bookshelf or the book appearing in the background behind the original image is almost indistinguishable from the result image. The information contained in the depth map represents the depth of an object or an object, so the information on the depth map is shown in FIG. 3.

4 illustrates a result image when the object discrimination method using MRF and the depth information estimation using trust spreading are performed according to an embodiment of the present invention.

In FIG. 4, the result image when the object discriminating unit 212 of FIG. 2 uses the MRF technique and the trust propagation technique is used by the depth information estimating unit 213 of FIG. 2 is illustrated.

FIG. 5 illustrates a result image when edge extraction is additionally performed before performing object discrimination method using MRF and depth information estimation using trust spread according to an embodiment of the present invention.

The process of FIG. 5 further includes an edge extraction step as compared to the case of FIG. 4. Edge extraction is compared with the operation of the detailed block of the difference estimation module of FIG. 2, the original image is input to the difference estimation module 210, and the result output by the edge extraction unit 211 of FIG. 2 is shown in the middle of FIG. 5. You can check it in the video. Here, the edge extraction according to an embodiment of the present invention is applied to the B band, that is, the blue band, which can be confirmed through the sharp edges indicated in blue in the middle image of FIG. 5. The resultant image of FIG. 5 shows a case where a Sobel edge is used. However, it has already been mentioned that other types of edges may be applied instead of the Sobel edge, for example, the Prewitt edge. The processing after edge extraction is the same as the MRF and BP processing applied to FIG.

Comparing the resultant image after completing all the processing in FIG. 5 with the resultant image of FIG. 4, it is possible to confirm the improved result at the edge of the object as a whole. In particular, the three parts displayed in the circular shape in the resultant image show a marked improvement over the resultant image of FIG. 4.

Although described with reference to the above embodiments, those skilled in the art will understand that various modifications and changes can be made without departing from the spirit and scope of the invention as set forth in the claims below. Could be.

In particular, the JPEG, JPEG2000, JPEG XR, etc. as an embodiment in the present specification is a still image as an object, but the present invention is one embodiment only and excludes the case of MPEG for a video. It does not mean to. Since it is obvious to those skilled in the art that each of the numerous frame images forming the moving image is one still image, embodiments of the present invention may be similarly applied to moving image processing. In addition, the brightness control module 110 and the like will be said to be applicable to the imaging device such as a camera.

Claims (10)

In the stereoscopic stereoscopic video encoding method,
Generating a depth map by estimating a difference between the first image and the second image;
Compressing the first image;
Compressing the generated depth map;
Performing channel encoding on the compressed first image and the compressed depth map. The method according to claim 1,
Generating the depth map,
Extracting an edge of an input image;
Performing object discrimination on the image from which the edge is extracted; And
Estimating depth information with respect to the image classified by the object. The method according to claim 2,
Extracting the edge of the input image,
And applying edges to the blue band of the input image. The method according to claim 1,
The first image is a right image, and the second image is a left image. The method according to claim 1,
Before estimating the difference between the first image and the second image to generate a depth map,
Comparing the brightness of the first image with the brightness of the second image, and adjusting the brightness of the second image based on the first image. Stereoscopic stereoscopic imaging system,
A difference estimation module for estimating a difference between the first image and the second image to generate a depth map;
A first compression module to compress the first image;
A second compression module for compressing the generated depth map
And an encoder including a channel encoder to perform channel encoding on the compressed first image and the compressed depth map. The method of claim 6,
A channel decoder for receiving data transmitted from the encoder and performing decoding on a transmission channel;
A first decoder for decoding a first image output from the channel decoder;
A second decoder for decoding the difference map output from the channel decoder; And
And a decoder including a stereoscopic image reproducing unit configured to reproduce a stereoscopic image using data output from the first decoder and the second decoder. Extracting an edge of an input image;
Performing object discrimination on the image from which the edge is extracted; And
Estimating depth information on an image classified by objects. The method according to claim 8,
Extracting the edge of the input image,
And applying an edge to the blue band of the input image. Depth map generator,
Extract edges of the input image, perform object classification on the image from which the edge is extracted, and extract depth information on the image classified by object,
And extracting the edges applies edges to a blue band.

Images