KR101232870B1 - Stereoscopic image processor and method for processing image using the same - Google Patents

Abstract

Disclosed are a stereoscopic image processing apparatus and a method thereof. The stereoscopic image processing apparatus according to an embodiment of the present invention comprises a synthesizer for synthesizing a stereoscopic image, and converting the synthesized stereoscopic image into a Y-YV format image by combining the synthesized stereoscopic image for each pixel or line with similarity of color and And an image processing unit for restoring the weaved-format image to the Algivy image without color loss, and a display unit for displaying the restored Algivy image. Accordingly, YUV conversion and RGB restoration of stereoscopic images can be performed without color loss, method change, and information amount change.

Description

Stereoscopic image processing device and method {Stereoscopic image processor and method for processing image using the same}

One aspect of the present invention relates to an image processing technology, and more particularly, to a technology for processing and displaying stereoscopic images.

Human vision is highly sensitive to luminance and less sensitive to color (color). Therefore, even if the color is replaced with the color of the adjacent pixel, the subjective quality of the color of the image does not have a big difference. The YUV 4: 2: 2 conversion or 4: 2: 0 conversion method is widely used as an image format for image transmission and image compression in that image information can be reduced by downscaling color information horizontally or vertically. However, in the case of a stereoscopic image, adjacent images are different because the two images are merged by line or pixel. Therefore, when downscaling and restoring the RGB image in a general manner, serious color loss occurs compared to the original image.

In general, as shown in FIG. 2, image compression and image transmission reduce the color information by horizontally downscaling 1/2. Alternatively, as shown in FIG. 3, color down is reduced by 1/2 downscaling horizontally and vertically. Through the above-described method, color information can be reduced by 1/2 or 1/4 during image compression and image transmission. The reduced color information is compressed or transmitted in BT.656 / 601 or YUV planner format. The above-described method is applicable to one image.

However, in the case of a stereoscopic image in which two images are mixed into one image, color loss may occur depending on the mixing method. For example, color loss occurs when a 4: 2: 2 conversion or a YUV 4: 2: 0 conversion method is applied to a line-based or pixel-based method commonly used for mixing stereoscopic images. The line-based method is a method of generating a single image by mixing a mixed image for each line as shown in FIG. 4. The pixel-based method is a method of generating a single image by alternately mixing a mixed image for each pixel, that is, for each column, as shown in FIG. 7.

When the general YUV 4: 2: 2 conversion or 4: 2: 0 conversion is applied to the above-described two-dimensional stereoscopic image, color loss occurs. In the case of using the line based method, since there is a different picture for each line as shown in FIG. 4, there is no similarity in color between lines. As shown in FIG. 6, when the general YUV 4: 2: 0 method is applied, RGB information is not restored to the original image because color information of odd and even lines is used regardless of color information of another image. .

In the case of using the pixel-based method, since there is a different picture for each pixel as shown in FIG. 7, there is no similarity in color in the horizontal direction. In the general YUV 4: 2: 2 conversion and 4: 2: 0 conversion methods, since only the color information of one image is obtained as shown in FIGS. 8 and 9, the RGB reconstruction to the original image is not performed.

According to one aspect, a technique for YUV conversion and RGB reconstruction of stereoscopic images without color loss, scheme change, and information amount change is proposed.

According to an aspect, an apparatus for processing a stereoscopic image includes: a synthesizer for synthesizing a stereoscopic image, and converting the synthesized stereoscopic image into a Y-YV format image by combining each pixel or line with similarities in color and converting the Y-YV format And an image processing unit for restoring the image to the Algivy image without color loss, and a display unit for displaying the restored Algibi image.

A stereoscopic image processing method of a stereoscopic image processing apparatus according to another aspect includes synthesizing a stereoscopic image, converting the synthesized stereoscopic image into a Y-YV format image by combining pixels or lines with similarities in color Restoring the stereoscopic image to the Algivy image without color loss, and displaying the restored Algivy image.

According to an embodiment of the present disclosure, a color loss problem generated during transmission and compression of a pixel-based and line-based stereoscopic image during YUV conversion may be minimized. Furthermore, the stereoscopic image can be compressed and transmitted while maintaining the same method and amount of information as before, and RGB restoration can be performed without losing color. Furthermore, the present invention can be applied to a YV Planar format or a color space domain that separates luminance information and color information, such as YCbCr.

1 is a diagram illustrating an image obtained by converting a 2D RGB image by a YUV 4: 4: 4 conversion method;
2 is a diagram illustrating an image obtained by converting a 2D RGB image by a YUV 4: 2: 2 conversion method;
3 is a view illustrating an image obtained by converting a 2D RGB image by a YUV 4: 2: 0 conversion method;
4 is a diagram illustrating an image obtained by converting a mixed RGB image by a YUV 4: 4: 4 conversion method based on a general line-based method;
5 is a diagram illustrating an image obtained by converting a mixed RGB image by a YUV 4: 2: 2 conversion method based on a general line-based method;
6 is a diagram illustrating an image obtained by converting a mixed RGB image into a YUV 4: 2: 0 conversion scheme based on a general line-based scheme.
7 is a diagram illustrating an image obtained by converting a mixed RGB image into a YUV 4: 4: 4 conversion scheme based on a general pixel-based scheme.
8 is a diagram illustrating an image obtained by converting a mixed RGB image into a YUV 4: 2: 2 conversion scheme based on a general pixel-based scheme.
9 is a diagram illustrating an image obtained by converting a mixed RGB image into a YUV 4: 2: 0 conversion scheme based on a general pixel-based scheme.
10 is a view showing the configuration of a stereoscopic image processing apparatus according to an embodiment of the present invention;
11 and 12 illustrate an embodiment of converting a stereoscopic image into a YUV format image and restoring the same based on a pixel-based method according to an embodiment of the present invention;
13 and 14 illustrate an embodiment of converting a stereoscopic image into a YUV format image and restoring the same based on a line-based method according to an embodiment of the present invention;
FIG. 15 illustrates an embodiment of converting a stereoscopic image into a YUV format image and restoring the same based on a line based method according to another embodiment of the present invention; FIG.
16 is a flowchart illustrating a stereoscopic image processing method according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention; In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the terms described below are defined in consideration of the functions of the present invention, and this may vary depending on the intention of the user, the operator, or the like. Therefore, the definition should be based on the contents throughout this specification.

10 is a diagram illustrating a configuration of a stereoscopic image processing apparatus 10 according to an embodiment of the present invention.

Referring to FIG. 10, the stereoscopic image processing apparatus 10 includes a synthesizing unit 100 for synthesizing a stereoscopic image using a 2D image captured by an image sensor, a storage unit 130 for storing the synthesized stereoscopic image, An image processing unit 110 in which a compression engine for compressing and image processing stereoscopic images stored in the storage unit 130 and an image processing program for scaling (enlarging or reducing) the stereoscopic images stored in the storage unit 130; And a display unit 120 for displaying the compressed and image processed stereoscopic image.

Hereinafter, each component of the stereoscopic image processing apparatus 10 of the present invention will be described in detail.

The synthesis unit 100 synthesizes a stereoscopic image. Stereoscopic images are left view images (LEFT) and right view images (RIGHT) obtained using a stereo camera having a binocular parallax to make the viewer feel a stereoscopic feeling. Two 2D images taken from the left and right sensors of the stereoscopic camera are synthesized into one 3D image through the stereoscopic image synthesis method (Muxing) of the synthesis unit 100.

Currently used stereoscopic image synthesis methods include Line by Line (L / L), Pixel by Pixel (P / P), Column by Column (C / C), Side by side, and Top & Side Top and bottom and two-color anaglyph. However, it is preferable to use the aforementioned methods selectively because the characteristics may vary depending on the use.

The image processing unit 110 converts the stereoscopic image synthesized by the synthesizing unit 100 into a YUV format image by combining pixels or lines with similarities in color. Y of the YUV format image represents luminance information, and U and V represent color signals. Luminance information is a measure of the brightness of light, and a color signal is a colorimetric difference between an arbitrary color and a reference color having the same luminance as that color. In other words, Y represents the information on the brightness of the original image including RGB values in gray scale divided into several levels, U is the blue signal minus the brightness (Y) , V is the red signal minus the brightness.

According to an exemplary embodiment, in the stereoscopic image synthesized for each pixel, if the color of the first pixel and the second pixel is similar, the image processor 110 transmits and compresses the first pixel and the second pixel in pairs, and transmits and compresses the YUV 4: Convert to 2: 0 format video. In this case, the image processing unit 110 may transmit and compress the UV components in U, V or V, U order of heat having similarities in color. This is also applicable to a color space domain that can separate luminance information and color information such as YCbCr. That is, the image processor 110 may transmit and compress the CbCr components in the order of Cb, Cr, or Cr, Cb of the columns having similarities in color. Where Y is the luminance component and Cb and Cr are the chrominance components. The image processor 110 converts a YUV 4: 2: 0 format image into an RGB image by combining a pair of YUVs having similarities in color for each column. A detailed embodiment of the pixel-based YUV 4: 2: 0 conversion method of the image processor 110 will be described later with reference to FIGS. 11 and 12.

According to another exemplary embodiment, in the stereoscopic image synthesized for each pixel, if the color of the first pixel and the second pixel is similar to each other, the image processor 110 transmits and compresses the first pixel and the second pixel in pairs, and transmits and compresses the YUV 4. Converts to 2: 2 format video. In this case, the image processing unit 110 may transmit and compress the UV components in U, V or V, U order of heat having similarities in color. This is also applicable to a color space domain capable of separating luminance information and color information such as YCbCr. That is, the image processor 110 may transmit and compress the CbCr components in the order of Cb, Cr, or Cr, Cb of the columns having similarities in color. In addition, the image processor 110 converts a YUV 4: 2: 2 format image into an RGB image by combining a pair of YUVs having similarities in color for each column.

According to another embodiment, in the stereoscopic image synthesized for each line, if the color of the first line and the second line is similar in color, the first line and the second line are paired with the first line and the second line, and transmitted and compressed to YUV. Convert to 4: 2: 0 format video. In this case, the image processor 110 may transmit and compress the UV components in U, V or V, U order of lines having similarities in color. This is also applicable to a color space domain capable of separating luminance information and color information such as YCbCr. That is, the image processor 110 may transmit and compress the CbCr components in the order of Cb, Cr, or Cr, Cb of lines having similarities in color. The image processor 110 converts a YUV 4: 2: 0 format image into an RGB image by combining a pair of YUVs having similarities in color for each line. A detailed embodiment of the line-based YUV 4: 2: 0 conversion method of the image processor 110 will be described later with reference to FIGS. 13 and 14.

The display unit 120 displays an RGB image. The image sensor used for the display unit 120 may be an image sensor of a CCD or CMOS type.

11 and 12 illustrate an embodiment of converting a stereoscopic image into a YUV format image and restoring the same based on a pixel-based method according to an embodiment of the present invention.

In the pixel-based method, as shown in FIG. 11, two images A and B are merged in units of pixels. Then, the UV component (or CbCr component) is transmitted and compressed in U, V order (or Cb, Cr order) or V, U order (or Cr, Cb order) of heat having similarity in color. In the BT601 / 656 transmission scheme, for example, as shown in FIG. 12A, the similarity is transmitted or compressed in U and V order of similar heat. The normal RGB image is reconstructed by combining U / Cb and V / Cr with images having similarities in color.

Since pixel 1 and pixel 3 of FIG. 12A are pixels of the A image before the stereoscopic image, color similarity exists between these pixels. Similarly, pixels 2 and 4 are pixels from the same B image, so color similarity exists. Therefore, as shown in FIG. 12B, the receiving side may restore the RGB image without losing color by converting the transmitted pixels 1, 3 and 2, 4 into RGB images.

Specifically, in the case of pixel 1, the RGB image is restored to pixel 1 Y, pixel 1 Cb, and pixel 3 Cr. In the case of pixel 3, the RGB image is restored to pixel 3 Y, pixel 1 Cb, and pixel 3 Cr. In contrast, in the case of pixel 2, the RGB image is restored to pixel 2 Y, pixel Cb, and pixel 4 Cr. In the case of the fourth pixel, the RGB image is restored to the fourth pixel Y, the second pixel Cb, and the fourth pixel Cr.

The above-described method may be applied to a YUV planar format or a color space domain that separates luminance information and color information, such as YCbCr. For example, UV components can be transmitted and compressed in U, V order as well as V, U order of color similarity. Further, even in a color space domain such as YCbCr, CbCr components can be transmitted and compressed in the order of Cb, Cr, or Cr, Cb in a column having similarities in color. Furthermore, the same applies to YUV 4: 2: 0 as well as YUV 4: 2: 2.

13 and 14 illustrate embodiments of converting a stereoscopic image into a YUV format image and restoring the same based on a line-based method according to an embodiment of the present invention.

In the case of the line-based method, as shown in FIG. 13, color is transmitted and compressed with only U component and line 3 with V component. Line 2 transmits and compresses only U component and line 4 only V component. As shown in FIG. 14, when the RGB image is reconstructed, the normal RGB image is restored by combining U / Cb and V / Cr with each other.

Specifically, in the case of transmitting and compressing the pixel in the square line of FIG. 14A, the similarity between the color of line 1 and the line 3 exists, and the similarity of the color between line 2 and the line 4 exists. Therefore, as shown in FIG. 14B, lines 1, 3 and 2 and 4 are transmitted in the same pair, respectively, and as shown in FIG. 14C, a pair of YUVs having similarities in color is combined for each line. Accordingly, RGB image conversion to the original image is possible.

Specifically, in the case of the first line, the RGB image is restored to the first line Y, the first line Cb, and the third line Cr. For the third line, the RGB image is restored to the third line Y, the first line Cb, and the third line Cr. On the other hand, in the case of the second line, the RGB image is restored to the second line Y, the second line Cb, and the fourth line Cr. In the case of the fourth line, the RGB image is restored to the fourth line Y, the second line Cb, and the fourth line Cr.

The above-described scheme may also be applied in a YUV planar format or a color space domain that separates luminance information and color information, such as YCbCr. For example, UV components can be transmitted and compressed in V, U order as well as U, V order of lines of similarity in color. Further, even in a color space domain such as YCbCr, CbCr components may be transmitted and compressed in the order of Cb, Cr, Cr or Cb of lines having similarities in color.

FIG. 15 is a diagram for one embodiment of converting a stereoscopic image into a YUV format image and restoring it based on a line based method according to another embodiment of the present invention.

Referring to FIGS. 10 and 15, the image processor 110 changes a color value of a predetermined line so that a stereoscopic image has a color value of the same image for each line having similarities in color while maintaining the YUV format, thereby changing the color value of the YUV format image. Convert to and send. Then, the converted YUV format image is restored to the RGB image using the changed color value. In this case, the image processor 110 may restore the RGB image of the lost line by referring to the color value of the line that is not lost when converting the YUV format image.

For example, when changing a stereoscopic image to a YUV 4: 2: 2 or YUV 4: 2: 0 format image, lines 1 and 4 have color values of image A, and lines 2 and 3 The line changes to have the color value of the image B. In this case, even when the YUV 4: 2: 2 or YUV 4: 2: 0 format image is converted, the color of lines 1 and 3 disappears or the colors of lines 2 and 4 disappear, You can restore the color values of the missing lines.

That is, as shown in FIG. 15, when the YUV 4: 2: 0 format image is converted, when the colors of lines 2 and 4 disappear, the color values of lines 1 and 3 use the color values of line 1. Can be restored. In addition, the color values of lines 2 and 4 can be restored with reference to the line 3 color values (that is, the color values of the original line 2).

16 is a flowchart illustrating a stereoscopic image processing method according to an embodiment of the present invention.

10 and 16, the stereoscopic image processing apparatus 10 first synthesizes a stereoscopic image (1500).

Subsequently, the synthesized stereoscopic image is converted into a YUV format image by combining pixels or lines having similarities in color (1510).

According to an embodiment, in operation 1510, when the stereoscopic image processing apparatus 10 converts the image into a YUV format image, the stereoscopic image processing apparatus 10 may generate a first pixel and a second pixel if there is a similarity in color between the first pixel and the second pixel. Two pixels are paired together for transmission and compression to convert to YUV 4: 2: 0 format video. Then, a pair of YUVs having similarities in color are combined for each column, thereby converting a YUV 4: 2: 0 format image into an RGB image.

According to another exemplary embodiment, in operation 1510, when the stereoscopic image processing apparatus 10 converts the image into a YUV format image, the stereoscopic image processing apparatus 10 may have a similarity between the first pixel and the second pixel for the stereoscopic image synthesized for each pixel. Two pixels are paired together for transmission and compression to convert to YUV 4: 2: 2 format video. Then, a pair of YUVs having similarities in color are combined for each column to convert a YUV 4: 2: 2 format image into an RGB image.

According to another exemplary embodiment, in operation 1510, when the stereoscopic image processing apparatus 10 converts the image into a YUV format image, the stereoscopic image processing apparatus 10 may match the first line and the second line if the color of the stereoscopic image synthesized for each line is similar to each other. The second line is paired and transmitted and compressed to convert to a YUV 4: 2: 0 format image. A pair of YUVs having similarities in color are combined for each line to convert a YUV 4: 2: 0 format image into an RGB image.

According to another embodiment, in operation 1510, the stereoscopic image processing apparatus 10 converts the YUV format image to have the same color value for each line having similar color, while maintaining the YUV format for the stereoscopic image. The color value of a predetermined line is changed and converted into a YUV format image.

Subsequently, the stereoscopic image processing apparatus 10 reconstructs and displays the converted YUV format image as an RGB image (1520). In this case, the stereoscopic image processing apparatus 10 may restore the RGB image of the lost line by referring to the color value of the line that is not lost when converting the YUV format image.

A stereoscopic image processing method according to an embodiment of the present invention includes a computer readable medium including program instructions for performing operations implemented by various computers. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The media may be program instructions that are specially designed and configured for the present invention or may be available to those skilled in the computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

The embodiments of the present invention have been described above. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

10: stereoscopic image processing apparatus 100: synthesis unit
110: image processing unit 120: display unit
130:

Claims (15)

A synthesis unit for synthesizing an original image to generate a stereoscopic image;
The stereoscopic image generated by the synthesizing unit converts the Y-YV-format image by combining the pixel-by-pixel or line-like similarity, and restores the converted Y-YV-format image to the Algivy image without color loss. Processing unit; And
A display unit to display the restored Algivy image;
3D image processing apparatus comprising a. The image processing apparatus of claim 1, wherein the image processing unit
The color value of the predetermined line or the predetermined pixel is changed by changing the color value of the predetermined line or the predetermined pixel so that the stereoscopic image generated by the synthesizing unit has the same image for each line or pixel having similarity of color while maintaining the YUBY format. And converting the V-format image into a V-format image, and restoring the converted Y-V-format image to the Algivy image using the changed color value. The image processing apparatus of claim 2, wherein the image processing unit
3. The stereoscopic image processing apparatus of claim 1, wherein the Y-VY format image is reconstructed by referring to color values of lines or pixels that are not lost when the Y-YV format image is converted. The image processing apparatus of claim 1, wherein the image processing unit
If there is a similarity in color between the first pixel and the second pixel of the original image constituting the stereoscopic image, the first pixel and the second pixel are paired, transmitted, and compressed to convert to Y-YV 4: 2: 0 format image. And converting the Y-YV 4: 2: 0 format image into the Algivy image by combining a pair of Y-YVs having similarities in color for each column. The method of claim 4, wherein the image processing unit
Transmitting and compressing a UV component (or CbCr component) in U, V order (or Cb, Cr order) or V, U order (or Cr, Cb order) of a column with similarity of color of the original image constituting the stereoscopic image Stereoscopic image processing apparatus, characterized in that. The image processing apparatus of claim 1, wherein the image processing unit
If there is a similarity between the colors of the first pixel and the second pixel of the original image constituting the stereoscopic image, the first pixel and the second pixel are paired, transmitted, and compressed to convert to Y-YV 4: 2: 2 format image. And converting the Y-YV 4: 2: 2 format image into the Algivy image by combining pairs of Y-YVs having similarities in color for each column. The method of claim 6, wherein the image processing unit
Transmitting and compressing a UV component (or CbCr component) in U, V order (or Cb, Cr order) or V, U order (or Cr, Cb order) of a column with similarity of color of the original image constituting the stereoscopic image Stereoscopic image processing apparatus, characterized in that. The image processing apparatus of claim 1, wherein the image processing unit
If there is a similarity in color between the first line and the second line of the original image constituting the stereoscopic image, the first line and the second line are paired, transmitted, and compressed to convert to Y-YV 4: 2: 0 format image. And converting the Y-YV 4: 2: 0 format image into the Algivy image by combining a pair of Y-YVs having similarities in color for each line. The method of claim 8, wherein the image processing unit
Transmission and compression of UV components (or CbCr components) in U, V order (or Cb, Cr order) or V, U order (or Cr, Cb order) of lines with similarity of color of original image constituting stereoscopic image Stereoscopic image processing apparatus, characterized in that. The image processing apparatus of claim 1, wherein the image processing unit
3. The stereoscopic image processing apparatus of claim 1, wherein the scaling is performed before displaying the Algivy image on the display unit. In the stereoscopic image processing method of the stereoscopic image processing apparatus,
Synthesizing the original image to generate a stereoscopic image;
Combining the generated stereoscopic image for each pixel or line having similarity of color to a Y-YV format image, and reconstructing the converted stereoscopic image into an Algivy image without color loss; And
Displaying the reconstructed Algivy image;
Stereoscopic image processing method comprising a. The method of claim 11, wherein the converting the Y-YV format image and reconstructing the converted stereoscopic image into an Algivy image include:
If there is a similarity between the colors of the first pixel and the second pixel of the original image constituting the stereoscopic image, the Y and V may be transmitted and compressed by pairing the first pixel and the second pixel, and then may be YV 4: 2: 2 or 4: 2. : Converting to a zero format image; And
Converting the Y-YV 4: 2: 2 or 4: 2: 0 format image into the Algivy image by combining a pair of Y-YV pairs having similarities in color for each column;
Stereoscopic image processing method comprising a. The method of claim 11, wherein the converting the Y-YV format image and reconstructing the converted stereoscopic image into an Algivy image include:
If there is a similarity in color between the first line and the second line of the original image constituting the stereoscopic image, the first line and the second line are paired, transmitted, and compressed to convert to Y-YV 4: 2: 0 format image. Doing; And
Converting the Y-YV 4: 2: 0 format image into the Algivy image by combining a pair of Y-YV pairs having similarities in color for each line;
Stereoscopic image processing method comprising a. The method of claim 1,
And the original image is a left view image and a right view image. The method of claim 11,
And the original image is a left view image and a right view image.

Images