KR101158678B1 - Stereoscopic image system and stereoscopic image processing method - Google Patents

Abstract

The stereoscopic imaging system includes an image photographing apparatus for photographing a multiview image, a preprocessor for correcting and outputting a multiview image photographed by the image capturing apparatus using a calibration value of the image capturing apparatus, and a reference image selected from the multiview images. An image matching unit that matches the search image based on the reference image, calculates a parallax value between the reference image and the matched pixel of the search image, and generates a stereoscopic image using the reference image and the parallax value do. Accordingly, the stereoscopic image system according to the embodiment may generate a stereoscopic image using a multiview image and parallax information.

Description

Stereoscopic image system and stereoscopic image processing method {STEREOSCOPIC IMAGE SYSTEM AND STEREOSCOPIC IMAGE PROCESSING METHOD}

The disclosed technology relates to a system and a processing method for processing a stereoscopic image, and to a system and a method for generating a stereoscopic image using an image captured by an imaging apparatus such as a camera and processing the generated stereoscopic image.

Unlike two-dimensional planar images, three-dimensional stereoscopic images may have the same effect as a user viewing an object in a three-dimensional space. In the case of a 3D stereoscopic image, parallax is given to both eyes of a user who views the image, and the image is synthesized based on the parallax so that the image is viewed as one stereoscopic image. That is, the 3D stereoscopic image presents the same image as seen from the left and right directions to the left and right eyes of the user so that the stereoscopic image is visible to the user. As described above, examples of a method of separating the left and right bidirectional images to be visible to the left and right eyes include a method using polarized glasses, a color filter glasses, or a screen.

In one embodiment, the three-dimensional imaging system is an image capturing apparatus for photographing a multi-view image, a pre-processing unit for correcting and outputting a multi-view image captured by the image capturing apparatus using the correction value of the image capturing apparatus, the multi-view An image matching unit that matches a search image based on a selected reference image among images, calculates a parallax value between the reference image and a matching pixel of the search image, and generates a stereoscopic image using the reference image and the parallax value It includes a stereoscopic image generating unit.

In one embodiment, the three-dimensional imaging system is an image capturing apparatus for photographing a multi-view image, a pre-processing unit for correcting and outputting a multi-view image captured by the image capturing apparatus using the correction value of the image capturing apparatus, the multi-view An image matching unit for generating a composite image by matching images, calculating a parallax value between matching pixels of the multi-view image, and a stereoscopic image generating unit generating a stereoscopic image using the synthesized image and the parallax value Include.

According to an embodiment, the method may further include photographing a multiview image, correcting and outputting the multiview image using a calibration value of the image photographing apparatus photographing the multiview image, among the multiview images. Matching the multiview image based on the selected reference image, calculating a parallax value between matching pixels of the multiview image, and generating a stereoscopic image using the reference image and the parallax value .

The description of the disclosed technique is merely an example for structural or functional explanation and the scope of the disclosed technology should not be construed as being limited by the embodiments described in the text. That is, the embodiments may be variously modified and may have various forms, and thus the scope of the disclosed technology should be understood to include equivalents capable of realizing the technical idea.

Each step may occur differently from the stated order unless the context clearly dictates the specific order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. Terms defined in commonly used dictionaries should be interpreted to be consistent with meaning in the context of the relevant art and can not be construed as having ideal or overly formal meaning unless expressly defined in the present application.

An operation of a stereoscopic image system and a stereoscopic image processing method according to an embodiment of the disclosed technology will be described with reference to the accompanying drawings. According to an embodiment, a stereoscopic image system and a stereoscopic image processing method may generate a stereoscopic image using a multiview image photographed at various viewpoints, and transmit or store the stereoscopic image.

1 illustrates a stereoscopic imaging system according to an embodiment of the disclosed technology. The stereoscopic image system 100 of FIG. 1 is an example of a system for creating a stereoscopic image using a stereo image which is one of multi-view images. Referring to FIG. 1, the stereoscopic image system 100 includes a first image capturing apparatus 110a, a second image capturing apparatus 110b, a preprocessor 120, an image matcher 130, and a stereoscopic image generator 140. ) And the transmission unit 150.

The first image capturing apparatus 110a and the second image capturing apparatus 110b photograph a subject. For example, the image capturing apparatus 110a or 110b photographs a subject using a complementary metal-oxide semiconductor (CMOS) module, a charge-coupled device (CCD) module, or the like. The image (or video) of the subject is provided to the CCD module or the CMOS module through the lens, and the CCD module or the CMOS module of the imaging apparatuses 110a and 110b converts the optical signal of the subject passing through the lens into an image signal. Output The imaging apparatuses 110a and 110b perform exposure adjustment, gamma adjustment, gain adjustment, white balance adjustment, or color matrix adjustment, and then digitally convert the optical signal through an analog to digital converter (ADC). The video signal is converted and output.

The preprocessor 120 rectifies the stereo image through the correction parameter value of the image capturing apparatus, and outputs the same epipolar line of the image. The preprocessor 120 corrects the stereo image by using correction parameters for correcting characteristic differences, brightness differences, and the like of an image photographing apparatus such as a stereo camera, and corrects the epipolar line of the image to be horizontal.

The image matching unit 130 calculates disparity information by matching the stereo image received by the preprocessor 120. The image matching unit 130 matches the left image and the right image photographed by the left and right image photographing apparatuses and calculates a parallax between the matching pixels. The image matching unit 130 may calculate a disparity map for calculating a disparity value for each pixel of the stereo image.

The stereoscopic image generator 140 generates a stereoscopic image by using the image received from the image matching unit 130 and parallax information. The image matching unit 130 may select and output one of the stereo images (left image and right image), or may output one image obtained by synthesizing the stereo image into one image. The stereoscopic image generator 140 analyzes the color components of the image output from the image matcher 130 and generates a stereoscopic image using the analyzed color components and parallax information.

The image matching unit 130 and the stereoscopic image generating unit 140 may be implemented in the form of a software module using software in the stereoscopic image system, or by using graphic processing hardware such as a graphic processing unit (GPU) of the stereoscopic image system. It may be implemented.

The transmitter 150 receives the stereoscopic image generated by the stereoscopic image generator 140 and transmits the stereoscopic image to the destination. For example, the transmitter 150 may transmit and store a stereoscopic image to a storage device (not shown). Alternatively, the transmitter 150 may modulate a 3D image and transmit the modulated stereoscopic image to a receiver (not shown) through various wired / wireless communication and network systems. When transmitting to the receiver, the transmitter 150 may transmit a stereoscopic image generated in real time to the receiver.

When the stereoscopic imaging system 100 has a control function based on a feedback channel from the receiver, the receiver may transmit a control signal, and the stereoscopic imaging system 100 may receive a control signal to provide a control signal of the imaging apparatuses 100a and 100b. You can also change the shooting angle or zoom to a specific area.

FIG. 2 is a diagram illustrating an image matching unit of the stereoscopic image system illustrated in FIG. 1. Referring to FIG. 2, the image matcher 130 includes a stereo matcher 210 and a parallax calculator 220.

The stereo matching unit 210 matches a stereo image (left image and right image) received from the preprocessor 120 to find a pixel corresponding to the stereo image. The stereo matching unit 210 selects one image of the left image and the right image as a reference image, and finds and matches a pixel corresponding to the other image, that is, the search image, based on the reference image. For stereo matching, feature-based matching and area-based matching may be used according to matching elements. For example, in the case of feature-based matching, stereo images are matched using zero crossings, boundary lines, and corners as matching elements. In the case of region-based matching, a pixel having a high similarity is matched and a stereo image is matched using a shape having the flat or uniform brightness change of the pixel, an average brightness of the region, etc. as a matching element.

The stereo matching unit 210 outputs pixel matching information, that is, information about a pixel of the search image corresponding to the pixel of the reference image to the parallax calculating unit 220, and outputs one of the stereo images to the stereo image generating unit 140. ) Hereinafter, a case in which the stereo matching unit 210 outputs a reference image among the stereo images will be described as an example.

The parallax calculating unit 220 calculates parallax information in a pixel by using pixel matching information of a stereo image. Parallax refers to the difference in image coordinates in a reference image and a search image for the same point in space. The parallax calculator 220 calculates, as a parallax value, a pixel distance between pixels matched with the same pixel in the reference image and the search image based on the reference image. The parallax calculator 220 may calculate a parallax value for each pixel of the reference image, and the parallax value may be calculated in pixel distance units (eg, 1 pixel distance and 2 pixel distance).

The image matching unit 130 buffers the reference image data output from the stereo matching unit 210 while the disparity information is calculated by the parallax calculating unit 220, so that the parallax information and the reference image are transferred to the stereoscopic image unit 140. It can also be output at the same time.

3 is a diagram illustrating a stereoscopic image generator according to an exemplary embodiment of the disclosed technology. Referring to FIG. 3, the stereoscopic image generator 140 includes an image receiver 310, a first color component extractor 320, a second color component extractor 330, and an image synthesizer 340.

The image receiver 310 receives an image from the image matcher 130 and outputs the image to the color component extractors 320 and 330. The color component extractors 320 and 330 extract specific color components from the image received by the image receiver 310, respectively.

The first color component extractor 320 and the second color component extractor 330 extract color component values having chromaticity opposite colors from each pixel of the image. For example, when the first color component extractor 320 extracts a red component value from each pixel of the image, the second color component extractor 330 extracts a blue component value. Alternatively, when the first color component extractor 320 extracts a red component value, the second color component extractor 330 may extract a cyan component value. The type of color components extracted by the color component extracting units 320 and 330 may vary according to embodiments, and when there is a color to make a better stereoscopic image according to the experiment example, the color components according to the experiment example may be extracted. can do.

The image synthesizer 340 may receive an image received by the image receiver 310, a first color component value extracted by the first color component extractor 320, and a second color extracted by the second color component extractor 330. The component values are synthesized to generate a stereoscopic image. When generating a stereoscopic image, the image synthesizing unit 340 generates a stereoscopic image using the disparity information received from the image matching unit 130. For example, the image synthesizer 340 synthesizes the first color component value extracted by the first color component extractor 320 by moving the pixels left or right by the parallax component (or parallax value) to the image. . In addition, the image synthesizing unit 340 sets the second color component value extracted by the second color component extracting unit 330 to the right or the left (or a direction opposite to the moving direction of the first color component) by the parallax component (or parallax value). Go to and synthesize with the image.

4 illustrates a stereoscopic image generating unit according to another exemplary embodiment of the disclosed technology. Referring to FIG. 4, the stereoscopic image generator 140 includes an image receiver 410, a color component extractor 420, and an image synthesizer 430. In the embodiment of FIG. 3, each color component extracting unit 320 and 330 are connected in parallel with respect to each color component, but FIG. 4 illustrates the extraction of a plurality of color component values from one color component extracting unit 420. This is the output form.

The image receiver 410 receives an image from the image matcher 130 and outputs the image to the color component extractor 420. The color component extractor 420 extracts a plurality of color component values having a chromatically opposite color relationship from each pixel of the image received by the image receiver 310. The image synthesizer 430 generates a stereoscopic image by combining the image received by the image receiver 410 and the plurality of color component values extracted by the color component extractor 420. For example, the image synthesizing unit 340 synthesizes the first color component value extracted by the color component extraction unit 420 by moving the pixels left or right by the parallax component (or parallax value) and synthesizing it with the image. The second color component value extracted by the component extractor 420 is moved to the right or left side (the direction opposite to the moving direction of the first color component) by the parallax component (or parallax value) and synthesized with the image.

The stereoscopic image generated as described above may be stored in a storage device or may be modulated by a transmitter and transmitted to a receiver. For example, a stereo camera is generated using a web cam or a camera such as a mobile communication terminal, and then a stereoscopic image is generated and stored in a computer, a mobile communication terminal, or a wired / wireless computer network or a mobile communication network. Can be transmitted to the receiver on the other end.

In the above embodiment, a stereoscopic image is generated using a reference image and parallax information, which is one of stereo images, but a stereoscopic image may be generated using a synthesized stereo image and parallax information. For example, the image matching unit 130 may synthesize a stereo image into one intermediate image by using a morphing technique of synthesizing a multiview image. Morphing techniques include image morphing, which generates an intermediate image by interpolating shapes and colors included in a plurality of images, and view morphing, which generates an intermediate image such as an image taken at a virtual position between multiview images. Etc. The image matching unit 130 synthesizes the stereo image matched by the stereo matching unit 210 into an intermediate image by a multiview image synthesis unit (not shown). The stereoscopic image generator 140 synthesizes the stereoscopic image by moving the plurality of color component values extracted from the synthesized intermediate image by the parallax information.

The user watching the stereoscopic image generated as described above may feel the stereoscopic sense of the image by using the stereoscopic glasses dyed in complementary colors. For example, when the stereoscopic image generator 140 extracts a red component and a blue component to generate a stereoscopic image, the user dyes one eye side in red and the other eye side in blue. Stereoscopic images can be viewed using the stereoscopic glasses. When the stereo component is generated by extracting the red component and the cyan component from the stereoscopic image generator 140, the user may wear stereoscopic glasses in which one eye side is dyed red and the other eye side is dyed cyan. 3D video can be viewed.

5 is a diagram illustrating an example of a stereoscopic image generated in the stereoscopic imaging system of FIG. 1. Each image shown in the upper portion of FIG. 5 represents an original image before making a stereoscopic image, and each image shown in the lower portion represents a stereoscopic image generated through the stereoscopic image generation process as described above.

When a user watching a stereoscopic image views the image illustrated in the lower part of FIG. 5 using the stereoscopic image dyed in complementary colors, the user may feel a stereoscopic feeling.

6 is a flowchart illustrating a stereoscopic image processing method according to an embodiment of the disclosed technology. Referring to FIG. 6, FIG. 6 illustrates a method of generating and processing a stereoscopic image using a stereo image which is one of multi-view images.

The stereoscopic image system captures a stereo image by using a stereo image capture device (S600). The stereo image capturing apparatus captures an image of a subject by using an image module such as a CCD module or a CMOS module.

The stereoscopic imaging system pre-processes stereo images captured by the imaging apparatus. The stereoscopic image system corrects the stereoscopic image by using the imaging apparatus calibration parameter (S610). For example, the stereoscopic image system may correct characteristic differences, brightness differences, and the like of the stereo image, or correct the epipolar line of the image to be horizontal.

The stereoscopic image system calculates parallax information by matching stereo images (S620). The stereoscopic image system selects one of the stereo images as a reference image, performs stereo matching, and calculates disparity information for each pixel.

The stereoscopic image system extracts a specific color component from one image (S630). For example, the stereoscopic image system may extract a specific color component using a reference image among the stereo images, or may extract a specific color component using an image obtained by synthesizing the stereo image. When the color component is extracted, the stereoscopic image system may extract two color components having a complementary color relationship.

The stereoscopic image system generates a stereoscopic image using the values of color components and parallax information extracted from the image (S640). The stereoscopic imaging system shifts the first color component value extracted from the image left or right by parallax information and synthesizes it with the image. The stereoscopic image may be generated by combining with the image.

When the stereoscopic image system is configured to bypass the stereoscopic image generation function, the stereoscopic image system may directly store the stereoscopic image in a storage device or transmit the stereoscopic image to a counterpart receiver.

The stereoscopic image system according to the embodiment may generate a stereoscopic image using a multiview image and parallax information. The stereoscopic image system may store the generated stereoscopic image or transmit the generated stereoscopic image to the opposite receiver.

The stereoscopic image system according to the embodiment may generate stereoscopic images quickly and efficiently, and may transmit the generated stereoscopic images to the opposite receiver in real time.

The stereoscopic image system according to the embodiment may make a stereoscopic feeling to an image viewer by using the generated stereoscopic image. Therefore, video conferencing or audiovisual education can be performed using the stereoscopic images generated as described above, and the subject can be observed more precisely. For example, when applied to a real imager, a conference participant or an educator can feel a three-dimensional effect of an object or an object included in an image.

Although described above with reference to an embodiment of the present invention, those skilled in the art will be variously modified and changed within the scope of the present invention without departing from the spirit and scope of the invention described in the claims below. I can understand that you can.

1 illustrates a stereoscopic imaging system according to an embodiment of the disclosed technology.

FIG. 2 is a diagram illustrating an image matching unit of the stereoscopic image system illustrated in FIG. 1.

3 is a diagram illustrating a stereoscopic image generator according to an exemplary embodiment of the disclosed technology.

4 illustrates a stereoscopic image generating unit according to another exemplary embodiment of the disclosed technology.

5 is a diagram illustrating an example of a stereoscopic image generated in the stereoscopic imaging system of FIG. 1.

6 is a flowchart illustrating a stereoscopic image processing method according to an embodiment of the disclosed technology.

Claims (14)

An image capturing apparatus for photographing a multiview image; A preprocessor for correcting and outputting a multi-view image captured by the image capturing apparatus by using the correction value of the image capturing apparatus; An image matching unit matching a search image based on a reference image selected from the multiview images, and calculating a parallax value between the reference image and a matching pixel of the search image; A stereoscopic image generator configured to generate a stereoscopic image using the reference image and the parallax value; And And a transmitter configured to modulate and transmit the stereoscopic image or the searched image and the parallax value generated by the stereoscopic image generator, The stereoscopic image generating unit, An image receiving unit receiving the reference image; A color component extractor configured to extract a plurality of color component values from the reference image; And And an image synthesizer configured to move the color component values extracted by the color component extractor by the parallax value and synthesize the synthesized image with the reference image. The image matching unit of claim 1, wherein the image matching unit A stereo matching unit which selects one of the multi-view images as a reference image and finds and matches a pixel corresponding to the search image based on the reference image; And And a parallax calculator configured to calculate, as a parallax value, a pixel distance between pixels matched with the same pixel in the reference image and the search image. delete The method of claim 1, wherein the color component extraction unit Extracting a color component value of a first color and a color component value of a second color from the reference image, wherein the first color and the second color have a complementary color relationship. The image synthesizing unit of claim 1, wherein the image synthesizing unit The color component value of the first color is shifted by the parallax value in the left or right direction, and the color component value of the second color is shifted by the parallax value in the opposite direction to the color component value of the first color and synthesized. system. An image capturing apparatus for photographing a multiview image; A preprocessor for correcting and outputting a multi-view image captured by the image capturing apparatus by using the correction value of the image capturing apparatus; An image matching unit matching a search image based on a reference image selected from the multiview images, and calculating a parallax value between the reference image and a matching pixel of the search image; A stereoscopic image generator configured to generate a stereoscopic image using the reference image and the parallax value; And And a transmitter configured to modulate and transmit the stereoscopic image or the searched image and the parallax value generated by the stereoscopic image generator, The stereoscopic image generating unit, A first color component extracting unit extracting a first color component value from the reference image; A second color component extracting unit extracting a second color component value from the searched image; And A stereoscopic synthesizer configured to move the color component values extracted by the first color component extractor and the second color component extractor, respectively, by the parallax value, and to synthesize the reference image and the search image received by the image receiver; Imaging system. The method of claim 1, And a storage device configured to store the stereoscopic image generated by the stereoscopic image generator. delete An image capturing apparatus for photographing a multiview image; A preprocessor for correcting and outputting a multi-view image captured by the image capturing apparatus by using the correction value of the image capturing apparatus; An image matching unit configured to generate one composite image by matching the multiview image, and to calculate a parallax value between matching pixels of the multiview image; A stereoscopic image generator for generating a stereoscopic image using the synthesized image and the parallax value; And Further comprising a transmitter for modulating and transmitting the stereoscopic image generated by the stereoscopic image generating unit, The stereoscopic image generating unit, The color component value of the first color is shifted by the parallax value in the left or right direction, and the color component value of the second color is shifted by the parallax value in the direction opposite to the color component value of the first color and Stereoscopic imaging system to synthesize. 10. The method of claim 9, wherein the image matching unit 3. A stereoscopic image system, which generates an intermediate image by synthesizing a multiview image by a morphing method of synthesizing a multiview image. delete Photographing a multiview image; Correcting and outputting the multi-view image using a correction value of the image photographing apparatus photographing the multi-view image; Matching the multiview image based on a reference image selected from the multiview images, and calculating a parallax value between matching pixels of the multiview image; Generating a stereoscopic image using the reference image and the parallax value; And The method further includes modulating and transmitting the stereoscopic image generated in the generating of the stereoscopic image. Generating the stereoscopic image, Extracting a first color component value and a second color component value from the reference image; And And moving the extracted first and second color component values by the parallax value, respectively, and synthesizing the extracted first and second color component values with the reference image. delete delete

Images