KR20110101722A - Image processing system and method using multi view image - Google Patents

Abstract

An imaging system using a multiview image may include an image capturing apparatus for capturing a multiview image, a preprocessor for correcting and outputting a multiview image captured by the image capturing apparatus using a correction value of the image capturing apparatus, and the multiview image An image matching unit matching a search image based on the selected reference image among the selected reference images and calculating a parallax value between the reference image and a matching pixel of the search image, and extracting an object from the reference image using the reference image and the parallax value And an object extractor for synthesizing the extracted object and the background image to generate a synthesized image. Accordingly, an image system using a multiview image may naturally synthesize the extracted object and a new background image by extracting an object included in the multiview image.

Description

IMAGE PROCESSING SYSTEM AND METHOD USING MULTI VIEW IMAGE}

The disclosed technology relates to an image system and an image processing method using a multiview image, and a system and method for generating a stereoscopic image or synthesizing an image using a multiview image captured by an image capturing apparatus such as a camera.

The multi-view image refers to an image photographed using a plurality of image capturing apparatuses or cameras. One of the multi-view images, the stereo image is an image captured by two image capturing devices. The stereo image includes a left image and a right image captured from two image capturing devices.

The stereo imaging system selects one image from the photographed stereo images as a reference image and selects another image as a search image, so that a point in a three-dimensional space matches a pixel position corresponding to each of the reference image and the search image. The stereo imaging system may obtain 3D information of the captured image by using corresponding pixel position information.

In an embodiment, the imaging system using a multiview image may include an image capturing apparatus photographing a multiview image and a preprocessing unit correcting and outputting a multiview image captured by the image capturing apparatus by using a correction value of the image capturing apparatus. And an image matching unit matching a search image based on a reference image selected from among the multiview images, and calculating a disparity value between the reference image and a matching pixel of the search image, and using the reference image and the disparity value. An object extractor for extracting an object from an image and a synthesizer for synthesizing the extracted object and a background image to generate a synthesized image.

In an embodiment, the imaging system using a multiview image matches a search image based on a reference image selected from an image capturing apparatus photographing a multiview image, a multiview image captured by the image capturing apparatus, and the reference image. An image matching unit configured to calculate a parallax value between matching pixels of the search image, an object extractor extracting an object from the reference image using the reference image and the disparity value, and a synthesized image by synthesizing the extracted object and a background image A synthesizing unit configured to generate an image, a stereoscopic image generating unit generating a stereoscopic image using the reference image and the parallax value, and an intermediate image by synthesizing an image photographed by the multiview image capturing apparatus It includes a multi-view image synthesis unit.

In one embodiment, a method of processing a multiview image may include photographing a multiview image, correcting and outputting the multiview image using a calibration value of the image photographing apparatus photographing the multiview image, and Matching the search image based on the selected reference image among the point images, and calculating a parallax value between the reference image and a matching pixel of the search image, and extracting an object from the reference image using the reference image and the parallax value And generating a composite image by synthesizing the extracted object with the background image.

According to an embodiment, the method of processing a multiview image may include: photographing a multiview image, and when the multiview image synthesis function is set, generating an intermediate image by multiviewing the captured image and generating a stereoscopic image. When the function or the background composition function is set, matching the search image based on the reference image selected from the multi-view image photographed by the multi-view image capturing apparatus, and calculates the parallax value between the reference image and the matching pixel of the search image; Step, when the stereoscopic image generation function is set, generating a stereoscopic image by using the reference image and the parallax value; when the background synthesis function is set, the object in the reference image using the reference image and the parallax value And extracting the extracted object and the background image to generate a synthesized image.

1 is a diagram illustrating a stereoscopic image generation system of an imaging system according to an embodiment of the disclosed technology.
FIG. 2 is a diagram illustrating an image matching unit of the stereoscopic image generating system illustrated in FIG. 1.
3 is a diagram illustrating a stereoscopic image generator according to an exemplary embodiment of the disclosed technology.
4 illustrates a stereoscopic image generating unit according to another exemplary embodiment of the disclosed technology.
5 is a diagram illustrating an example of a stereoscopic image generated by the stereoscopic image generating system of FIG. 1.
6 illustrates a background synthesis system of an imaging system according to an embodiment of the disclosed technology.
FIG. 7 is a diagram illustrating an object extracting unit of the background synthesis system illustrated in FIG. 6.
FIG. 8 is a diagram illustrating a synthesis unit of the background synthesis system illustrated in FIG. 6.
FIG. 9 is a diagram illustrating a matting method of mixing an object and a background image in the background synthesizer of FIG. 8.
10 is a diagram illustrating a process of extracting an object from an image and composing a new background image and an object.
11 is a diagram illustrating a process of modeling a lighting environment of a background image.
12 is a diagram illustrating a cell-divided background image.
FIG. 13 is a diagram illustrating an image synthesized by applying a lighting environment of a background image to an object region.
14 illustrates a multiview image synthesis system of an imaging system, according to an embodiment of the disclosed technology.
15 is a view for explaining a view morphing technique for generating an intermediate image.
FIG. 16 is a diagram illustrating a multiview image system including a stereoscopic image generation system, a background synthesis system, and a multiview image synthesis system, according to an exemplary embodiment.
17 is a flowchart illustrating a stereoscopic image processing method according to an embodiment of the disclosed technology.
18 is a flowchart illustrating a background composite image processing method according to an embodiment of the disclosed technology.
19 is a flowchart illustrating a multiview image synthesis processing method according to an embodiment of the disclosed technology.
20 is a flowchart illustrating a multiview image processing method according to an embodiment of the disclosed technology.

The description of the disclosed technique is merely an example for structural or functional explanation and the scope of the disclosed technology should not be construed as being limited by the embodiments described in the text. That is, the embodiments may be variously modified and may have various forms, and thus the scope of the disclosed technology should be understood to include equivalents capable of realizing the technical idea.

Each step may occur differently from the stated order unless the context clearly dictates the specific order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. Terms defined in commonly used dictionaries should be interpreted to be consistent with meaning in the context of the relevant art and can not be construed as having ideal or overly formal meaning unless expressly defined in the present application.

An operation of an image system and an image processing method according to an embodiment of the disclosed technology will be described with reference to the accompanying drawings. The image system and the stereoscopic image processing method according to an embodiment may create a stereoscopic image or synthesize an image using a multiview image photographed at various viewpoints. The stereoscopic image or the composite image generated in the imaging system may be transmitted to the other receiver or stored in the system and processed.

1 is a diagram illustrating a stereoscopic image generation system of an imaging system according to an embodiment of the disclosed technology. The stereoscopic image generating system 100 of FIG. 1 is an example of a system for creating a stereoscopic image using a stereoscopic image captured by a stereo camera. Referring to FIG. 1, the stereoscopic image generating system 100 includes a first image capturing apparatus 110a, a second image capturing apparatus 110b, a preprocessing unit 120, an image matching unit 130, and a stereoscopic image generating unit ( 140, a transmitter 150, and a storage 160.

The first image capturing apparatus 110a and the second image capturing apparatus 110b photograph a subject. For example, the image capturing apparatus 110a or 110b photographs a subject using a complementary metal-oxide semiconductor (CMOS) module, a charge coupled device (CCD) module, or the like. The image (or video) of the subject is provided to the CCD module or the CMOS module through the lens, and the CCD module or the CMOS module of the imaging apparatuses 110a and 110b converts the optical signal of the subject passing through the lens into an image signal. Output The imaging apparatuses 110a and 110b perform exposure adjustment, gamma adjustment, gain adjustment, white balance adjustment, or color matrix adjustment, and then digitally convert the optical signal through an analog-to-digital converter (ADC). The video signal is converted and output.

The preprocessor 120 rectifies the stereo image through the correction parameter value of the image capturing apparatus, and outputs the same epipolar line of the image. The preprocessor 120 corrects the stereo image by using correction parameters for correcting characteristic differences, brightness differences, and the like of an image photographing apparatus such as a stereo camera, and corrects the epipolar line of the image to be horizontal.

The image matching unit 130 calculates disparity information by matching the stereo image received by the preprocessor 120. The image matching unit 130 matches the left image and the right image photographed by the left and right image photographing apparatuses and calculates a parallax between the matching pixels. The image matching unit 130 may calculate a disparity map for calculating a disparity value for each pixel of the stereo image.

The stereoscopic image generator 140 generates a stereoscopic image by using the image received from the image matching unit 130 and parallax information. The image matching unit 130 may select and output one of the stereo images (left image and right image), or may output one image obtained by synthesizing the stereo image into one image. The stereoscopic image generator 140 analyzes the color components of the image output from the image matcher 130 and generates a stereoscopic image using the analyzed color components and parallax information.

 The image matching unit 130 and the stereoscopic image generating unit 140 may be implemented in the form of a software module using software in the stereoscopic image system, or by using graphic processing hardware such as a graphic processing unit (GPU) of the stereoscopic image system. Or may be implemented through dedicated hardware logic on a field-programmable gate array (FPGA).

The transmitter 150 receives the stereoscopic image generated by the stereoscopic image generator 140 and transmits the stereoscopic image to the destination. The transmitter 150 modulates a stereoscopic image and transmits the stereoscopic image to a counterpart receiver (not shown) through wired / wireless communication and a network system according to an embodiment. The transmitter 150 may transmit the stereoscopic image generated by the stereoscopic image generating system to the computer network according to the IP (Internet Protocol) protocol, or may modulate the stereoscopic image generated according to the mobile communication modulation scheme and transmit the stereoscopic image to the mobile communication network.

When the stereo image photographing apparatus or the camera is installed in a computer, a notebook, a personal mobile communication terminal, or the like, the stereoscopic image generating system may be used in the form of an application such as a computer, a notebook, or a personal mobile communication terminal. For example, the 3D image generating system may be used as a video conference application using a computer, a notebook computer, a personal mobile communication terminal, a video call application, a video messenger function of a messenger application, and the like.

The stereoscopic image generating system 100 may receive a control signal from a receiver through a receiver (not shown), and the stereoscopic imaging system 100 receives the control signal to change the photographing angles of the image capturing apparatuses 100a and 100b. Or zoom in on a specific area.

The storage unit 160 stores the stereoscopic image generated by the stereoscopic image generating unit 140. The user may copy and store the stereoscopic image stored in the storage unit 160 to storage media such as a removable memory, a removable hard disk, or an optical disk, or may play the stereoscopic image at a desired time through a playback device. In addition, the user may filter or reprocess the stereoscopic image stored in the storage 160 or process an image effect.

FIG. 2 is a diagram illustrating an image matching unit of the stereoscopic image generating system illustrated in FIG. 1. Referring to FIG. 2, the image matcher 130 includes a stereo matcher 210 and a parallax calculator 220.

The stereo matching unit 210 matches a stereo image (left image and right image) received from the preprocessor 120 to find a pixel corresponding to the stereo image. The stereo matching unit 210 selects one image of the left image and the right image as a reference image, and finds and matches a pixel corresponding to the other image, that is, the search image, based on the reference image. For stereo matching, feature-based matching and area-based matching may be used according to matching elements. For example, in the case of feature-based matching, stereo images are matched using zero crossings, boundary lines, and corners as matching elements. In the case of region-based matching, a pixel having a high similarity is matched and a stereo image is matched using a shape having the flat or uniform brightness change of the pixel, an average brightness of the region, etc. as a matching element.

The stereo matching unit 210 outputs pixel matching information, that is, information about a pixel of the search image corresponding to the pixel of the reference image to the parallax calculating unit 220, and outputs one of the stereo images to the stereo image generating unit 140. ) Hereinafter, a case in which the stereo matching unit 210 outputs a reference image among the stereo images will be described as an example.

The parallax calculating unit 220 calculates parallax information (or parallax value) in the pixel using the pixel matching information of the stereo image. Parallax refers to the difference in image coordinates in a reference image and a search image for the same point in space. The parallax calculator 220 calculates, as a parallax value, a pixel distance between pixels matched with the same pixel in the reference image and the search image based on the reference image. The parallax calculator 220 may calculate a parallax value for each pixel of the reference image, and the parallax value may be calculated in pixel distance units (eg, 1 pixel distance and 2 pixel distance). As a result of calculating the parallax value for each pixel of the reference image, the parallax calculator 220 may obtain a disparity map.

The image matching unit 130 buffers the reference image data output from the stereo matching unit 210 while the parallax calculation unit 220 calculates the parallax value, thereby displacing the parallax value and the reference image to the stereoscopic image unit 140. It can also be output at the same time.

3 is a diagram illustrating a stereoscopic image generator according to an exemplary embodiment of the disclosed technology. Referring to FIG. 3, the stereoscopic image generator 140 includes an image receiver 310, a first color component extractor 320, a second color component extractor 330, and an image synthesizer 340.

The image receiver 310 receives an image from the image matcher 130 and outputs the image to the color component extractors 320 and 330. The color component extractors 320 and 330 extract specific color components from the image received by the image receiver 310, respectively.

The first color component extractor 320 and the second color component extractor 330 extract color component values having chromaticity opposite colors from each pixel of the image. For example, when the first color component extractor 320 extracts a red component value from each pixel of the image, the second color component extractor 330 extracts a blue component value. Alternatively, when the first color component extractor 320 extracts a red component value, the second color component extractor 330 may extract a cyan component value. The type of color components extracted by the color component extracting units 320 and 330 may vary according to embodiments, and when there is a color to make a better stereoscopic image according to the experiment example, the color components according to the experiment example may be extracted. can do.

The image synthesizer 340 may receive an image received by the image receiver 310, a first color component value extracted by the first color component extractor 320, and a second color extracted by the second color component extractor 330. The component values are synthesized to generate a stereoscopic image. When generating a stereoscopic image, the image synthesizing unit 340 generates a stereoscopic image using the disparity value received from the image matching unit 130. For example, the image synthesizer 340 synthesizes the first color component value extracted by the first color component extractor 320 by moving the pixels left or right by the parallax component (or parallax value) to the image. . In addition, the image synthesizing unit 340 sets the second color component value extracted by the second color component extracting unit 330 to the right or the left (or a direction opposite to the moving direction of the first color component) by the parallax component (or parallax value). Go to and synthesize with the image.

4 illustrates a stereoscopic image generating unit according to another exemplary embodiment of the disclosed technology. Referring to FIG. 4, the stereoscopic image generator 140 includes an image receiver 410, a color component extractor 420, and an image synthesizer 430. In the embodiment of FIG. 3, each color component extracting unit 320 and 330 are connected in parallel with respect to each color component, but FIG. 4 illustrates the extraction of a plurality of color component values from one color component extracting unit 420. This is the output form.

The image receiver 410 receives an image from the image matcher 130 and outputs the image to the color component extractor 420. The color component extractor 420 extracts a plurality of color component values having a chromatically opposite color relationship from each pixel of the image received by the image receiver 310. The image synthesizer 430 generates a stereoscopic image by combining the image received by the image receiver 410 and the plurality of color component values extracted by the color component extractor 420. For example, the image synthesizing unit 340 synthesizes the first color component value extracted by the color component extraction unit 420 by moving the pixels left or right by the parallax component (or parallax value) and synthesizing it with the image. The second color component value extracted by the component extractor 420 is moved to the right or left side (the direction opposite to the moving direction of the first color component) by the parallax component (or parallax value) and synthesized with the image. The stereoscopic image generated as described above may be stored in the storage 160 or may be modulated by the transmitter 150 and transmitted to the receiver.

In the above embodiment, a stereoscopic image is generated using one image and a parallax value among the stereo images, but a stereoscopic image may be generated using a synthesized stereo image and a parallax value. For example, a stereo image may be synthesized into one intermediate image by using a morphing technique of synthesizing a multiview image, and then a stereoscopic image may be generated. Using a morphing technique, a plurality of images may be used to generate one image by interpolating shapes and colors included in each image.

The user watching the stereoscopic image generated as described above may feel the stereoscopic sense of the image by using the stereoscopic glasses dyed in complementary colors. For example, when the stereoscopic image generator 140 extracts a red component and a blue component to generate a stereoscopic image, the user dyes one eye side in red and the other eye side in blue. Stereoscopic images can be viewed using the stereoscopic glasses. When the stereo component is generated by extracting the red component and the cyan component from the stereoscopic image generator 140, the user may wear stereoscopic glasses in which one eye side is dyed red and the other eye side is dyed cyan. 3D video can be viewed.

5 is a diagram illustrating an example of a stereoscopic image generated by the stereoscopic image generating system of FIG. 1. Each image shown in the upper portion of FIG. 5 represents an original image before making a stereoscopic image, and each image shown in the lower portion represents a stereoscopic image generated through the stereoscopic image generation process as described above. When a user watching a stereoscopic image views the image illustrated in the lower part of FIG. 5 using the stereoscopic image dyed in complementary colors, the user may feel a stereoscopic feeling.

6 illustrates a background synthesis system of an imaging system according to an embodiment of the disclosed technology. The background synthesis system 600 of FIG. 6 is an example of a system for extracting an object using a stereo image photographed by a stereo camera, and synthesizing the extracted object with another background to create an image.

Referring to FIG. 6, the background synthesis system 600 includes a first image capturing apparatus 610a, a second image capturing apparatus 610b, a preprocessor 620, an image matcher 630, and an object extractor 640. , A synthesizer 650, a transmitter 660, and a storage 670. A description of the first image capturing apparatus 610a, the second image capturing apparatus 610b, the preprocessor 620, the image matching unit 630, the transmitter 660, and the storage 670 is shown in FIG. As described in the embodiment, a detailed description thereof will be omitted.

The object extractor 640 extracts an object included in the image using the image output from the image matcher 630 and parallax information (or parallax value). The object extractor 640 extracts an object using a background difference technique or an edge detection technique, or extracts an object using parallax information. The object extractor 640 may extract at least one object included in the image.

The synthesis unit 650 selects a background image to synthesize the object, and synthesizes the background image with the object extracted by the object extractor 640. The background synthesis system 600 may select one of the background images already stored in the memory (not shown), or synthesize the object and the background image by receiving the background image from the user. The synthesis unit 650 may model a lighting environment of the background image to synthesize the object, and then apply the modeled lighting environment to the synthesized image obtained by synthesizing the object and the background image to create a natural composite image.

The transmitter 660 transmits the synthesized image generated by the synthesizer 650 to the opposite receiver, and the storage unit 670 stores the synthesized image generated by the synthesizer 650.

FIG. 7 is a diagram illustrating an object extracting unit of the background synthesis system illustrated in FIG. 6. Referring to FIG. 7, the object extractor 640 includes a parallax separator 710, a background difference unit 720, an edge detector 730, and an object determiner 740.

The parallax separator 710 extracts an object included in the image using the image received from the image matching unit 630 and the parallax value. The disparity value is larger as the distance from the image capturing apparatus becomes larger, and the value becomes smaller as the distance from the image capturing apparatus becomes larger. In the case of the pixel of the background area included in the image, it has a parallax value less than or equal to a threshold value, and in the case of the pixel of the object area that includes an object, unlike the background area, a pixel of the background area includes a certain range of parallax values above a certain value. Can have For example, when a user performs image communication using a stereo image capturing apparatus, the face region of the user captured by the image capturing apparatus may have a predetermined range of parallax, unlike a background region. The parallax value range of the face region may vary according to the distance between the image photographing apparatus and the user, and the parallax value range may be obtained through an experiment. The parallax separator 710 may determine and extract a pixel area having a predetermined range of parallax as an object area by using the characteristics of the parallax as described above.

After determining the object area, the parallax separator 710 may remove the noise by filtering the object area. For example, the parallax separator 710 may remove noise in the object region by using a morphological filter. The morphology filter removes noise for each pixel determined as the object region using a pixel window (eg, a 3x3 pixel window, a 5x5 pixel window, etc.) having each pixel as a center pixel. That is, when the number of pixels of the object region included in the pixel window is greater than a certain number, the morphology filter determines the center pixel as the pixel of the object region, and the number of pixels of the object region included in the pixel window is determined by a certain number. If smaller, the center pixel is determined as the pixel of the background area. The morphology filter may reduce the amount of computation by filtering only the pixels around the region determined as the object region, for example, only the pixels within a predetermined distance from the boundary pixels of the object region.

The background difference unit 720 extracts an object by differentiating a background image that does not include an object and a currently received image. The background difference unit 720 compares an already stored background image with a currently received image to determine an area where a difference between a pixel value (for example, a brightness value or a color difference value of a pixel) differs by a predetermined value or more. It can be determined as an object area. The background synthesis system 600 may pre-capture a background image without an object each time the system is newly operated, or allow the user to manually photograph a background image without an object in advance. For example, in the case of video communication, the background synthesis system 600 may extract an object after photographing a background image in which the user is not included in advance before performing the video communication. After determining the object region, the background difference unit 720 may filter the object region and remove noise by using a filter such as a morphology filter.

The edge detector 730 detects an edge of the currently received image. The edge detector 730 may detect the edge of the object by using pixels whose brightness values are discontinuous. For example, the edge detector 730 may detect, as an edge, a pixel whose brightness value of the pixel suddenly changes in comparison with a neighboring pixel. The edge detector 730 may detect edges using horizontal and vertical differential operations by first order derivatives such as gradient operations, and second derivatives such as laplacian operations, Gaussian difference, etc. Edges can be detected using various detection operations, such as of Gaussian).

The object determiner 740 determines the object by using the object region extracted by the parallax separator 710 and the background difference unit 720 and the edge detected by the edge detector 730. For example, the object determiner 740 determines the area extracted as the object area at the same time by the parallax separator 710 and the background difference unit 720 as the object area. A pixel determined as an object region in only one of the parallax separator 710 and the background difference unit 720 determines the object region using the edge information extracted by the edge detector 730. For example, when the pixel determined as the object area is only within one of the parallax separator 710 and the background difference unit 720, the object is within a certain distance (for example, 1 to 2 pixel distance) from the edge of the object. The determiner 740 determines the pixel as a pixel included in the object area. If the object is separated from the edge by a distance or more, the object determiner 740 determines the pixel as a pixel included in the background area.

FIG. 8 is a diagram illustrating a synthesis unit of the background synthesis system illustrated in FIG. 6. Referring to FIG. 8, the combiner 660 includes a background combiner 810 and an illumination combiner 820.

The background synthesizer 810 selects a background image for synthesizing an object and synthesizes the background image with the object extracted by the object extractor 640. The background synthesizer 810 may select one of the background images already stored in the memory (not shown), or may synthesize the background image and the object extracted by the object extractor 640 by receiving a background image from the user. . The background synthesizer 810 sets a blending region to be mixed with the background image based on the boundary of the object and synthesizes the object and the background image.

FIG. 9 is a diagram illustrating a matting method of mixing an object and a background image in the background synthesizer of FIG. 8.

The background synthesizer 810 sets the mixed region in the vertical direction to the background region and the object region based on the boundary of the object. The mixed region may have a range of pixel distances in the background region direction and the object region direction. When the blending region is determined, the background synthesizing unit 810 synthesizes the object region and the background region by using a linear blending method. That is, the background synthesizer 810 calculates pixel values in the mixed region by using a linear blending ratio according to the pixel value of the object region, the pixel value of the background region, and the distance. For example, when the pixel value of the object area is X, the pixel value of the background area is Y, and the calculated pixel value in the mixing area is Z, the mixing may be performed in a mixing ratio of Z = aX + bY. As the distance from the object boundary toward the object area increases, the value a increases and the value b decreases. On the contrary, the value of a decreases and the value of b increases as the distance from the object boundary toward the background region increases.

10 is a diagram illustrating a process of extracting an object from an image and composing a new background image and an object.

FIG. 10A illustrates an original image in which an object is captured, and FIG. 10B illustrates a parallax value of each pixel calculated by matching a stereo image with brightness. The closer to the image capturing apparatus, the larger the parallax value, so that it is expressed in bright colors. FIG. 10C is a diagram illustrating an object region extracted from the image a of FIG. 10, and FIG. 10D is a diagram illustrating the synthesis of the object extracted from the image a of FIG. 10 and a new background image.

The synthesizing unit 650 may create a more natural synthetic image using the lighting synthesis function. The lighting synthesis unit 820 included in the synthesis unit 650 models a lighting environment of a background image to synthesize an object, and then applies the modeled lighting environment to the image synthesized by the background synthesis unit 810. To create a natural composite image.

11 is a diagram illustrating a process of modeling a lighting environment of a background image. 11 is a diagram illustrating a shape conversion of a background image into a mirror ball shape. FIG. 11 is a diagram illustrating a sample of a mirror ball shape divided by latitude and longitude. The right view of FIG. 11 illustrates a sample of the mirror ball shape divided by a radius and an angle around a center point. It is a figure which shows.

The lighting synthesis unit 820 models the lighting environment of the background image by using the mirror ball. The mirror ball can know the lighting effect that is affected by the ambient light at the point where the mirror ball is located.

The lighting synthesis unit 820 converts the background image to be synthesized with the object into a semi-spherical mirror ball shape as shown in the left figure of FIG. 11. For example, the lighting synthesizing unit 820 may expand the center portion of the image by using a shape conversion algorithm of the image, and contract the peripheral portion of the image to convert it into a semi-spherical mirror ball shape. Alternatively, the lighting synthesis unit 820 divides the background image into cells, and maps the divided cells to each cell of the mirror ball shown in the center view of FIG. 11 or the right view of FIG. 11. You can also convert to ball form.

12 is a diagram illustrating a cell-divided background image. FIG. 12A illustrates a cell dividing method in the case of mapping to the middle view (mirror ball divided by latitude and longitude) of FIG. 11, and FIG. 12B illustrates the right view (divided by radius and angle) of FIG. 11. Cell division method in the case of mapping to a mirror ball). However, the cell division scheme of FIG. 12 is an example and the cell division scheme may be performed in another manner. In addition, as the size of the cell is smaller, the lighting synthesis unit 820 may model a more accurate lighting environment. The lighting synthesizing unit 820 divides the cells in the same manner as in FIG. 12 and maps each cell to the cells of the mirror ball of the middle or right side of FIG. 11. Each cell of the background image may be expanded or contracted to fit the corresponding cell shape of the mirror ball.

After converting the background image into a mirror ball shape, the lighting synthesis unit 820 calculates an average illumination value for each cell of the mirror ball. The illumination value may be expressed as a color value of the illumination and a brightness value of the illumination. The lighting composition unit 820 may know the color, direction or angle, distance, etc. of the light source using the average illumination value. For example, the lighting combination unit 820 may determine that the light source is positioned in a cell direction in which the brightness value is equal to or greater than a predetermined value. In addition, the lighting combination unit 820 may determine that the light source is relatively far when the brightness value is small among the brightness values exceeding a predetermined value, and the light source is relatively close when the brightness value is large. It can be judged that.

The lighting synthesizing unit 820 recombines the lighting effect by adding the lighting effect to the image synthesized by the background synthesizing unit 810 by using the color, direction or angle, and distance information of the light source. The lighting composition unit 820 changes the surface color of the image by using the color, direction or angle, distance information of the light source, and depth information of the object according to the parallax value of the object region. For example, the lighting composition unit 820 may add a lighting effect to the object area by using a shading function. The shading function changes the color of the texture by using the direction or angle of the light source and the distance from the light source.

FIG. 13 is a diagram illustrating an image synthesized by applying a lighting environment of a background image to an object region. In FIG. 13, the upper figure shows a background image, and the lower figure shows an image synthesized by applying a lighting environment of each background image to an object region.

The transmitter 660 receives the synthesized image generated by the synthesizer 650 and transmits the synthesized image to the destination. The synthesizing unit 650 may output a synthesized image obtained by synthesizing the background image and the object according to a function setting, or may output a synthesized image obtained by applying a lighting environment of the background image to the synthesized image of the background image and the object. The storage unit 670 stores the stereoscopic image generated by the synthesis unit 650.

14 illustrates a multiview image synthesis system of an imaging system, according to an embodiment of the disclosed technology. The multiview image synthesis system 1400 of FIG. 14 is an example of a system for creating a synthesized image using a stereo image among the multiview images.

Referring to FIG. 14, the multiview image synthesizing system 1400 includes a first image capturing apparatus 1410a, a second image capturing apparatus 1410b, a preprocessor 1420, a multiview image synthesizing unit 1430, and a transmitting unit. 1440 and the storage 1450. Since the description of the first image capturing apparatus 1410a, the second image capturing apparatus 1410b, the preprocessor 1420, the transmission unit 1440, and the storage unit 1450 is the same as described with reference to the embodiment of FIG. 1. Detailed description thereof will be omitted.

The multiview image synthesizer 1430 synthesizes the input multiview image and outputs one image. The multiview image synthesizer 1430 synthesizes a plurality of images by using a morphing technique. The morphing technique generates one image by interpolating shapes and colors included in the image. In the case of using a view morphing technique, the multi-view image synthesizing unit 1430 captures an image by using an image capturing apparatus between the first image capturing apparatus 1410a and the second image capturing apparatus 1410b. The same intermediate view image may be generated. 15 is a view for explaining a view morphing technique for generating an intermediate image. Referring to FIG. 15, a view morphing technique may be performed by using an image captured by a left image capturing apparatus and an image captured by a right image capturing apparatus, such as an image captured by a virtual camera at a position of a virtual camera. view).

FIG. 16 is a diagram illustrating a multiview image system including a stereoscopic image generation system, a background synthesis system, and a multiview image synthesis system, according to an exemplary embodiment. The multi-view image system may be configured by combining at least two or more systems of a stereoscopic image generation system, a background synthesis system, and a multiview image synthesis system. FIG. 16 illustrates a system in which all three systems are combined.

Referring to FIG. 16, the multiview imaging system 1600 may include a first image capturing apparatus 1610a, a second image capturing apparatus 1610b, a preprocessor 1620, an image matcher 1630, and an object extractor 1640. ), A synthesizer 1650, a stereoscopic image generator 1660, a multiview image synthesizer 1670, a transmitter 1680, and a storage 1690.

The multi-view image system 1600 includes the stereoscopic image generation system described with reference to FIG. 1, the background synthesis system described with reference to FIG. 6, and the multiview image synthesis system described with reference to FIG. 14.

The multi-view imaging system 1600 may generate a stereoscopic image by using the multi-view images captured by the image capturing apparatus 1610a or 1610b, or may synthesize a new background image with an object included in the image. In addition, the multi-view image system 1600 may generate a single image by synthesizing the multi-view image. The image matcher 1630, the object extractor 1640, the synthesizer 1650, the stereoscopic image generator 1660, and the multiview image synthesizer 1670 of the multi-view image system 1600 may use software. It may be implemented in the form of a software module, or may be implemented using graphics processing hardware such as a GPU, or may be implemented through dedicated hardware logic on an FPGA.

When the image capturing apparatus or the camera is installed in a computer, a notebook, a personal mobile communication terminal, or the like, the multi-view video system 1600 may be used in the form of an application such as a computer, a notebook, or a personal mobile communication terminal. For example, the 3D image generating system may be used as a video conference application using a computer, a notebook computer, a personal mobile communication terminal, a video call application, a video messenger function of a messenger application, and the like.

When the user uses the background compositing function of the multiview imaging system 1600, the multiview imaging system 1600 may display an image captured by the image capturing apparatus 1610a or 1610b in the image matching unit 1630 and the object extractor ( 1640, the synthesizer 1650 extracts an object from the image, and synthesizes the extracted object and a new background image. When the user uses the 3D image generating function of the multiview image system 1600, the multiview image system 1600 generates an image matched unit 1630 and a 3D image by using the images captured by the image capturing apparatus 1610a or 1610b. The unit 1660 generates a stereoscopic image. When the user uses the multi-view image synthesis function of the multi-view image system 1600, the multi-view image system 1600 may display the image captured by the image capturing apparatus 1610a or 1610b. Through the multi-view image.

The multi-view imaging system 1600 may use a background synthesis function and a stereoscopic image generation function at the same time. For example, the 3D image generating unit 1660 of the multiview image system 1600 receives a background composite image from the combining unit 1650, and receives parallax information of the object area from the image matching unit 1630, thereby receiving the object and the object. An image obtained by synthesizing a background image may be a stereoscopic image. The stereoscopic image generator 1660 may extract the color component values of the image obtained by synthesizing the object and the background image and move the color component values left or right by the parallax value to create a stereoscopic image.

A stereoscopic image, a multiview composite image, or a background composite image generated by the multiview image system 1600 may be transmitted to the other receiver through the transmitter 1680 or may be stored in the storage unit 1690.

17 is a flowchart illustrating a stereoscopic image processing method according to an embodiment of the disclosed technology. Referring to FIG. 17, FIG. 17 illustrates a method of generating and processing a stereoscopic image by using a stereo image which is one of multi-view images.

The 3D image generating system captures a stereo image using the image capturing apparatus (S1700). The image capturing apparatus photographs an image of a subject using an image module such as a CCD module or a CMOS module.

The stereoscopic image generating system preprocesses stereo images captured by the image capturing apparatus. The stereoscopic image generating system corrects the stereoscopic image by using the imaging apparatus calibration parameter (S1710). For example, the stereoscopic image generating system may correct characteristic differences, brightness differences, and the like of the stereo image or correct the epipolar line of the image to be horizontal.

The stereoscopic image generating system calculates parallax information by matching stereo images (S1720). The stereoscopic image generating system selects one of the stereo images as a reference image, performs stereo matching, and calculates disparity information for each pixel.

The stereoscopic image generating system extracts a specific color component from one image (S1730). For example, the stereoscopic image generating system may extract a specific color component using a reference image among stereo images, or may extract a specific color component using an image obtained by synthesizing the stereo image. When the color component is extracted, the stereoscopic image generating system may extract two color components having a complementary color relationship.

The stereoscopic image generating system generates a stereoscopic image using the values of color components extracted from the image and parallax information (S1740). The stereoscopic image generating system shifts the first color component value extracted from the image left or right by the parallax information and synthesizes the image, and moves the second color component value to the right or left (the opposite direction to the first color component value) by the parallax information. The stereoscopic image may be generated by combining with the image.

18 is a flowchart illustrating a background composite image processing method according to an embodiment of the disclosed technology. Referring to FIG. 18, FIG. 18 illustrates a method of generating and processing a background composite image by using a stereo image which is one of multi-view images.

The background synthesis system captures a stereo image using the image capturing apparatus (S1800). The background synthesis system pre-processes stereo images captured by the image capturing apparatus. In operation S1810, the background synthesis system corrects the stereo image by using the imaging apparatus calibration parameter. The background synthesis system calculates parallax information by matching stereo images (S1820).

After calculating the parallax information, the background synthesis system extracts an object included in the image (S1830). For example, a background synthesis system extracts an object using parallax separation, or extracts an object using background difference. The background synthesis system may extract an object by detecting an edge included in the image. Alternatively, the region extracted into the object region at the same time by the parallax separation and the background difference may be determined as the object region, and the pixel extracted as the object region in only one may be determined as the object region based on the distance from the edge.

The background synthesis system extracting the object synthesizes the extracted object and a new background image (S1840). The background synthesis system sets a blending area including the object area and the background area based on the boundary of the object, and determines the pixel value of the blending area using the linear blending ratio according to the distance. The background synthesis system, which synthesizes the object and the background image, may model the lighting environment of the background image, and then synthesize the object regions according to the modeled lighting environment to create a more natural composite image.

19 is a flowchart illustrating a multiview image synthesis processing method according to an embodiment of the disclosed technology. Referring to FIG. 19, FIG. 19 illustrates a method of synthesizing and processing a multiview image using a stereo image which is one of multiview images.

The multi-view image synthesizing system captures a stereo image using the image capturing apparatus (S1900). The multi-view image synthesis system preprocesses stereo images captured by the image capturing apparatus. The multi-view image synthesizing system corrects the stereoscopic image by using the imaging apparatus calibration parameter (S1910).

The multi-view image synthesizing system synthesizes stereo images to generate one image (S1920). For example, a multi-view image synthesis system may synthesize an image by interpolating a shape and a color of a stereo image by using a morphing technique. Alternatively, the multiview image synthesis system may synthesize an image, such as photographed from a viewpoint between stereo images, using a view morphing technique.

20 is a flowchart illustrating a multiview image processing method according to an embodiment of the disclosed technology. Referring to FIG. 20, FIG. 20 illustrates a method of generating a stereoscopic image, generating a background composite image, or synthesizing and processing a multiview image using a stereo image which is one of multiview images.

The multiview image system may generate a stereoscopic image, generate a background composite image, or synthesize a multiview image according to a user's function setting. If the user sets the function, the multi-view imaging system captures a stereo image using the image capturing apparatus (S2000). The multi-view imaging system corrects the stereo image using the imaging apparatus calibration parameter (S2010).

The multi-view image system determines the function set by the user, and determines whether the user has set the 3D image generation function, the background synthesis function, or the multi-view image synthesis function (S2020). As a result of the determination in step S2020, when the user sets the multiview image synthesis function, the multiview image system synthesizes the stereo images to generate one image (S2030). As a result of the determination in step S2020, when the user sets the stereoscopic image generation function or the background synthesis function, the multiview image system calculates parallax information by matching the stereoscopic image (S2040).

After calculating the parallax information, the multi-view image system determines whether the user has set the 3D image generation function or the background synthesis function (S2050). As a result of the determination in step S2050, when the user sets the stereoscopic image generation function, the multi-view image system extracts a specific color component from one image of the stereoscopic image (S2060), and the value and color difference information of the color component extracted from the image. A stereoscopic image is generated using S2070. As a result of the determination in step S2050, when the user sets the background synthesis function, the multiview image system extracts an object included in the image (S2080), and synthesizes the extracted object and a new background image (S2090). Detailed description of the processing procedure at each step is as described with reference to FIGS. 1 to 19.

An image system and an image processing method using a multiview image according to an embodiment may generate a stereoscopic image using a multiview image and parallax information. The imaging system can generate stereoscopic images simply and efficiently, and can enhance stereoscopic effects of the images by using parallax information.

An image system and an image processing method using a multiview image according to an embodiment may extract an object included in a multiview image and naturally synthesize the extracted object and a new background image. The imaging system can accurately extract object regions using parallax separation, background difference, and edge detection. In addition, the imaging system synthesizes an image by modeling a lighting environment of a background area, thereby making a natural composite image, and reducing a user's dissensation of the composite image.

An image system and an image processing method using a multiview image according to an embodiment may generate a single image by synthesizing a multiview image. When synthesizing a multi-view image using the view morphing technique, it is possible to obtain a synthesized image as if the image was taken from a virtual position.

An image system and an image processing method using a multiview image according to an embodiment may transmit the generated stereoscopic image, the background composite image, or the multiview composite image to a counterpart receiver in real time. The imaging system may store the generated image, and the user may post-process or reprocess the stored stereoscopic image.

An image system and an image processing method using a multi-view image according to an embodiment may perform video conferencing or video communication with an opposite user, or perform audiovisual education for the opposite user. The opposite user receiving the stereoscopic image can more accurately observe the subject included in the image. When video conferencing or video communication is performed using the background composite image, the user can change his or her background image to a desired background image at any time, thereby increasing the immersion or message delivery effect of the video conferencing or video communication. In addition, the aesthetic effect of video conferencing or video communication can be enhanced. In the case of video conferencing or video communication using a multi-view composite image, it is possible to provide the opposite user with the expected captured image at the time when the image capturing apparatus cannot capture the image.

Although described above with reference to an embodiment of the present invention, those skilled in the art will be variously modified and changed within the scope of the present invention without departing from the spirit and scope of the invention described in the claims below. I can understand that you can.

Claims (28)

An image capturing apparatus for photographing a multiview image;
A preprocessor for correcting and outputting a multi-view image captured by the image capturing apparatus by using the correction value of the image capturing apparatus;
An image matching unit matching a search image based on a reference image selected from the multiview images, and calculating a parallax value between the reference image and a matching pixel of the search image;
An object extracting unit extracting an object from the reference image by using the reference image and the parallax value; And
And a multi-view image including a synthesizer configured to synthesize the extracted object and the background image to generate a synthesized image. The method of claim 1, wherein the object extraction unit
A parallax separator extracting an object region including an object from the reference image using the parallax value;
A background difference unit configured to extract an object region including an object from the reference image by differentially distinguishing the reference image from the image not including an object;
An edge detector detecting an edge in the reference image; And
And an object determiner configured to determine an object region by using the object region extracted by the parallax separator and the background difference unit and the edge detected by the edge detector. The method of claim 2, wherein the parallax separator
An imaging system using a multiview image which extracts a region including pixels having parallax values within a predetermined range into an object region. The method of claim 2, wherein the parallax separator
An image system using a multi-view image to remove noise by filtering the extracted object region using a morphological filter. The method of claim 2, wherein the background difference unit
An image system using a multiview image to remove noise by filtering the extracted object region using a morphology filter. The method of claim 2, wherein the object determination unit
An image system using a multi-view image for determining the area extracted into the object area at the same time in the parallax separation unit and the background difference unit. The method of claim 2, wherein the object determination unit
In the case of a pixel determined as an object area only in one of the parallax separator and the background difference part, an image using a multi-view image that determines the pixel as a pixel included in the object area when the pixel is within a predetermined distance from an edge of the object. system. The method of claim 1, wherein the synthesis unit
A background synthesizer configured to combine the object region and the background region by determining a mixed region including a part of the object region and a part of the background region based on the boundary of the extracted object; And
An image using a multiview image including an illumination synthesizer for modeling an illumination environment of the background image and changing a surface color of the image synthesized by the background synthesizer by using the parallax between the modeled illumination environment and the object region. system. The method of claim 8, wherein the lighting synthesis unit
An image system using a multi-view image that transforms the background image into a semi-spherical mirror ball shape to model the lighting environment of the background image. The method of claim 9, wherein the lighting synthesis unit
And a multi-view image for dividing the background image into cells and mapping the divided cells to corresponding cells of the mirror ball to convert them into mirror ball shapes. The method of claim 10, wherein the lighting synthesis unit
Imaging system using a multi-view image for dividing and sampling the cells of the mirror ball on the basis of latitude and longitude. The method of claim 10, wherein the lighting synthesis unit
Imaging system using a multi-view image for dividing and sampling the cells of the mirror ball based on the radius and angle. The method of claim 1,
And a stereoscopic image generator configured to generate a stereoscopic image using the synthesized image synthesized by the synthesizer and the parallax value. The apparatus of claim 13, wherein the stereoscopic image generating unit
Extracting a first color component value and a second color component value from the synthesized image, the color component value of the first color is shifted by the parallax value in a left or right direction, and the color component value of the second color is the first color. An image system using a multiview image to generate a stereoscopic image by moving the parallax value in a direction opposite to the color component value of. An image capturing apparatus for photographing a multiview image;
An image matching unit matching a search image based on a selected reference image among multiview images captured by the image capturing apparatus, and calculating a parallax value between the reference image and a matching pixel of the search image;
An object extracting unit extracting an object from the reference image by using the reference image and the parallax value;
A synthesizer configured to synthesize the extracted object and the background image to generate a synthesized image;
A stereoscopic image generator configured to generate a stereoscopic image using the reference image and the parallax value; And
And a multiview image synthesizer configured to generate an intermediate image by synthesizing a multiview image of the image photographed by the multiview image capturing apparatus. The method of claim 15, wherein the object extraction unit
A parallax separator extracting an object region including an object from the reference image using the parallax value;
A background difference unit configured to extract an object region including an object from the reference image by differentially distinguishing the reference image from the image not including an object;
An edge detector detecting an edge in the reference image; And
And an object determiner configured to determine an object region by using the object region extracted by the parallax separator and the background difference unit and the edge detected by the edge detector. The method of claim 15, wherein the synthesis unit
A background synthesizer configured to combine the object region and the background region by determining a mixed region including a part of the object region and a part of the background region based on the boundary of the extracted object; And
An image using a multiview image including an illumination synthesizer for modeling an illumination environment of the background image and changing a surface color of the image synthesized by the background synthesizer by using the parallax between the modeled illumination environment and the object region. system. The method of claim 15, wherein the stereoscopic image generating unit
An image receiving unit which receives an image;
A color component extracting unit extracting color component values from the image received by the image receiving unit; And
And an image synthesizer configured to move the color component value extracted by the color component extractor by the parallax value and to synthesize the image received by the image receiver. The method of claim 15, wherein the multi-view image synthesis unit
An image system using a multiview image for synthesizing the multiview image and generating an intermediate image using a morphing method. Photographing a multiview image;
Correcting and outputting the multi-view image using a correction value of the image photographing apparatus photographing the multi-view image;
Matching a search image based on a reference image selected from the multi-view images, and calculating a parallax value between the reference image and matching pixels of the search image;
Extracting an object from the reference image by using the reference image and the parallax value; And
And synthesizing the extracted object and the background image to generate a composite image. The method of claim 20, wherein extracting the object
Extracting a region including a pixel having a parallax value within a predetermined range from the reference image as an object region;
Extracting an object region including an object from the reference image by differentiating the reference image from the image not including the object;
Detecting an edge in the reference image; And
And determining an object region by using the object regions extracted in the first and second stages and the edge detected in the third stage. The method of claim 20, wherein generating the composite image
Synthesizing the object region and the background region by determining a mixed region including a part of the object region and a part of the background region based on the boundary of the extracted object; And
Modeling a lighting environment of the background image, and changing a surface color of an image obtained by combining the object region and the background region by using a parallax between the modeled lighting environment and the object region. How to. The method of claim 20,
Extracting a first color component value and a second color component value from the synthesized image, the color component value of the first color is shifted by the parallax value in a left or right direction, and the color component value of the second color is the first color. And generating a stereoscopic image by moving the parallax value in a direction opposite to the color component value of the multi-view image. Photographing a multiview image;
Generating a intermediate image by synthesizing the photographed image by the multi-view image when the multi-view image synthesizing function is set;
When the 3D image generation function or the background synthesis function is set, the search image is matched based on the selected reference image among the multiview images captured by the multiview image capturing apparatus, and a parallax value between the reference image and the matching pixel of the search image Calculating;
Generating a stereoscopic image using the reference image and the parallax value when a stereoscopic image generating function is set;
Extracting an object from the reference image using the reference image and the disparity value when a background synthesis function is set; And
And synthesizing the extracted object and the background image to generate a composite image. The method of claim 24, wherein generating the intermediate image
A method for processing a multiview image, which synthesizes the multiview image and generates an intermediate image using a morphing method. The method of claim 24, wherein generating the stereoscopic image
Extracting a first color component value and a second color component value from the reference image; And
The first color component value is shifted by the parallax value in the left or right direction, and the color component value of the second color is shifted by the parallax value in a direction opposite to the color component value of the first color to generate a stereoscopic image. How to process multiview images. The method of claim 24, wherein extracting the object comprises
Extracting a region including a pixel having a parallax value within a predetermined range from the reference image as an object region;
Extracting an object region including an object from the reference image by differentiating the reference image from the image not including the object;
Detecting an edge in the reference image; And
And determining an object region by using the object regions extracted in the first and second stages and the edge detected in the third stage. The method of claim 24, wherein generating the composite image
Synthesizing the object region and the background region by determining a mixed region including a part of the object region and a part of the background region based on the boundary of the extracted object; And
Modeling a lighting environment of the background image, and changing a surface color of an image obtained by combining the object region and the background region by using a parallax between the modeled lighting environment and the object region. How to.

Images