KR101983586B1 - Method of stitching depth maps for stereo images - Google Patents

Abstract

The depth map stitching method for stereo images includes a depth map generation step of receiving stereo images from a plurality of stereo cameras to generate depth maps, a depth map adjustment step of adjusting each of the depth maps in a virtual space, and a depth map each. Determining an object based on the segment having a depth map stitching step of calculating the depth of the determined object to stitch the depth maps. Therefore, the depth map stitching method for stereo images performs matching between segments appearing in an area where the depth maps overlap and detects overlapping types of matched segments to determine matched segments according to the detected overlapping type as one depth value. It can provide a technique that can be integrated into a segment having a.

Description

Depth map stitching method for stereo images {METHOD OF STITCHING DEPTH MAPS FOR STEREO IMAGES}

FIELD OF THE INVENTION The present invention relates to stereo image matching techniques, and more particularly, to a depth map stitching method for stereo images capable of performing in real time stitching on depth maps associated with stereo images.

The three-dimensional image may be obtained through a stereo camera. More specifically, the stereo camera produces right and left two-dimensional images, respectively obtained through both eyes. The 3D image may be obtained through image processing on the 2D image of the right and the left, and the depth map may be used in this process.

Korean Patent Publication No. 10-1370785 discloses a depth map of a stereoscopic image in a method and apparatus for generating a depth map of a stereoscopic image so that the depth of the stereoscopic image can be expressed in detail by considering not only vanishing points but also detailed lines in the image. It relates to a production method and apparatus.

Korean Patent Laid-Open Publication No. 10-2016-0086802 relates to a depth map generating apparatus and a method and a stereoscopic image converting apparatus and method using the same, specifically, a feature information extracting unit extracting at least one feature information on an input image A depth map initializer for generating an initial depth map of the input image based on the characteristic information, an FFT converter for performing an FFT on the input image, and converting the image into a frequency image, a representative value of the frequency image, and the initial value Provided is a depth map generator including a depth map determiner for obtaining a correlation value using an average value of a depth map and determining a final depth map based on the correlation value.

Korean Registered Patent Publication No. 10-1370785 (2014.02.27) Korean Patent Publication No. 10-2016-0086802 (2016.07.20)

An embodiment of the present invention is to provide a depth map stitching method for stereo images that can stitch the depth maps by determining the object based on a segment in each of the depth maps and calculating the depth of the determined object.

One embodiment of the present invention is to provide a depth map stitching method for stereo images capable of stitching depth maps by matching and integrating segments in overlapping regions of the depth maps.

One embodiment of the present invention is to provide a depth map stitching method for stereo images capable of integrating matched segments to have a single depth value by detecting overlapping types of matched segments.

Among the embodiments, a depth map stitching method for stereo images includes a depth map generation step of receiving stereo images from a plurality of stereo cameras to generate depth maps, adjusting each of the depth maps in a virtual space; And a depth map stitching step of determining an object based on a segment in each of the depth maps, and calculating the depth of the determined object to stitch the depth maps.

The depth map stitching may further include detecting a shape of an object by performing segment labeling on the segments.

The depth map stitching step may further include adjusting the global depth of the object by obtaining the approximate value of the depth by placing the detected object in a single plane.

The depth map stitching step may further include calculating a local depth of the detected object by performing local optimization on the detected object based on the stereo images.

The depth map stitching step may include stitching the depth maps by matching and integrating segments in an overlapping area of the depth maps.

The depth map stitching step may further include matching segments in the overlap region based on a depth of each object in the segments and an average RGB distance of each corresponding object in the stereo images.

The depth map stitching step may further include incorporating the matched segments to have a single depth value by detecting an overlapping type of the matched segments.

The depth map stitching may include integrating the depth maps into an average depth value of the background area measured in the corresponding depth maps when there is a background area in which the appearance of a specific object is not determined at the boundary of the stitching or the overlapping area. It may further comprise a step.

The disclosed technique can have the following effects. However, since a specific embodiment does not mean to include all of the following effects or only the following effects, it should not be understood that the scope of the disclosed technology is limited by this.

The depth map stitching method for stereo images according to an embodiment of the present invention may provide a technique for stitching depth maps by matching and integrating segments in overlapping regions of the depth maps.

The map stitching method for stereo images according to an embodiment of the present invention may provide a technique capable of integrating matched segments to have a single depth value by detecting an overlapping type of matched segments.

1 is a diagram illustrating an entire system for generating a virtual view image synthesized in real time by applying a depth map stitching method according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an apparatus for generating a depth map in FIG. 1.
3 is a block diagram illustrating a depth map stitching unit in FIG. 2.
FIG. 4 is a diagram for describing an embodiment of a process of generating a depth map sphere by the depth map controller of FIG. 2.
FIG. 5 is a diagram for describing an exemplary embodiment in which the depth map stitching unit of FIG. 2 performs a process for integrating and projecting depth maps aligned on a 3D virtual space into one projection sphere.
FIG. 6 is a diagram illustrating an embodiment of a process of determining, by the segment labeling module of FIG. 3, depths of overlapping objects through depth quantization.
FIG. 7 is a diagram illustrating an embodiment of a process of reprojecting segments of each patch onto a 3D virtual space by the segment matching module of FIG. 3 to perform segment matching.
FIG. 8 is a diagram illustrating an embodiment of a process of performing matching between overlapping segments within an overlapping area between depth maps by the segment matching module of FIG. 3.
FIG. 9 is a flowchart illustrating a depth map stitching process performed by the depth map generating apparatus of FIG. 2.

Description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as limited by the embodiments described in the text. That is, since the embodiments may be variously modified and may have various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical idea. In addition, the objects or effects presented in the present invention does not mean that a specific embodiment should include all or only such effects, the scope of the present invention should not be understood as being limited thereby.

On the other hand, the meaning of the terms described in the present application should be understood as follows.

Terms such as "first" and "second" are intended to distinguish one component from another component, and the scope of rights should not be limited by these terms. For example, the first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being "connected" to another component, it should be understood that there may be other components in between, although it may be directly connected to the other component. On the other hand, when a component is referred to as being "directly connected" to another component, it should be understood that there is no other component in between. On the other hand, other expressions describing the relationship between the components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring to", should be interpreted as well.

Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "comprise" or "have" refer to a feature, number, step, operation, component, part, or feature thereof. It is to be understood that the combination is intended to be present and does not exclude in advance the possibility of the presence or addition of one or more other features or numbers, steps, operations, components, parts or combinations thereof.

In each step, an identification code (e.g., a, b, c, etc.) is used for convenience of description, and the identification code does not describe the order of the steps, and each step clearly indicates a specific order in context. Unless stated otherwise, they may occur out of the order noted. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be embodied as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all kinds of recording devices in which data can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. Generally, the terms defined in the dictionary used are to be interpreted to coincide with the meanings in the context of the related art, and should not be interpreted as having ideal or excessively formal meanings unless clearly defined in the present application.

1 is a diagram illustrating an entire system for generating a virtual view image synthesized in real time by applying a depth map stitching method according to an embodiment of the present invention.

Referring to FIG. 1, the virtual view point image generation system 100 may match stereo images in real time through a depth map stitching technique, and may include a camera manager 110 and an image handler. An image handler 130, a depth map generator 150, and a scene image generator 170.

The camera manager 110 may capture a camera image from a plurality of stereo cameras. Here, the plurality of stereo cameras correspond to a stereo camera device for shooting a three-dimensional space, each of the plurality of stereo cameras is implemented as a three-dimensional stereo camera to obtain a stereoscopic image using two camera pairs for both eyes, It may be implemented as a stereoscopic camera capable of capturing two images or images of the same subject at the same time, including two lenses for photographing.

The camera manager 110 may store the captured images in the main memory. The main role of the camera manager 110 is to maintain synchronization between the left and right cameras of each stereo camera and the multiple stereo cameras. After synchronization, camera manager 110 can upload a series of camera images to GPU memory. From camera image capture to image upload, the task of processing each camera is assigned to an independent thread. The camera manager 110 may participate in thread synchronization and perform an operation of the synchronized camera.

The image handler 130 may adjust the camera image along with the correction data to be suitable for performing correction and alignment after the camera image is uploaded to the GPU memory by the camera manager 110. The main role of the image handler 130 is to properly adjust the stereo image and determine the image area valid for generating the depth map.

The depth map generator 150 generates a depth map for each stereo camera using the stereo image adjusted by the image handler 130. The depth map generator 150 may generate a depth map through pixel-by-pixel correspondence between left and right images. The depth map generating apparatus 150 may use various stereo matching methods including a graph cut or a constant belief propagation method in the depth map generation process.

The scene image generating apparatus 170 generates a synthetic virtual view point image using the depth map when the depth map is generated by the depth map generating apparatus 150. The scene image generating apparatus 170 corresponds to a computing device that may be connected to a plurality of stereo cameras and a depth map generating apparatus. For example, the scene image generating apparatus 170 may be connected to the plurality of stereo cameras by wire or wirelessly, and may be connected to the depth map generating apparatus through a wireless network. According to an embodiment, the scene image generating apparatus 170 may generate a virtual view image synthesized in real time by analyzing the scene image generated by the plurality of stereo cameras.

FIG. 2 is a block diagram illustrating an apparatus for generating a depth map in FIG. 1.

Referring to FIG. 2, the depth map generator 150 may include a depth map generator 210, a depth map controller 230, a depth map stitching unit 250, and a controller 270.

The depth map generator 210 may generate a depth map by receiving a stereo image from a plurality of stereo cameras. When the camera manager 110 captures a camera image, the image handler 130 may adjust the camera image through an image warping step. The depth map generator 210 may generate a depth map based on the adjusted camera image. Depth maps can be generated by pixel-by-pixel correspondence between left and right images.

In an embodiment, the depth map generator 210 may generate a depth map by generating a raw depth map through stereo block matching and then applying a boundary preserving filter, and graph-cut. Depth maps can be optimized using cut or constant belief propagation methods to improve the quality of depth maps.

The depth map controller 230 adjusts each of the depth maps generated by the depth map generator 210 in a virtual space through a depth map transformation step. The depth map controller 230 may align the plurality of depth maps in a virtual space defined by a 3D coordinate system using physical characteristics of the stereo camera. Here, the physical characteristics of the stereo camera may include rotational rotation and translation between the plurality of stereo cameras.

In one embodiment, the depth map adjusting unit 230 may define a depth map as a spherical coordinate system to manage depth information about a scene having a 360-degree wide viewing angle, and in this process, an image coordinate system related to an image is Euclidean ) Can be converted into a sphere coordinate system, and each of the depth maps can be converted and stored in an equictangular form in which the distance between two pixels is proportional to the angle between the projection rays. Thereafter, the depth map adjuster 230 may arrange the depth maps on the 3D virtual space based on settings of the plurality of stereo cameras.

In one embodiment, the depth map adjusting unit 230 may arbitrarily set one left camera among the plurality of stereo cameras as the main camera for alignment, and the optical center of the main camera of the 3D virtual space in which the depth maps are aligned. It can be set as the origin and can be set so that the z axis of the space is aligned with the main axis of the main camera. Accordingly, the depth map adjuster 230 may generate an image sphere in which the center of the sphere is the origin of the space and the radius of the sphere is the focal length of the main camera. According to this setting, the projection plane of the main camera may be a plane perpendicular to the surface of the sphere, and the projected sphere may provide a guideline for generating a depth map sphere.

In one embodiment, the depth map adjusting unit 230 may generate the depth map sphere by using the depth information of the plurality of depth maps, and adjust the depth information of each depth map to have one common optical point. . This will be described with reference to FIG. 4.

FIG. 4 is a diagram for describing an embodiment of a process in which the depth map controller 230 of FIG. 2 generates a depth map sphere. More specifically, the depth map adjusting unit 230 may generate a depth map sphere through the following steps.

First, the depth map adjusting unit 230 may reproject the pixels of each depth map into a three-dimensional space in consideration of the attitude and the optical point of the image plane and then project the image spheres. In Figure _4, represents the j-th pixel on the image plane i. The depth map adjusting unit 230 may reproject the _{, m} as a three-dimensional point _{, m} after the reprojection process, _{and the} reprojected point _{, m} is the distance between ( _{, m} and the center of the image sphere ₁ -the radius of the sphere ) May be projected to the same pixel point of the stored image sphere. When the reprojected three-dimensional point is projected onto the image sphere, the depth map adjusting unit 230 may project one or more three-dimensional points to the same pixel point of the image sphere, in which case the depth alignment of the three-dimensional point is necessary.

Second, the depth map adjusting unit 230 may select a point having a minimum distance by comparing the distance between each 3D point and the projection point. In Figure 4, the depth map adjustment section 230 is the _2,2 and _2,3 the pixel point in the image _sphere, can be projected on the _{t, 2,} and _2, with _t if the distance is closer _{2, 2, t} Can be projected by The depth map adjusting unit 230 may simply perform distance alignment by sequentially projecting three-dimensional points. The depth map adjusting unit 230 compares this value with the depth of the current three-dimensional point when the image sphere already has a depth value at the pixel point when attempting to project a specific three-dimensional point to the image sphere. If smaller, the value of the projection point can be changed to the depth of the current three-dimensional point.

In one embodiment, the depth map adjusting unit 230 is a three-dimensional in parallel processing to solve the problem that the sequential processing of the three-dimensional point is not suitable when performing the projection operation of the three-dimensional point reprojected in parallel processing on the GPU You can project points.

For example, in the problem of selecting the depth value of the pixel _{point, t,} depth map adjusting unit 230 is a projection light beam _{1,, t} and the first intersection or fully close the three-dimensional point in order to determine the depth of the _pixel, In this case, ₂ , ₁ and _, may always be located in the same half plane, _and three-dimensional points projected to _{, t} may also exist in the half plane. In this case, the half plane may intersect the image plane # 2, and only three-dimensional points re-projected from pixels on the intersection line may exist in the rays ₁ , _{and t} . Here, unless the settings of the cameras are changed, the position of the intersection line for each projection pixel point of the image and the angle between the intersection line and the projection ray do not change, and also the three-dimensional reprojected for each pixel on the intersection line. The depth value that makes the point meet rays C ₁ ,, _t is constant and can be calculated using the angle between the projection ray and the intersection line and the distance between the pixel and the intersection point p _{2, x, t} . Accordingly, the depth map adjusting unit 230 sequentially searches for the target pixel on the line along the intersection line starting from the intersection point _{2, t} and finds the first pixel where the reprojected three-dimensional point is projected to the pixel point _{, t} . You can stop and set the depth value of _{2, x} to (the distance between the 3D point and the pixel point _{2, x} )-(the radius of the image sphere). In one embodiment, this algorithm of the present invention eliminates the unnecessary cost of reprojection of pixels that are not displayed in the image area and makes the structure suitable for processing the entire process in parallel.

The depth map controller 230 may generate the spherical depth map for each stereo image pair by repeating the above operation for each depth map. Here, only the area covered by the original depth map in each spherical depth map has depth information, and the patch may be defined as such an area. The depth map adjusting unit 230 may combine the patches in one sphere, and thus overlap may occur between the patches. This occurs because the stereo camera's field of view overlaps, and because the depth mismatch of objects occurs in the overlapping area of the patch, each object must have a certain depth map for perfect integration. The depth map stitching unit 250 may solve the overlap between scenes and the depth mismatch in the area through the following processes.

The depth map stitching unit 250 determines an object based on a segment in each of the depth maps adjusted by the depth map adjusting unit 230, and then calculates the depth of the determined object to stitch the depth maps. Here, stitching means generating one depth map for the entire image area by selecting an optimal depth value from depth values included in the depth map for the overlapping areas between the depth maps.

In one embodiment, the depth map stitching unit 250 may integrate and integrate the depth maps aligned on the 3D virtual space into one projection sphere. As described above, it is important to determine overlaps between adjacent depth maps and resolve depth inconsistencies for complete integration between depth maps. This will be described with reference to FIG. 5.

FIG. 5 is a diagram for describing an exemplary embodiment in which the depth map stitching unit 250 of FIG. 2 performs a process for integrating and projecting depth maps aligned on a 3D virtual space into one projection sphere.

More specifically, FIG. 5 shows a case where an overlap occurs between two views of stereo cameras. In FIG. 5, the shaded areas represent areas that overlap between left and right camera pairs. If an object point is in an overlapping area, the object appears in both stereo cameras, and the depth of the object can be measured by each stereo camera. The depth map stitching unit 250 may detect the objects displayed in the overlapping area of the neighboring stereo camera in the process of integrating the aligned depth maps, and adjust the depth so that the detected objects have a consistent depth.

Overlapping areas occur differently according to camera settings in three-dimensional space. In FIG. 5, the overlapping area occupies the image area indicated by line _{1, l 1, k} if theoretically all objects in the scene are infinitely far from the camera in the case of the overlap. Here, the _first, corresponds to a line consisting of a left border, _2, the right border, _{1, k} is the pixel points by which the projection light of the image region # _22, parallel to the _r in the image region # 2, # 1 of the image area do.

The controller 270 controls the overall operation of the depth map generator 150 and manages a control flow or data flow between the depth map generator 210, the depth map adjuster 230, and the depth map stitching unit 250. Can be. In one embodiment, the controller 270 may operate inside the CPU module or the GPU module and control the tasks performed in the depth map generating apparatus 150 to be distributed to the CPU module or the GPU module for parallel processing.

3 is a block diagram illustrating a depth map stitching unit in FIG. 2.

Referring to FIG. 3, the depth map stitching unit 250 includes a segment labeling module 310, a segment optimization module 330, a segment matching module 350, a segment depth measuring module 370, and a control module 390. do.

The segment labeling module 310 may detect an object contour by performing labeling on image segments displayed on each depth map. Specifically, labeling means adjusting the depth by approximating the depth value so that the corresponding segment exists in one plane. In one embodiment, the segment labeling module 310 may adjust the global depth of the object by placing the detected object in a single plane to obtain an approximate value of the depth. Here, the global depth means the depth value of the object adjusted through segment labeling.

In one embodiment, the segment labeling module 310 classifies the objects in the image of each of the aligned depth maps into a background object and a foreground object, which are reprojected to an overlapping area of the foreground objects. If an object exists, labeling can be performed by designating it as a target object requiring depth adjustment. Here, the foreground object may correspond to an object showing sufficient inconsistency in the stereo image, and the background object may correspond to an object in which no inconsistency is observed.

In one embodiment, the segment labeling module 310 performs depth quantization to exaggerate the depth difference between objects in the depth map and reduce the depth difference within the object to make each object more distinguishable for labeling. can do. This will be described with reference to FIG. 6.

FIG. 6 is a diagram illustrating an embodiment of a process in which the segment labeling module 310 of FIG. 3 determines the depth of overlapping objects through depth quantization.

More specifically, the segment labeling module 310 may compare and analyze depth information between objects by selecting sample depth values of each object through depth quantization. In one embodiment, the segment labeling module 310 may perform the depth quantization process in a manner that appropriately selects the sample depth in consideration of the depth information of the objects in the current scene, and more specifically, the depth value for each patch. An average depth value of each cluster calculated by generating a histogram of the cluster and performing a K average clustering of depth values based on the histogram may be selected as a sample depth value of each object. Here, the patch represents an area covered by the original depth map in each spherical depth map when generating a spherical depth map for each stereo image pair for each depth map, and K represents the number of clusters and can be preset by the designer. have.

In one embodiment, the segment labeling module 310 may cluster the pixels having the same depth value and are connected after the depth quantization process to perform image segmentation, thereby defining an object shape.

The segment optimization module 330 may estimate the depth of the object image segment shown in the current depth map using a left / right image used for generating the depth map. In the previous step, the depth of the object image area on the depth map is a result of optimization of the entire image. In the present step, the optimization is additionally performed only on the image area in which the segment exists.

In one embodiment, the segment optimization module 330 may calculate a local depth of the detected object by performing local optimization on the detected object based on the stereo images. Here, the local depth represents a specific depth value for the detailed components of the object in the object image area, that is, the internal area occupied by the object, for the object detected by the segment labeling module 310.

The segment matching module 350 may perform matching between the segments displayed on the overlapped depth map. In one embodiment, the segment matching module 350 may first reproject the segments of each patch onto the three-dimensional virtual space to perform segment matching. This will be described with reference to FIG. 7.

FIG. 7 is a diagram illustrating an embodiment of a process of reprojecting segments of each patch onto a 3D virtual space by the segment matching module 350 of FIG. 3 to perform segment matching. Here, the segments of the patch may be sized when converted at the surface of the sphere, unlike segment reprojection to a planar depth map, and the adjustment factor may be proportional to the depth d _s of the segment.

In one embodiment, the segment matching module 350 uses the depth of the object determined by the segment and the average RGB distance of the image used to create the depth to match the segments appearing in the region where the depth maps overlap. Can be done. This will be described with reference to FIG. 8.

FIG. 8 is a diagram illustrating an embodiment of matching between overlapping segments within the overlapping region between depth maps by the segment matching module in FIG. 3. Referring to FIG. 8, in case 2, the center of gravity of s1.2 and s2 and the average RGB are used to measure the similarity between the segments s1 and s2. Here, the similarity indicates the degree of agreement between the depth value on the modified depth map and the depth value on the raw depth map. Similarly, Case 3 measures the similarity between s1.2 and s2.1. The segment matching module 350 performs matching between the segments with the highest similarity.

The segment depth measurement module 370 may detect the overlapping type of the segments matched by the segment matching module 350 and integrate the segments according to the detected overlapping type.

In one embodiment, the overlapping type of the segments matched by the segment matching module 350 is defined when the boundary of the segment exists inside the overlapping region, when the overlapping region exists over the overlapping region and one depth map, and the overlapping region and both depth maps. It may correspond to any of the cases present over.

In one embodiment, segment depth measurement module 270 may integrate the matched segments to have a single depth value by detecting an overlapping type of matched segments. In detail, when the boundary of the segment exists inside the overlap region, the depth value of the highly reliable segment among the segments is selected as the depth value of the final segment on both depth maps. Here, the reliability means the degree of agreement between the depth value on the modified depth map and the depth value on the original depth map. If the boundary of the segment exists over the overlapping region and one depth map, the depth value of the final segment is estimated based on the reliability of each of the corresponding segment's depth map and the size ratio of the segment. If the boundary of the segment exists over the overlapped region and both depth maps, the depth value of the final segment is estimated based on the reliability of each segment of the segments and the size ratio of the segment surrounding the overlapped region.

In one embodiment, the segment depth measurement module 270 defines an area as a background when there is an area whose appearance of the object is not determined in the overlap area of the stitching and the depth maps, and measures the area in both depth maps. Depth values can be calculated as the average of the depth values to integrate the depth maps.

The control module 390 controls the overall operation of the depth map stitching unit 250 and controls between the segment labeling module 310, the segment optimization module 330, the segment matching module 350, and the segment depth measurement module 370. You can manage flows or data flows.

FIG. 9 is a flowchart illustrating a depth map stitching process performed by the depth map generating apparatus of FIG. 2.

Referring to FIG. 9, the depth map generator 210 may generate a depth map by receiving stereo images from a plurality of stereo cameras (step S910). The depth map controller 230 may adjust each of the depth maps generated by the depth map generator 210 on the virtual space through the depth map transformation step (step S920).

The depth map stitching unit 250 may determine the object based on the segment in each of the depth maps adjusted by the depth map adjusting unit 230, and then calculate the depth of the determined object to stitch the depth maps.

Specifically, the segment labeling module 310 in the depth map stitching unit 250 may detect an object contour by labeling image segments displayed on each depth map (step S930). . Detecting an object contour here means determining which object the image segments represented on each depth map represent.

In one embodiment according to the present invention, the segment labeling module 310 may adjust the global depth of the object by obtaining the approximate value of the depth by placing the detected object in a single plane.

The segment optimization module 330 in the depth map stitching unit 250 may estimate the depth of the object image region shown in the current depth map using the original image used to generate the depth map (step S940). In the previous step, the depth of the object image area on the depth map is the result of optimizing the entire image. Therefore, additional optimization is performed on the image area where the segment exists in the current step to achieve depth optimization of the entire image. Can be.

In one embodiment according to the present invention, the segment optimization module 330 may calculate a local depth of the detected object by performing local optimization on the detected object based on the stereo images.

The segment matching module 350 in the depth map stitching unit 250 may perform matching between the segments displayed on the overlapped depth map (step S950). In one embodiment, the segment matching module 350 uses the depth of the object determined by the segment and the average RGB distance of the image used to create the depth to match the segments appearing in the region where the depth maps overlap. Can be done.

The segment depth measurement module 370 in the depth map stitching unit 250 detects the overlapping type of the segments matched by the segment matching module 350 and segments the matched segments according to the detected overlapping type with a single depth value. Can be integrated (step S960).

The overlapping type of the segments matched by the segment matching module 350 is when the boundary of the segment exists inside the overlapping region, when it exists over the overlapping region and one depth map, and when it exists over the overlapping region and both depth maps. It may correspond to any one.

In one embodiment, the segment depth measurement module 270 may integrate the overlapped depth maps according to the overlap type of the matched segments. In one embodiment, the segment depth measurement module 270 maps the depth map to an average depth value with respect to the background area measured in the corresponding depth maps when there is a background area where the appearance of a specific object is not determined at the boundary or overlapping area of the stitching. Can be integrated.

Although described above with reference to a preferred embodiment of the present invention, those skilled in the art will be variously modified and changed within the scope of the invention without departing from the spirit and scope of the invention described in the claims below I can understand that you can.

100: virtual view image generation system synthesized in real time
110: camera manager 130: image handler
150: depth map generator 170: scene image generator
210: depth map generator 220: depth map controller
250: depth map stitching unit 270: control unit
310: segment labeling module 330: segment optimization module
350: segment matching module 370: segment depth measurement module
390: control module

Claims (8)

(a) a depth map generation step of receiving stereo images from a plurality of stereo cameras and generating depth maps;
a depth map adjustment step of adjusting each of the depth maps in a virtual space; And
(c) determining a object based on a segment in each of the depth maps, and calculating a depth of the determined object to stitch the depth maps;
Step (c) is
Stitching the depth maps by matching and integrating segments in an overlapping area based on a depth of each object in the segments and an average RGB distance of each corresponding object in the stereo images. Depth map stitching method for stereo images.
The method of claim 1, wherein step (c)
And performing segment labeling on the segments to detect the shape of the object.
The method of claim 2, wherein step (c)
Placing the detected object in a single plane to obtain an approximate value of depth to adjust the global depth of the object.
The method of claim 3, wherein step (c)
And calculating a local depth of the detected object by performing local optimization on the detected object based on the stereo images.
delete delete The method of claim 1, wherein step (c)
And detecting the overlapping types of the matched segments and incorporating the matched segments to have a single depth value.
The method of claim 1, wherein step (c)
Incorporating the depth maps into an average depth value of the background area measured in the corresponding depth maps when there is a background area in which the appearance of a specific object is not determined at the boundary of the stitching or the overlapping area. Characterized by depth map stitching method for stereo images.

Images