KR20140118083A - System for producing stereo-scopic image or video and method for acquiring depth information - Google Patents

System for producing stereo-scopic image or video and method for acquiring depth information Download PDF

Info

Publication number
KR20140118083A
KR20140118083A KR1020130033350A KR20130033350A KR20140118083A KR 20140118083 A KR20140118083 A KR 20140118083A KR 1020130033350 A KR1020130033350 A KR 1020130033350A KR 20130033350 A KR20130033350 A KR 20130033350A KR 20140118083 A KR20140118083 A KR 20140118083A
Authority
KR
South Korea
Prior art keywords
depth information
area
depth
image
color
Prior art date
Application number
KR1020130033350A
Other languages
Korean (ko)
Inventor
장우석
호요성
Original Assignee
인텔렉추얼디스커버리 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 인텔렉추얼디스커버리 주식회사 filed Critical 인텔렉추얼디스커버리 주식회사
Priority to KR1020130033350A priority Critical patent/KR20140118083A/en
Publication of KR20140118083A publication Critical patent/KR20140118083A/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0077—Colour aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Landscapes

  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Disclosed are a system for producing a stereoscopic image and a method for acquiring depth information. More specifically, the system for producing a stereoscopic image comprises: a detecting unit to detect a misalignment area from results of stereo alignment performed based on color images acquired by multiple color cameras to photograph certain scenes; an extracting unit to extract misalignment depth information corresponding to the misalignment area within depth images acquired from one or more depth cameras to photograph the scenes; and a supplementing unit to fill a portion of the detected misalignment area corresponding to the extracted misalignment depth information using the extracted misalignment depth information, and fill the rest of the detected misalignment area based on the misalignment depth information used to fill the portion, wherein a stereoscopic image is produced based on the depth information for each scene acquired by the supplementing unit.

Description

TECHNICAL FIELD [0001] The present invention relates to a stereoscopic image production system and a depth information acquisition method,

The present invention relates to a stereoscopic image production system and a depth information acquisition method, and more particularly, to a system for producing a stereoscopic image and a method for acquiring depth information for stereoscopic image production.

Recently, 3D stereoscopic image production technology has been widely used and many video images have been produced, and related fields have been actively studied.

In order to produce such a three-dimensional stereoscopic image, it is necessary to acquire depth information.

The active sensor method is a method of acquiring depth information, which uses a depth camera. A depth camera or a time-of-flight (TOF) camera can acquire a depth image containing depth information of a scene using an optical signal as shown in FIG. 1 is a view showing an example of a depth camera and a depth image obtained therefrom.

However, depth cameras have considerably lower resolution than typical color cameras. As a result, the depth image obtained from the depth camera does not include the depth information on the entire color image as shown in FIG. FIG. 2 is a diagram showing a resolution difference between a color image and a depth image. In addition, in the case of a depth camera, lens distortion may occur severely, so up-sampling operation is indispensably required for a depth image.

However, in order to upsample the resolution of the depth image to the same level as that of the color image, a process of predicting the remaining information based on a part of the information included in the depth image is required, so that the accuracy and reliability of the information finally obtained is very low There is a problem.

In addition, the passive sensor method is another method of acquiring depth information, in which a color image obtained from a color camera is used. Two color images can be obtained from a color camera arranged close to each other and the depth can be predicted using a displacement difference between matching points corresponding to each other in a color image. This method is called stereo matching.

Depth prediction in stereo matching can extract high resolution depth information, but there is a problem that matching operation is performed based on inaccurate information in some areas. In particular, since the color cameras are physically located at different positions as shown in FIG. 3, the occlusion region A, which exists only in one color image and is not visible or not visible in other color images, occurs. In this occlusion region A, since there is no coincidence with each other, the depth prediction can not be precisely predicted. In addition, since it is not possible to accurately determine which pixel is a coincident pixel even in a texture-free or repetitive region B as shown in Fig. 3, erroneous matching is achieved and the depth prediction can not be accurately made. 3 is a diagram showing an example of an area where inaccurate stereo matching is performed.

On the other hand, a hybrid type sensor system which acquires depth information by mixing the active sensor system and the passive sensor system and enhances the performance is attracting attention.

In this regard, Korean Unexamined Patent Publication No. 2002-0026662 (entitled: Apparatus and method for restoring an occluded region) discloses a technique for restoring an occlusion region generated in a process of generating a virtual viewpoint image. In this paper, we introduce the inpainting method using surrounding information. However, since it uses incorrect peripheral information, accuracy and reliability of obtained depth information are limited.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems of the conventional art, and it is an object of the present invention to provide a method of acquiring depth information of a hybrid type that compensates a result of stereo matching by utilizing an accurate depth image obtained from a depth camera The purpose is to do.

Further, in some embodiments of the present invention, when the result of stereo matching is supplemented, the color image obtained from the color camera is not utilized any more, but the depth information having high reliability is obtained by using the depth image obtained only from the depth camera , Thereby providing a stereoscopic image production system for producing a stereoscopic image of high quality through this.

It is to be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

As a technical means to achieve the above technical object, a stereoscopic image production system according to an embodiment of the present invention is a stereoscopic image production system for synthesizing a stereoscopic image, which is a result of stereo matching performed based on a color image obtained from a plurality of color cameras A detection unit for detecting an unmatched area from the detection unit; An extracting unit for extracting unmatched depth information corresponding to the detected unmatched area in a depth image obtained from at least one depth camera that photographs the scene; And extracting the unmatched depth information based on the extracted unmatched depth information and filling the partial area of the detected unmatched area corresponding to the extracted unmatched depth information, And a complementary unit that fills the remaining area of the detected non-matching area, and creates a stereoscopic image based on the depth information of each scene acquired by the supplementary unit.

According to another aspect of the present invention, there is provided a stereoscopic image producing system including: a photographing device including a plurality of color cameras for photographing a predetermined scene and at least one depth camera; And an image authoring device for acquiring depth information on the scene based on the color image obtained from the color camera and the depth image obtained from the depth camera and for authoring the stereoscopic image based on the depth information for each scene, The image authoring apparatus includes a detection unit detecting a non-matching region from a result of stereo matching based on the color image; An extracting unit for extracting non-registration depth information corresponding to the detected non-registration area in the depth image; And extracting the unmatched depth information based on the extracted unmatched depth information and filling the partial area of the detected unmatched area corresponding to the extracted unmatched depth information, And a complement to fill the remaining area of the detected unmatched area.

According to another aspect of the present invention, there is provided a method of acquiring depth information for stereoscopic image production, comprising: capturing a predetermined scene using a plurality of color cameras and at least one depth camera; Detecting an unmatched region from a result of stereo matching performed based on the color image obtained from the color camera; Extracting non-registration depth information corresponding to the detected non-registration area in the depth image obtained from the depth camera; Determining a partial area corresponding to the extracted unmatched depth information among the detected unmatched areas; And filling the partial area using the extracted unmatched depth information and filling the remaining area of the detected unmatched area based on the unmatched depth information used to fill the partial area.

According to the above-mentioned object of the present invention, it is possible to fill a partial area of the unmatched area using the unmatched depth information extracted in the depth image, By filling the remaining area of the matching area, depth information with high reliability and accuracy can be obtained.

Further, according to the above-mentioned object of the present invention, it is possible to improve the accuracy of the depth information for a predetermined scene by filling in the non-matching region in a state in which the result of inaccurate stereo matching is excluded, It is possible to produce a high-quality three-dimensional content or stereoscopic image that provides a high degree of satisfaction to the user based on the depth information.

1 is a view showing an example of a depth camera and a depth image obtained therefrom,
2 is a diagram showing a resolution difference between a color image and a depth image,
3 is a diagram showing an example of an area where inaccurate stereo matching is performed,
FIG. 4 is a general view of a stereoscopic image production system according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating an example of searching for a matching point in a color image obtained by photographing with the color camera shown in FIG. 4 to acquire depth information;
FIG. 6 is a block diagram showing each configuration included in a stereoscopic image production system according to another embodiment of the present invention. FIG.
7 is a diagram showing an example for explaining the operation of the detection unit shown in FIG. 6;
8 is a diagram illustrating an example of operations of the extracting unit and the complementing unit shown in FIG. 6,
9 is a view for explaining a conventional hybrid method for acquiring depth information,
Figs. 10 and 11 are diagrams showing an example for explaining the operation of the supplementary unit shown in Fig. 6 for acquiring depth information; Fig.
12 is a flowchart for explaining a depth information acquisition method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise. The word " step (or step) "or" step "used to the extent that it is used throughout the specification does not mean" step for.

Hereinafter, a video production system proposed in the present invention will be described with reference to FIG. FIG. 4 is a general view of a stereoscopic image production system according to an embodiment of the present invention.

An image production system according to an embodiment of the present invention includes a photographing apparatus 100, an image authoring apparatus 200 and a display apparatus 300 and includes a photographing apparatus 100, a video authoring apparatus 200, 300 are connected to each other via wired / wireless lines to transmit / receive various data.

The photographing apparatus 100 includes a plurality of color cameras 110 and one or more depth cameras 120 for photographing a predetermined scene. The plurality of color cameras 110 may be installed substantially in parallel, and the depth camera 120 may be installed near the color camera 110 as shown in FIG. 4, and the installation position is not particularly limited. However, when the number of the color cameras 110 is increased beyond a predetermined number, it is impossible to cover the area captured by the plurality of color cameras 110 by using the one depth camera 120, ) Must be installed. The color camera 110 and the depth camera 120 may be installed on the structure 130 as shown in FIG.

In addition, the color camera 110 and the depth camera 120 preferably use the same kind, but are not limited thereto. For example, the color camera 110 may use a Basler pylon model capable of shooting at 30 fps with a resolution of up to 1920 x 1080 (HD), and the depth camera 120 may use a resolution of 176 x 144 (QCIF) Possible Mesa imaging SR4000 models are available.

FIG. 5 is a diagram illustrating an example of searching for a matching point in a color image obtained by photographing with the color camera shown in FIG. 4 to acquire depth information.

Since the predetermined scene is photographed by the two color cameras 110 installed at different positions, the first color image and the second color image have a slight parallax.

In general, the depth information of a specific image can be expressed as a depth map represented by 8 bits from 0 to 255 or a displacement map expressing a parallax with respect to an adjacent point of the corresponding pixel in pixel units. The depth and disparity They have an inverse relationship with each other. That is, although the displacement of an object close to the actual distance from the color camera 110 is large, the depth value is small. On the contrary, the displacement value of the object or the background which is far from the actual distance from the color camera 110 is relatively small, but the depth value is large.

When there are two color images having parallaxes as shown in FIG. 5, a displacement value for a specific pixel in the first color image can be obtained by searching where the corresponding pixel (coincidence point) is located in the second color image at an adjacent time, Knowing the displacement can calculate the depth. The first color image and the second color image may correspond to the left image and the right image, respectively. Such a displacement search method is called stereo matching, and is the most general method for acquiring depth information.

Specifically, a block of a second color image in blocks (a 1 to a 5 ) located on the same scan line as a block a 0 including a specific pixel in the first color image is best A matching point search process is performed as shown in FIG. 5 to determine whether the matching is performed or not. At this time, it is assumed that both binocular or multi-point color images used for stereo matching are rectified.

Hereinafter, a general procedure for performing stereo matching will be described. First, two color images for the right and left viewpoints are selected, and the maximum parallax (the displacement value of the object with the closest actual distance to the color camera 110) and the minimum parallax (the actual distance between the color camera 110 and the color camera 110) Distant object or background displacement value). This process is usually done manually.

According to one example, the maximum / minimum displacement values thus obtained can be input to the stereo matching algorithm, and the range from the minimum displacement value to the maximum displacement value can be determined as the search range of the displacement candidate group. Thereafter, the most suitable displacement value is determined for each pixel in the color image based on the determined search range. The candidate having the smallest energy among the displacement candidates may be determined as the displacement value of the corresponding pixel. Generally, since the background is fixed even when the foreground or object moves in a predetermined scene, the minimum / maximum displacement value can be easily set. However, in the case of stereo matching, there is a limitation in generating a disparity map in which accuracy is degraded by performing a prediction process based on an incorrect color image in a certain region in obtaining displacement values or depth information.

According to another example, 0, which is the smallest value that the 8-bit displacement map can have, is set as the minimum displacement value, and the largest value 255 can be set as the maximum displacement value. That is, 0 and 255 can be input to the stereo matching algorithm, and the range of 0 to 255 can be determined as the search range of the displacement candidate group. In this case, it takes a long time to calculate and it is difficult to acquire accurate displacement value or depth information.

Accordingly, in the present invention, the depth image of the depth camera 120 having a low resolution but accurate depth information as well as a stereo matching result is utilized. At this time, the depth camera 120 photographs a predetermined scene together with the color camera 110 as described above, and acquires a depth image as shown in FIG. The image photographed by the photographing apparatus 100 is transmitted to the image authoring apparatus 200.

4, the video authoring apparatus 200 acquires depth information on a predetermined scene based on the color image obtained from the color camera 110 and the depth image obtained from the depth camera 120, And author a stereoscopic image based on depth information of a scene.

In particular, the video authoring apparatus 200 includes a detection unit, an extraction unit, and a supplementary unit.

The detection unit detects the non-matching region from the result of the stereo matching performed based on the color image.

The extracting unit extracts the unmatched depth information corresponding to the detected unmatched area in the depth image.

The complementary part fills a partial area of the detected non-matched area corresponding to the extracted non-matched depth information using the extracted non-matched depth information, and detects a non-matched area based on the non- Fills the remaining area of the area.

A detailed description of each of these configurations will be described later.

Also, the display device 300 displays at least one of various images to the user. At this time, the color image obtained from the color camera 110, the depth image obtained from the depth camera 120, the result image of the stereo matching, the image obtained by highlighting the non-matching region detected by the detecting unit in the resultant image, The depth image after the non-matching area is filled in and the stereoscopic image authored by the image authoring apparatus 200 can be displayed. Accordingly, the user can check the overall progress and modify some processes based on various images displayed on the display device 300 in order to acquire the depth information with high reliability.

The stereoscopic image production system described so far can acquire depth information of high reliability and accuracy by including a photographing device, a video authoring device, and a display device, and can produce a high quality stereoscopic image based on this.

FIG. 6 is a block diagram showing each configuration included in a stereoscopic image production system according to another embodiment of the present invention. The image production system according to another embodiment of the present invention includes a detection unit 210, an extraction unit 220, and a supplementary unit 230. These configurations may be implemented in the video authoring apparatus 200 described above, and some of the configurations may be implemented outside the video authoring apparatus 200. Hereinafter, for the sake of convenience of explanation, it is assumed that these configurations are implemented in the video authoring apparatus 200. However, since this is only an example, it will be appreciated that these configurations may operate in the following manner with different combinations or some modified forms.

The detection unit 210 detects the non-matching region from the result of the stereo matching performed based on the color image obtained from the plurality of color cameras that photographs a predetermined scene, as shown in FIG. 7 is a diagram showing an example for explaining the operation of the detection unit shown in FIG.

First, the detection unit 210 receives the stereo matching result images C 1 and C 2 based on the color image. Since the stereo matching result can represent the overall depth of the entire image for a given scene, it is used as an initial depth map or a displacement map.

Subsequently, the detection unit 210 detects the non-matching regions d 1 , d 2 , d 3 from the result of the received stereo matching. Here, the unmatched region is an area in which stereo matching is not accurately performed, and includes an area (d 1 ) existing only in one of two color images for a predetermined scene, a region A region d 2 , an area in which a texture is repeated, and an area d 3 in the same color.

Specifically, the detection unit 210 may include a classifier 211 and a stereo matcher 212.

The classifier 211 can classify the coincidence points corresponding to each other in the two color images for a predetermined scene and the inconsistency points excluding the coincidence points. For example, the classifier 211 may classify any pixel in the first color image with the second color image, and classify it as a non-match point if the corresponding result is a one-to-one correspondence or a one-to-many correspondence. The stereo matching unit 212 may perform stereo matching based on the set of matching points classified in the classifier 211 to predict the depth information at each matching point. At this time, the method of performing the stereo matching can be applied to the method already disclosed.

The detection unit 210 determines the set of discordant points classified by the classifier 211 as a non-matching region. For example, in the D image of FIG. 7, a portion indicated by black is a set of coincidence points, and a portion indicated by white is a set of discrete points.

The extracting unit 220 extracts the unmatched depth information corresponding to the non-matching area detected by the detecting unit 210 in the depth image obtained from the one or more depth cameras 120 photographing a predetermined scene.

8 is a diagram illustrating an example of operations of the extracting unit and the complementing unit shown in FIG. As shown in FIG. 8, the unmatched depth information may be depth information for the e 1 , e 2 , and e 3 regions in the depth image E.

The complementing unit 230 fills the non-matching area detected by the detecting unit 210 as shown in FIG. 8 using the non-matching depth information extracted by the extracting unit 220.

Particularly, the partial area corresponding to the extracted unmatched depth information is filled using the extracted unmatched depth information, and the remaining area excluding the partial area is filled with the unmatched depth Based on information. At this time, the process of filling some areas can be performed through a known projection matrix, a camera parameter defining a relationship between the two-dimensional image and the three-dimensional space, or other algorithms.

For example, the unmatched depth information extracted by the extracting unit 220 may be depth information for the e 1 , e 2 , and e 3 regions in the depth image E, Some of the regions corresponding to e 1 , e 2 , and e 3 may be regions d 1 , d 2 , and d 3 . The complementary unit 230 may fill the regions d 1 , d 2 , and d 3 using the depth information of the e 1 , e 2 , and e 3 regions.

Since the depth image obtained from the depth camera 120 contains depth information having a higher accuracy than the result of the stereo matching, the complementary unit 230 may generate a displacement map in which a part of the non- Fill with the extracted unmatched depth information. Through such a process, it is possible to reliably compensate the non-matching region, which is an area in which the stereo matching is inaccurately performed.

Also, the complementary unit 230 fills the remaining area based on the unmatched depth information used to fill a part of the area, which will be described with reference to FIGS. 10 to 11. FIG.

The hybrid method is a method of enhancing the stereo matching result with the help of the depth image obtained mainly from the depth camera. The method of mixing the result of the stereo matching and the depth information contained in the depth image, The method of limiting the range of displacement is representative.

9 is a diagram for explaining a conventional hybrid method for acquiring depth information. In particular, FIG. 9 shows a technique for restricting the displacement range of each pixel to depth information obtained from a depth camera in order to improve the accuracy of the stereo matching process. In this method, the displacement values (30, 45, 49, etc.) obtained from the depth images are arranged in the corresponding regions in the color image, and the displacement range of the pixels which do not know the depth information in the color image Limit. It can be finally predicted that the displacement value of the F pixel in the first case is predicted initially at 45 and is within the range of the displacement including 45 considering the result of the stereo matching. In the case of the second case, the displacement value of the F pixel is initially predicted to be about 39 in consideration of the displacement values such as 30, 45, 49, etc., and is within the range of the displacement including 39 considering the result of the stereo matching. Can be predicted. In the case of the third case, since there is no displacement value obtained from the depth image around a predetermined F pixel which does not know the depth information, and there is no object to be referred to, the displacement value of the F pixel can not be initially predicted. Therefore, it can be finally predicted that the displacement value of the F pixel exists within the range of the unclear displacement considering only the result of the stereo matching. Thus, the conventional hybrid method uses the depth information acquired from the depth camera to limit the displacement range of pixels that do not know the depth information, and acquires the depth information through the stereo matching process. Therefore, the stereo matching process, There was a problem that it had to go through.

FIGS. 10 and 11 are views showing an example for explaining the operation of the complementary unit shown in FIG. 6 for acquiring depth information. In particular, FIGS. 10 and 11 illustrate the process of filling the remaining area of the non-matching area detected by the detecting unit 210 based on the unmatched depth information used by the complementing unit 230 to fill a partial area.

In FIG. 10, the process of filling the unmatched area, which is an area in which stereo matching is not accurately performed, in units of pixels. In the case of the present invention, a partial area corresponding to the angular displacement value of the non-matching area detected by the detecting unit 210 is filled with the displacement value (30, 45, 49, etc.) obtained from the depth image. Subsequently, based on the displacement values (30, 45, 49, etc.) used to fill some of the areas, the remaining areas of the unmatched areas detected by the detector 210 are filled. That is, in filling the non-matching region, the complementing unit 230 can fill the non-matching region based on only the non-matching depth information extracted in the depth image, without considering the result of the stereo matching.

For example, in the case of the first case, the complementary unit 230 fills the partial area g 1 corresponding to the displacement value 45 using the displacement value 45 extracted by the extraction unit 220, The remaining area h 1 may be filled based on the displacement value 45 used to fill the partial area g 1 . In the case of the second case, the complementary unit 230 fills the partial area g 2 corresponding to the displacement values using the displacement values 30, 45, and 49 extracted by the extraction unit 220, (h 2 ) based on the displacement values (30, 45, 49) used to fill the gaps (g 1 , g 2 ). In the case of the third case, there is no partial region filled with the displacement value extracted by the extraction unit 220 in the same size region as the first case or the second case. However, there is a partial area g 3 corresponding to these displacement values using the displacement values 30 and 45 extracted by the extraction unit 220 in a wider area. Therefore, the complementary part 230 can fill the remaining area h 3 based on the displacement values 30 and 45 used to fill the partial area g 3 .

11 illustrates a process of filling a non-matching region, which is an area in which stereo matching is not accurately performed, using the non-matching depth information extracted in the depth image. That is, the extracting unit 220 extracts the unmatched depth information in the depth image, and the complementing unit 230 projects the unmatched depth information to a partial area (g 4 , g 5 , g 6 ) of the unmatched area, the rest of the non-matching non-matching area on the basis of depth information used in (h 4, h 5, h 6) fills.

In operation of the complementary unit 230 as described above with reference to FIGS. 10 and 11, the criteria for filling the non-conformal region may vary. The complementary unit 230 may fill the remaining area with the unmatched depth information used to fill a part of the area, apply a method of considering the difference in distance between the part of the area and the remaining area, You can fill the area. In other words, the complementary unit 230 may fill the remaining area having the same depth information or displacement value as some of the unmatched areas, taking into consideration the distance and color difference.

As described above, the stereoscopic image production system proposed in the present invention can fill the non-matching region in a state in which the result of the inaccurate stereo matching is excluded. In addition, by filling the partial area of the unmatched area using the unmatched depth information extracted in the depth image and filling the remaining area of the unmatched area based on the unmatched depth information used to fill the partial area, Depth information with high reliability and accuracy can be obtained. In addition, the stereoscopic image production system can produce a high quality stereoscopic image based on the depth information of each scene acquired by the complementary unit 230 described above.

Hereinafter, a method of acquiring depth information for stereoscopic image production will be described with reference to FIG. 12 is a flowchart for explaining a depth information acquisition method according to an embodiment of the present invention. For this purpose, the above-described stereoscopic image production system may be utilized, but is not limited thereto. However, for convenience of explanation, a method of acquiring depth information using a stereoscopic image production system will be described.

First, a predetermined scene is photographed using a plurality of color cameras 110 and one or more depth cameras 120 (S110).

The stereoscopic image production system detects an unmatched area from a result of stereo matching performed based on the color image obtained from the color camera 110 (S120). Here, the unmatched area is an area in which stereo matching is not precisely performed as shown in FIG. 7, and includes an area (d 1 ) existing only in one of two color images for a predetermined scene, A region d 2 covered by the texture, a region where the texture is repeated, and a region d 3 made of the same color.

Specifically, the stereoscopic image production system can classify the coincidence points corresponding to each other in the two color images for a predetermined scene and the inconsistency points excluding the coincidence points. For example, if a result of mapping any pixel in the first color image to the second color image has a one-to-one correspondence or a one-to-many correspondence, the corresponding pixel can be classified as a non-match point. In addition, the stereoscopic image production system can perform stereo matching based on a set of matching points, predict depth information at each matching point, and can determine a set of matching points as an unconformity region.

Subsequently, the stereoscopic image production system extracts the unmatched depth information corresponding to the detected unmatched area in step S120, which is detected in the depth image obtained from the depth camera 120 (S130).

Subsequently, the stereoscopic image production system determines a partial area corresponding to the unmatched depth information extracted in step S130 of extracting the detected unmatched area in step S120 (S140).

In addition, the stereoscopic image producing system may include a step S120 of detecting a non-coincidence depth information extracted in the extracting step S130, based on the uncoordinated depth information used to fill a part of the area, The remaining area of the non-matching area detected in step S150 is filled.

Specifically, a distance difference between a partial region and the remaining region and a color difference between the partial region and the remaining region may be analyzed. According to the result of the analysis, the remaining area of the non-matching area can be filled. That is, in the case of the step of filling the non-matching region (S150), only the result of the inaccurate stereo matching can be excluded and only the unmatched depth information extracted in the depth image can be used.

In addition, the stereoscopic image production system includes a color image, a depth image, a result image of stereo matching, an image in which an inconsistent area detected in step S120 is highlighted, a filling step S150), at least one of the depth image and the stereoscopic image is displayed to the user (S160).

As described above, by utilizing the depth information acquiring method proposed in the present invention, the remaining area of the unmatched area is filled based on the unmatched depth information used to fill a part of the area, It is possible to obtain depth information on the scene of the scene. The depth information of each scene thus obtained can be used to produce a high quality stereoscopic image that can enhance the user's satisfaction.

Meanwhile, each of the components shown in FIG. 6 may be configured as a 'module'. The term 'module' refers to a hardware component such as software or a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the module performs certain roles. However, a module is not limited to software or hardware. A module may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. The functionality provided by the components and modules may be combined into a smaller number of components and modules or further separated into additional components and modules.

While the apparatus and method of the present invention has been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

In addition, an embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: photographing apparatus 110: color camera
120: depth camera 130: structure
200: video authoring device 300: display device

Claims (13)

In a stereoscopic image production system,
A detection unit for detecting an unmatched area from a result of stereo matching performed based on a color image obtained from a plurality of color cameras that photographs a predetermined scene;
An extracting unit for extracting unmatched depth information corresponding to the detected unmatched area in a depth image obtained from at least one depth camera that photographs the scene; And
The method comprising the steps of: filling a partial area of the detected unmatching area corresponding to the extracted unmatching depth information using the extracted unmatching depth information; A complementary portion that fills the remaining region of the unmatched region,
And a stereoscopic image producing unit for producing a stereoscopic image based on the depth information of each scene acquired by the supplement unit.
The method of claim 1, wherein the unmatched area includes a region existing only in one of the two color images for the scene, a region in which a texture is repeated in the color image for the scene, Comprising a stereoscopic image producing system.
The apparatus of claim 1, wherein the detecting unit
A classifier for classifying the coincidence points corresponding to each other in the two color images for the scene and the inconsistency points excluding the coincidence points; And
And a stereo matching unit for performing the stereo matching on the basis of the set of matching points to predict depth information at the matching point,
And determines the set of the inconsistency points as the non-matching region.
2. The stereoscopic image producing apparatus according to claim 1, wherein the complementary unit is configured to fill the remaining area with the unmatched depth information used to fill the partial area, and to fill the remaining area with considering the distance and color difference between the partial area and the remaining area, system.
In a stereoscopic image production system,
A photographing apparatus including a plurality of color cameras and at least one depth camera for photographing a predetermined scene;
And an image authoring device for acquiring depth information on the scene based on the color image obtained from the color camera and the depth image obtained from the depth camera and for authoring the stereoscopic image based on the depth information for each scene, ,
The video authoring device
A detection unit for detecting an unmatched region from a result of stereo matching performed based on the color image;
An extracting unit for extracting non-registration depth information corresponding to the detected non-registration area in the depth image; And
The method comprising the steps of: filling a partial area of the detected unmatching area corresponding to the extracted unmatching depth information using the extracted unmatching depth information, and detecting the detection area based on the unmatching depth information used to fill the partial area A complementary portion filling the remaining region of the unmatched region
Stereoscopic image production system.
The method according to claim 5, wherein the color image, the depth image, the result image of the stereo matching, the image in which the detected unmatching region is highlighted in the resultant image, the depth image after the unmating region is filled by the complementing unit And a display device for displaying at least one of the stereoscopic images to a user.
6. The method of claim 5, wherein the unmatched area comprises a region existing only in one of the two color images for the scene, an area obscured by the object in the color image for the scene, And a region of the same color.
A method for acquiring depth information for stereoscopic image production,
Capturing a predetermined scene using a plurality of color cameras and at least one depth camera;
Detecting an unmatched region from a result of stereo matching performed based on the color image obtained from the color camera;
Extracting non-registration depth information corresponding to the detected non-registration area in the depth image obtained from the depth camera;
Determining a partial area corresponding to the extracted unmatched depth information among the detected unmatched areas; And
Filling the partial area using the extracted unmatched depth information and filling the remaining area of the detected unmatched area based on the unmatched depth information used to fill the partial area,
Depth information acquisition method.
9. The method of claim 8, wherein detecting
Classifying the coincident points except for the coincidence points and the coincidence points corresponding to each other in the two color images of the scene; And
And performing the stereo matching based on the set of matching points to predict depth information at the matching points,
And the set of inconsistencies is determined as the unmatched region.
9. The method of claim 8, wherein the filling step includes analyzing a distance and a color difference between the partial area and the remaining area,
And filling the remaining area according to a result analyzed in the analyzing step.
9. The method as claimed in claim 8, further comprising displaying the color image, the depth image, the resultant image of the stereo matching, and the image obtained by highlighting the detected unmatched area in the resultant image through the display device, And displaying at least one of the depth image after the filling and the stereoscopic image to the user.
9. The method of claim 8, wherein the unmatched area comprises a region existing only in one of the two color images for the scene, an area obscured by the object in the color image for the scene, And a region of the same color.
A computer-readable recording medium recording a program for performing each step of the method according to any one of claims 8 to 12 on a computer.
KR1020130033350A 2013-03-28 2013-03-28 System for producing stereo-scopic image or video and method for acquiring depth information KR20140118083A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020130033350A KR20140118083A (en) 2013-03-28 2013-03-28 System for producing stereo-scopic image or video and method for acquiring depth information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020130033350A KR20140118083A (en) 2013-03-28 2013-03-28 System for producing stereo-scopic image or video and method for acquiring depth information

Publications (1)

Publication Number Publication Date
KR20140118083A true KR20140118083A (en) 2014-10-08

Family

ID=51990992

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020130033350A KR20140118083A (en) 2013-03-28 2013-03-28 System for producing stereo-scopic image or video and method for acquiring depth information

Country Status (1)

Country Link
KR (1) KR20140118083A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017026705A1 (en) * 2015-08-07 2017-02-16 삼성전자 주식회사 Electronic device for generating 360 degree three-dimensional image, and method therefor
WO2017112139A1 (en) * 2015-12-21 2017-06-29 Intel Corporation Enhanced imaging
US10595004B2 (en) 2015-08-07 2020-03-17 Samsung Electronics Co., Ltd. Electronic device for generating 360-degree three-dimensional image and method therefor
CN112652005A (en) * 2019-10-12 2021-04-13 宅妆股份有限公司 Method and system for generating three-dimensional pattern
CN113465252A (en) * 2020-05-29 2021-10-01 海信集团有限公司 Intelligent refrigerator and drawer state detection method in intelligent refrigerator

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017026705A1 (en) * 2015-08-07 2017-02-16 삼성전자 주식회사 Electronic device for generating 360 degree three-dimensional image, and method therefor
US10595004B2 (en) 2015-08-07 2020-03-17 Samsung Electronics Co., Ltd. Electronic device for generating 360-degree three-dimensional image and method therefor
WO2017112139A1 (en) * 2015-12-21 2017-06-29 Intel Corporation Enhanced imaging
CN112652005A (en) * 2019-10-12 2021-04-13 宅妆股份有限公司 Method and system for generating three-dimensional pattern
CN113465252A (en) * 2020-05-29 2021-10-01 海信集团有限公司 Intelligent refrigerator and drawer state detection method in intelligent refrigerator

Similar Documents

Publication Publication Date Title
EP2887311B1 (en) Method and apparatus for performing depth estimation
WO2019149206A1 (en) Depth estimation method and apparatus, electronic device, program, and medium
CN109360235B (en) Hybrid depth estimation method based on light field data
EP2992508B1 (en) Diminished and mediated reality effects from reconstruction
Zhu et al. Depth image based view synthesis: New insights and perspectives on hole generation and filling
US9237330B2 (en) Forming a stereoscopic video
KR102464523B1 (en) Method and apparatus for processing image property maps
KR100560464B1 (en) Multi-view display system with viewpoint adaptation
KR20100087685A (en) Method and apparatus for improving quality of depth image
US11803980B2 (en) Method for generating layered depth data of a scene
WO2019244944A1 (en) Three-dimension reconstruction method and three-dimension reconstruction device
KR20140118083A (en) System for producing stereo-scopic image or video and method for acquiring depth information
US9639944B2 (en) Method and apparatus for determining a depth of a target object
CN113711276A (en) Scale-aware monocular positioning and mapping
KR100926520B1 (en) Apparatus and method of matching binocular/multi-view stereo using foreground/background separation and image segmentation
EP2932710B1 (en) Method and apparatus for segmentation of 3d image data
KR20110060180A (en) Method and apparatus for producing 3d models by interactively selecting interested objects
CN116704111A (en) Image processing method and apparatus
Orozco et al. HDR multiview image sequence generation: Toward 3D HDR video
EP2932466B1 (en) Method and apparatus for segmentation of 3d image data
Nam et al. Hole‐Filling Methods Using Depth and Color Information for Generating Multiview Images
Ayatollahi et al. A taxonomy of depth map creation methods used in multiview video compression
KR101632069B1 (en) Method and apparatus for generating depth map using refracitve medium on binocular base
Kang et al. Generation of multi-view images using stereo and time-of-flight depth cameras
Yu et al. Combined hole-filling with spatial and temporal prediction

Legal Events

Date Code Title Description
WITN Withdrawal due to no request for examination