TWI627604B - System and method for generating depth map using differential patterns - Google Patents

Abstract

A method of generating a depth map is disclosed. The method includes generating a first candidate depth map in response to a first pair of images associated with a first textured pattern; generating a second candidate depth map in response to a second pair of images associated with a second textured pattern The second texture pattern is different from the first texture pattern; determining a pixel that is more reliable than another pixel in the pixel of the first candidate depth map and the second candidate depth map; and based on The one pixel produces a depth map.

Description

System and method for generating depth map by using differentiated pattern

The present invention relates to a system and method for generating a depth map using a differentiated pattern.

Parallax estimation or deep extraction is the subject of many years of attention. Parallax or depth represents the distance between the object and the measuring device. In order to extract depth information of objects in the scene, stereo matching is used to estimate the parallax distance between one of the stereo images or the corresponding pixels in the video captured from the parallel cameras. Stereo matching has many applications, such as three-dimensional (3D) gesture recognition, robot imaging, vehicle industry, perspective synthesis, and television (television, TV). Although stereo matching has many advantageous features and has been widely used, there are still many limitations. For example, if the object is untextured, it may be difficult to obtain a dense and high quality depth map. Stereo matching finds the corresponding points between the two images and calculates the 3D depth information. When the texture in the scene is low or repeating, stereo matching is difficult to obtain accurate depth. As a result, the untextured surface cannot be properly stereo-matched.

The present invention is directed to an imaging system and method for generating a depth map by differentiating structured light and reliability maps.

An imaging system including a candidate depth map generation module is provided in accordance with an embodiment of the present invention. A group, a reliability determination module, and a depth map forming module. The candidate depth map generating module is configured to generate a first candidate depth map in response to a first pair of images associated with a first textured pattern, and in response to a second pair associated with a second textured pattern The image produces a second candidate depth map that is different from the first textured pattern. The reliability determination module is configured to determine a pixel that is more reliable than another pixel in the pixel of the first candidate depth map and the second candidate depth map. The depth map forming module is configured to generate a depth map based on the one pixel.

In an embodiment, the reliability determination module includes a reliability calculation module for generating a first reliability map and generating a second reliability map, wherein the first reliability map includes Information about the reliability of the pixels in a candidate depth map, and the second reliability map includes information about the reliability of the pixels in the second candidate depth map.

In another embodiment, the reliability determination module includes a reliability comparison module for comparing the first reliability map with the second reliability map to identify the more reliable pixels.

In yet another embodiment, the first texturing pattern has a translational displacement relative to the second texturing pattern.

In still another embodiment, the first texturing pattern has an angular displacement relative to the second texturing pattern.

In yet another embodiment, the first texturing pattern relates to a different pattern than the second texturing pattern.

Some embodiments in accordance with the present invention provide a method of generating a depth map. According to the method, the first structured light system is projected onto an object. Furthermore, a first candidate depth map associated with the first structured light is generated, and a first reliability map is generated, the first reliability map including a first one in the first candidate depth map Information on the reliability value of a first pixel of the location. this Additionally, a second structured light system is projected onto the object, wherein the second structured light produces a textured pattern that is different from the first textured light. Furthermore, a second candidate depth map associated with the second structured light is generated, and a second reliability map is generated, the second reliability map including a second in the second candidate depth map Information of a reliability value of a second pixel of the location, wherein the second location in the second candidate depth map is the same as the first location in the first candidate depth map. Subsequently, one of the first pixel and the second pixel having a large reliability value is determined to be a third pixel. Then, using the third pixel, a depth map is generated.

A method of generating a depth map is also provided in accordance with an embodiment of the present invention. According to the method, a first depth map of the first pixel is generated and a first reliability map is generated based on a first texture pattern, the first reliability map including information about the reliability of the first pixel . Moreover, based on a second texturing pattern, a second depth map of the second pixel is generated and a second reliability map is generated, the second reliability map including information about the reliability of the second pixel. Moreover, based on a third texturing pattern, a third depth map of the third pixel is generated and a third reliability map is generated, the third reliability map including information about the reliability of the third pixel. Subsequently, by comparing the first reliability map, the second reliability map, and the third reliability map, the first reliability map, the second reliability map, and the third reliability map are the same A more reliable one of the first pixel, the second pixel, and the third pixel is identified, and a depth map is generated using the pixel.

The features and technical advantages of the present invention are broadly described in the foregoing, so that the detailed description of the invention will be understood. Additional features and advantages of the invention will be set forth in the description of the appended claims. It is to be understood by those of ordinary skill in the art that the present invention may be utilized as a basis for the modification or design of other structures or processes. Those with ordinary knowledge in the technical field should also It is to be understood that the equivalents are not to be construed as being limited to the spirit and scope of the invention as set forth in the appended claims.

100‧‧‧ system

10‧‧‧Camera and projector assembly

11‧‧‧ Stereo camera

11L‧‧‧Sensor or camera

11R‧‧‧Sensor or camera

12‧‧‧Projector

14‧‧‧A pair of original images

15‧‧‧Correction and correction module

16‧‧‧ imaging system

18‧‧‧Depth map

28‧‧‧ objects

48‧‧‧ cross check module

71‧‧‧ operation

72‧‧‧ operations

73‧‧‧ operations

74‧‧‧ operations

75‧‧‧ operation

76‧‧‧ operations

77‧‧‧ operations

78‧‧‧Operation

81‧‧‧ operation

82‧‧‧ operations

83‧‧‧ operation

84‧‧‧ operation

85‧‧‧ operation

86‧‧‧ operation

87‧‧‧ operation

88‧‧‧ operation

91‧‧‧First image

92‧‧‧Second image

95‧‧‧ (candidate) depth map

97‧‧‧reliability map

102‧‧‧ operation

104‧‧‧Operation

106‧‧‧ operation

108‧‧‧ operation

110‧‧‧ operation

112‧‧‧ operation

151‧‧‧ first image

152‧‧‧second image

162‧‧‧ candidate depth map generation module

165‧‧‧reliability determination module

168‧‧‧Deep map forming module

280‧‧Deep map

281‧‧‧First candidate depth map

282‧‧‧Second candidate depth map

401‧‧‧ first census conversion module

402‧‧‧Second census conversion module

411‧‧‧First Cost Calculation and Addition Module

412‧‧‧ Second Cost Calculation and Addition Module

421‧‧‧First Cost Addition Module

422‧‧‧Second cost plus module

431‧‧‧First parallax calculation module

432‧‧‧Second Parallax Calculation Module

451‧‧‧The first winner is the full module

452‧‧‧Second winners take all modules

461‧‧‧reliability calculation module

462‧‧‧Reliability comparison module

471‧‧‧Reliability calculation module

472‧‧‧Reliability comparison module

481‧‧‧ cross check module

48‧‧‧ cross check module

P1‧‧‧ first pattern

P2‧‧‧ second pattern

C1‧‧‧ first position

C2‧‧‧second position

C‧‧‧ same location

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is further described with reference to the accompanying drawings in which: FIG. 1 is a block diagram of a system for generating a depth map in accordance with an embodiment of the present invention; 1 is a schematic diagram of a camera and projector assembly shown in FIG. 1; FIG. 3 is a schematic diagram showing a conceptual model for generating a depth map by using a differentiated light pattern according to an embodiment of the present invention; FIG. 4A is implemented according to the present invention. 1B is a block diagram of the imaging system shown in FIG. 1; FIG. 5B is a block diagram of the imaging system shown in FIG. 1 according to another embodiment of the present invention; FIG. 5A is a schematic diagram of an exemplary pattern of structured light; 5C is a schematic diagram of a differentiated pattern according to some embodiments of the present invention with respect to the exemplary pattern depicted in FIG. 5A; FIG. 6A is a schematic diagram of another exemplary pattern; FIG. 6B is a differential according to some embodiments of the present invention. A schematic diagram of a pattern relative to the exemplary pattern depicted in FIG. 6A; FIG. 7 is a flow chart illustrating a method of generating a depth map by using a differentiated pattern, in accordance with an embodiment of the present invention; FIG. 8 is another embodiment in accordance with the present invention. Example illustrates by using the difference pattern to produce a flowchart of a method of depth map; FIG. 9 shows a schematic system by using the difference pattern to produce a depth map of the conceptual model of a further embodiment of the present invention embodiment; and FIG. 10 is a flow chart showing a method of generating a depth map by using a differentiated pattern according to still another embodiment of the present invention.

The embodiments of the present invention are illustrated in the following description, and the same or similar components are designated by like reference numerals.

1 is a block diagram of a system 100 for generating a depth map in accordance with an embodiment of the present invention. Referring to FIG. 1, system 100 includes a camera and projector assembly 10, a correction and correction module 15 and an imaging system 16.

The camera and projector assembly 10 includes a stereo camera 11 and a projector 12. The stereo camera 11 captures a pair of original images of objects in the scene from different perspectives in the field of view. Objects can be low or even textureless. The projector 12 illuminates the object with patterned structured light. With this pattern, the structured light provides a textured pattern on the object and facilitates the system 100 to produce a precise depth map. As a result, the camera and projector assembly 10 provides a pair of original images 14 having a textured pattern to the correction and correction module 15.

The correction module 15 corrects the original image 14 to remove the lens distortion and correct the original image 14 to remove the coplanar and polar mismatches, so that the pair of output images including the first image 151 and the second image 152 can be in single or multiple Line-to-line comparisons are based.

The imaging system 16 includes a candidate depth map generation module 162, a reliability determination module 165, and a depth map formation module 168. The candidate depth map generation module 162 generates a first candidate depth map in response to the first pair of images obtained using the first structured light, and generates a second candidate depth map in response to the second pair of images obtained using the second structured light. The first structured light and the second structured light exhibit a differential textured pattern on the object 28 when projected onto the object 28. Each of the first candidate depth map and the second candidate depth map includes depth information, such as a depth value, on each pixel. Reliability determination module 165 Determine the reliability (or reliability) of the depth information. Furthermore, the reliability determination module 165 generates a first reliability map on each pixel in the first reliability map, which includes reliability information, such as a reliability value, and is displayed on each pixel in the second reliability map. A second reliability map is generated that includes reliability information. The pixels at the same position of the first candidate depth map and the second candidate depth map are compared with each other in reliability. One of the pixels having a larger reliability value at the same position of the first candidate depth map and the second candidate depth map is identified. The depth map forming module 168 generates the depth map 18 by using the identified pixels as pixels at the same position in the depth map 18.

The term "depth map" is commonly used in three-dimensional (3D) computer graphics applications to describe images containing information about the distance from the camera's perspective to the surface of the object in the scene. The depth map 18 provides distance information of the object from the stereo camera 11 in the scene. The depth map 18 is used for implementation, such as 3D gesture recognition, perspective synthesis, and stereoscopic television presentation.

2 is a schematic illustration of the camera and projector assembly 10 shown in FIG. 1 in accordance with an embodiment of the present invention. Referring to FIG. 2, stereo camera 11 includes two sensors or cameras 11L and 11R aligned on a pole line to capture a pair of original images or video of object 28. Depending on the application, the cameras 11L and 11R can be integrated in one device or separately.

Projector 12 emits structured light onto object 28 in the field of view of projector 12. The emitted structured light has a pattern, which may include stripes, spots, dots, triangles, ruled lines, or the like. In the present embodiment, the cameras 11L and 11R are disposed on a common side of the projector 12. In another embodiment, the projector 12 is disposed between the cameras 11L and 11R. Moreover, the cameras 11L, 11R and the projector 12 can be integrated into one device or separately configured as in this embodiment to suit different applications.

In an embodiment, the projector 12 may include, for example, an infrared light laser having a wavelength of from 700 nanometers (nm) to 3,000 nm, including near-infrared light having a wavelength of 0.75 micrometers (mm) to 1.4 mm, having a wavelength of from 3 mm to 8 mm. Medium wavelength infrared light and long wavelength red with a wavelength of 8mm to 15mm External light. In another embodiment, projector 12 can include a light source that produces visible light. In still another embodiment, projector 12 can include a source of ultraviolet light. Moreover, the light generated by the projector 12 is not limited to any particular wavelength as long as the light can be detected by the cameras 11L and 11R. The projector 12 may also include a diffractive optical element (DOE) that receives the laser light and outputs a plurality of diffracted beams. Typically, DOE is used to provide multiple smaller beams, such as thousands of smaller beams, from a single collimated beam. Each of the smaller beams has a fraction of the power of a single collimated beam and is smaller and the diffracted beams can have a nominally equal intensity.

3 is a schematic diagram showing a conceptual model for generating a depth map by using a differential structured light pattern, in accordance with an embodiment of the present invention. Referring to FIG. 3, also referring to FIG. 2, the first structured light having the first pattern P1 (shown in dashed circle) is emitted by the projector 12 onto the object 28 toward the first position C1. The first position C1 is, for example, the geographic center or centroid of the first pattern P1. The image of the object 28 having the first textured pattern produced by the first structured light is captured by the stereo camera 11. Imaging system 16 generates a first candidate depth map 281 and a first reliability map. In the first candidate depth map 281, the pixel system in the region substantially substantially surrounding the first position C1 (shown in solid lines) is more likely to have a greater confidence value than the pixels in other regions (shown in dashed lines) and Therefore their depth values are more reliable.

Subsequently, the second structured light having the second pattern P2 (shown in dashed circle) is emitted by the projector 12 onto the object 28 to the second position C2. Similarly, the second location C2 is the geographic center or centroid of the second pattern P2. The image of the object 28 having the second textured pattern produced by the second structured light is captured by the stereo camera 11. The first textured pattern and the second textured pattern are different from each other. The difference in the textured pattern results from moving or changing the position of the second position C2 relative to the first position C1, as indicated by the arrows. Imaging system 16 generates a second candidate depth map 282 and a second reliability map. Similarly, in the second candidate depth map 282, the pixel system in the region (shown in solid lines) substantially surrounding the second location C2 is more likely to be larger than in other regions (with a dashed line) The confidence values of the pixels in the display are displayed and thus their depth values are more reliable.

Comparing the first reliability map with the second reliability graph by pixels across the first candidate depth map and the second candidate depth maps 281 and 282, having greater than the first candidate depth map and the second candidate depth map 281 Pixels of the confidence value of the other of the same position as 282 are identified. The identified pixels are selected from the first candidate depth map and the second candidate depth maps 281 and 282 according to the reliability values, and are filled in the depth map 280 to form a depth map 280. Since each pixel in depth map 280 represents a maximum confidence value, depth map 280 is more reliable and therefore more accurate than first candidate depth map and second candidate depth maps 281 and 282.

4A is a block diagram of the imaging system 16 shown in FIG. 1 in accordance with an embodiment of the present invention. Referring to FIG. 4A, the imaging system 16 includes a first cost calculation and summation module 411, a second cost calculation and summation module 412, a first parallax calculation module 431, a second parallax calculation module 432, and a reliability calculation module. The group 461, the reliability comparison module 462, the cross-check module 481, and the depth map forming module 168.

The first cost calculation and summation module 411 including the first window buffer (not shown) is used to obtain the correlation line of the first image 151, calculate the correct matching cost of the correlation line of the first image 151, and use The first window buffer adds the total matching cost. Similarly, the second cost calculation and summation module 412 including the second window buffer (not shown) is used to obtain the correlation line of the second image 152, and the correct matching cost of the correlation line of the second image 152 is calculated. And use the second window buffer to add the total matching cost. When the first textured pattern is projected on the object, the first image 151 and the second image 152 of the object are captured.

The difference in image positions of the objects seen by the left and right cameras 11L and 11R is calculated by the first parallax calculation module 431 and the second parallax calculation module 432, and the first parallax map and the second parallax map are obtained, respectively. Based on the first disparity map and the second disparity map, the reliability calculation module 461 generates a first reliability map related to the first texturing pattern. Subsequently, the reliability calculation module 461 generates and the second pattern A second reliability map related to physical and chemical patterns. The first reliability map and the second reliability map are compared with each other on the basis of pixel-to-pixel by the reliability comparison module 462 to determine the reliability of the pixel.

In addition, the cross-check module 481 is configured to cross-check the first reliability map and the second reliability map to identify one or more mismatched parallax levels between the first disparity map and the second disparity map. As a result, a first candidate depth map associated with the first texturing pattern is obtained. Subsequently, a second candidate depth map associated with the second texturing pattern is obtained. The depth map forming module 168 generates a depth map based on the comparison result from the reliability comparison module 462 and the candidate depth map from the cross module 481.

4B is a block diagram of the imaging system 16 shown in FIG. 1 in accordance with an embodiment of the present invention. Referring to FIG. 4B, in addition to including the depth map module 168, the imaging system 16 further includes a first census conversion module 401, a first cost summation module 421, and a first winner-take-all (winner-take-all, The WTA module 451, the second census conversion module 402, the second cost addition module 422, the second WTA module 452, the reliability calculation module 471, the reliability comparison module 472, and the cross check module 482. In stereo matching, since parallax estimation and cross-checking are known methods, their functions are briefly discussed below.

The first census conversion module 401 considers, for example, only 1 to 4 closest neighboring pixels, and obtains 1 to 4 binary digits representing higher or lower image intensities compared to pixels processed in the first image 151. Similarly, the second census conversion module 402 considers 1 to 4 nearest neighbor pixels to obtain 1 to 4 binary digits representing higher or lower image intensities compared to pixels processed in the second image 152. . Next, in order to determine the matching cost, the binary numbers calculated from the first census conversion module 401 and the second census conversion module 402 have different parallax distances compared to each other. In the first cost summation module 421 and the second cost summation module 422, the matching cost indicating the similarity between the pixels between the first image 151 and the second image 152 can be obtained by using a moving window having a reasonable size. , add up to the level of each parallax. Then, the total cost is sent to the first WTA module. The 451 and the second WTA module 452 are used to find the parallax with the least cost, and the parallax with the least cost is used as the determined disparity for the pixel. Subsequently, by comparing the parallax results from the first WTA module 451 and the second WTA module 452, the cross-check module 48 corrects most of the non-parallax by referring to the parallax of the surrounding area determined by the edge of the object in the disparity map. Reliable depth results.

The reliability calculation module 471 and the reliability comparison module 472 constitute a reliability determination module 165 that is described and illustrated in reference to FIG. After the cost summation phase, the costMap(x, y, d) is obtained, where x and y represent the position of the current pixel, and d represents the disparity. The costMap(x, y, d) records the matching cost between the first image and the second image 151, 152 of each pixel having different parallaxes. After the cost summation phase, the reliability calculation module 471 generates a reliability map by calculating a surrogate value for each pixel. The minimum generation value (min_cost) is expressed as the best matching disparity level of the current pixel. The average value of AvgCost(x,y) here is calculated by the following formula:

Where "totalCost(x, y)" indicates that the current pixel (x, y) has a sum of the values of the disparity levels, and "N" represents the total number of disparity levels. For the pixel (x, y), by subtracting min_cost from AvgCost, we can obtain the corresponding reliability of the current pixel (x, y): CL ( x , y ) = AvgCost ( x , y )-min_cos t ( x , y )

In general, for a desired depth value, min_cost should be close to zero and the difference between AvgCost and min_cost should be as large as possible. As a result, the more reliable the depth value, the greater the reliability value.

The reliability comparison module 472 compares the first reliability map with the second reliability map, and determines pixels having a larger reliability value in the first reliability map and the second reliability map for each pixel position. based on The depth map forming module 168 generates a depth map 18 after the reliability comparison module 472 has identified the pixels.

In the present embodiment, the reliability calculation module 471 is coupled to the first cost summation module 421 to determine the reliability map. In another embodiment, the reliability calculation module 471 is coupled to the second cost summation module 422 instead of the first cost total module 421. In yet another embodiment, the first reliability calculation module is coupled to the first cost summation module 421 and the second reliability calculation module is coupled to the second cost addition module 422. Further, in order to determine the reliability map, the reliability calculation module 471 is not limited to the above-described specific expression. Moreover, the reliability calculation module 471 may not be coupled to the first cost summation module 422 or the second cost addition module 421. As a result, other algorithms or mechanisms for determining the reliability map in imaging systems that use differentially structured light patterns are also within the scope of the present invention.

The imaging system 16 can be implemented in a hardware, such as in a Field Programmable Gate Array (FPGA) or in an Application-Specific Integrated Circuit (ASIC), or for general purposes. Implemented in a software system of a computer system, or a combination thereof. Compared to software implementations, hardware implementations can achieve higher performance but higher design costs. For instant applications, hardware implementations are often chosen due to speed requirements.

Figure 5A is a schematic illustration of an exemplary pattern P1 of structured light. Referring to FIG. 5A, a first structured light system having a first pattern P1 is projected toward a first position C1.

5B and 5C are schematic illustrations of a differentiated pattern P2 relative to the exemplary pattern P1 depicted in FIG. 5A, in accordance with some embodiments of the present invention. Referring to Figure 5B and also to Figure 5A, a second structured light system having a second pattern P2 is projected toward a second position C2. The second structured light or second pattern P2 is displaced from C1 to C2 relative to the first structured light or first pattern P1. In the present embodiment, the second pattern P2 is the same as the first pattern P1 but has a translational displacement from the first pattern P1. Effectively, different texturing patterns are obtained by moving or changing the position of the structured light.

Referring to Figure 5C and also to Figure 5A, a second structured light system having a second pattern P2 is projected toward a first location C2. Furthermore, the second pattern P2 is the same as the first pattern P1. However, the second pattern P2 has an angular displacement from the first pattern P1. Effectively, by rotating the position of the structured light, different textured patterns are obtained.

6A is a schematic illustration of another exemplary pattern, and FIG. 6B is a schematic illustration of a differentiated pattern relative to the exemplary pattern depicted in FIG. 6A, in accordance with some embodiments of the present invention. Referring to Figures 6A and 6B, the first structured light and the first structured light system are projected toward the same position C. Furthermore, the first position P1 and the second position P2 are different from each other. Effectively, by using different patterns, even if the structured light system with different patterns is projected to the same position as the previously structured light, a different textured pattern is obtained.

7 is a flow chart showing a method of generating a depth map by using a differentiated pattern, in accordance with an embodiment of the present invention. Referring to Figure 7, and also by reference to system 100 illustrated in Figure 1, in operation 71, the first structured light is projected onto the object. Next, in operation 72, a first candidate depth map associated with the first structured light is generated. Moreover, in operation 73, a first reliability map is generated, the first reliability map including information about a reliability value of the first pixel at the first position of the first candidate depth map.

Subsequently, in operation 74, the second structured light is projected onto the object. The second structured light produces a textured pattern that is different from the first structured light. Next, in operation 75, a second candidate depth map associated with the second structured light is generated. Moreover, in operation 76, a second reliability map is generated, the second reliability map including information regarding a confidence value of the second pixel at the second location of the second candidate depth map. The second position in the second candidate depth map is the same as the first position in the first candidate depth map on the pixel coordinates.

In operation 77, one of the first pixel and the second pixel having a larger reliability value is used. Determined to be a third pixel. Next, in operation 78, a depth map is generated using the third pixel at the same location as the first pixel and the second pixel. Accordingly, the final depth map can be compared by comparing the reliability values of the pixels at the same position of the first reliability map and the second reliability map and filling the pixels with the larger reliability values into their depth maps. Produced in the pixel coordinates.

8 is a flow chart showing a method of generating a depth map by using a differentiated pattern, in accordance with another embodiment of the present invention. Referring to Figure 8, and also by reference to the imaging system 16 illustrated in Figure 1 or Figure 4, in operation 81, a first pair of images associated with a first texturing pattern is received. Next, in operation 82, a first depth map is generated based on the first pair of images. Moreover, in operation 83, a first reliability map is generated, the first reliability map including information regarding the reliability of the pixels in the first depth map.

Subsequently, in operation 84, a second pair of images associated with a second texturing pattern is received. The second textured pattern is different from the first textured pattern. Next, in operation 85, a second depth map is generated based on the second pair of images. Moreover, in operation 86, a second reliability map is generated, the second reliability map including information regarding the reliability of the pixels in the second depth map.

In operation 87, the first reliability map and the second reliability map are compared to determine pixels that are more reliable in the depth value at the same position of the first reliability map and the second reliability map. Next in operation 88, a third depth map is generated based on the more reliable pixels.

In the above embodiment, two (candidate) depth maps and two reliability maps are generated to determine the final depth map. However, in other embodiments, to produce a more accurate depth map, three or more (candidate) depth maps and the same number of reliability maps may be used. 9 is a schematic diagram showing a conceptual model for generating a depth map by using a differentiated pattern, in accordance with another embodiment of the present invention. Referring to Figure 9, imaging system 16 can be used to receive a first image 91 and a second image 92 of group M, the first image 91 and the second image 92 being generated in pairs using differentially structured light, M being a natural number greater than two. For each group a pair of first image and second image, the first image 91 is obtained by using the first structured light and the second image 92 is obtained by using the second structured light, the second structured light having the first Structured light with different textured patterns. Imaging system 16 generates M (candidate) depth maps 95 and M reliability maps 97, and then determines the final depth map.

For example, imaging system 16 generates a first candidate depth map and a first reliability map in response to the first pair of images obtained using the first structured light. Furthermore, imaging system 16 generates a second candidate depth map and a second reliability map in response to the second pair of images obtained using the second structured light. Next, the imaging system 16 generates a third candidate depth map and a third reliability map in response to the third pair of images obtained using the third structured light, the third structured light generating and the first structured light and the second structured light. Different textured patterns. Subsequently, to determine the final depth map, imaging system 16 compares the first reliability map, the second reliability map, and the third reliability map.

FIG. 10 is a flow chart showing a method of generating a depth map by using a differentiated pattern according to still another embodiment of the present invention. Referring to FIG. 10, and also by referring to the conceptual model illustrated in FIG. 9, in operation 102, a depth map and a reliability map of the pixels are generated based on a textured pattern. Also, in operation 104, another depth map of the pixel and another reliability map are generated based on another textured pattern that is different from the previous textured pattern. Next, in operation 106, it is determined whether a further depth map is to be generated. For example, the depth map and the reliability map of the N sets can be determined in advance to determine the final depth map. If so, in operation 108, a further depth map and a further reliability map of the pixel are generated based on a further textured pattern that is different from the previous textured pattern. Operations 106 and 108 are repeated until a predetermined number of depth maps and reliability maps are obtained. When obtained, in operation 110, by comparing the reliability maps, one of the pixels having the same position in the same position of the reliability maps is identified. Subsequently, in operation 112, a depth map is generated using the identified pixels at the same location.

In summary, the present invention provides an imaging system and method that improves the quality of a depth map without increasing system complexity by differentially structuring light and reliability. With improved depth map quality and controlled complexity, the present invention is suitable for applications such as 3D gesture recognition, perspective synthesis, and stereoscopic television.

Having described the invention and its advantages, it is understood that various changes, substitutions, and substitutions may be made without departing from the spirit and scope of the invention as defined in the appended claims. For example, the operational flow discussed above can be implemented in different ways and can be replaced by other processes or a combination thereof.

Further, the scope of the present application is not intended to be limited to the specific embodiments of the process, machine, manufacture, compositions, means, methods, and steps described in the specification. It will be readily apparent to those skilled in the art from this disclosure that the present invention may be utilized in a process, machine, manufacture, composition, method, method, or step The functions substantially the same as the corresponding embodiments described herein are achieved or substantially the same results as the corresponding embodiments described herein are achieved. Accordingly, the scope of the appended claims is intended to cover such processes, machines, manufacture, compositions, means, methods, or steps.

Claims (20)

An imaging system includes: a candidate depth map generating module for generating a first candidate depth map in response to a first pair of images associated with a first textured pattern, and responsive to a second textured pattern Correlating a second pair of images to generate a second candidate depth map, the second texture pattern is different from the first texture pattern; a reliability determination module, configured to determine the first candidate depth map and the first a pixel in the same position of the second candidate depth map is more reliable than the other pixel; and a depth map forming module for generating a depth map based on the pixel. The imaging system of claim 1, wherein the reliability determination module includes a reliability calculation module for generating a first reliability map and generating a second reliability map, the first reliability The map includes information regarding the reliability of the pixels in the first candidate depth map, and the second reliability map includes information regarding the reliability of the pixels in the second candidate depth map. The imaging system of claim 2, wherein the reliability calculation module generates the first reliability map or the second reliability map based on the following formula: ,and Where costMap (x, y, d) represents a matching cost between the first pair of images and the second pair of images, x and y represent the position of one pixel, d represents the parallax, and N represents the total number of disparity levels. The imaging system of claim 3, wherein the reliability calculation module determines the reliability of the pixel based on the following formula: Where min_cost (x, y) represents the best matching disparity level for that pixel. The imaging system of claim 2, wherein the reliability determination module comprises a reliability comparison module for comparing the first reliability map with the second reliability map to identify the more reliable pixels. The imaging system of claim 1, wherein the first texturing pattern has a translational displacement relative to the second texturing pattern. The imaging system of claim 1, wherein the first texturing pattern has an angular displacement relative to the second texturing pattern. The imaging system of claim 1, wherein the first texturing pattern relates to a pattern different from the second texturing pattern. A method of generating a depth map, the method comprising: projecting a first structured light onto an object; generating a first candidate depth map associated with the first structured light; generating a first reliability map, the first A reliability map includes information about a confidence value of a first pixel at a first location of the first candidate depth map; projecting second structured light onto the object, the second structured light production and First texturing a different texture pattern; generating a second candidate depth map associated with the second structured light; generating a second reliability map, the second reliability map including Information about a reliability value of a second pixel of a second position of the depth map, the second location in the second candidate depth map being the same as the first location in the first candidate depth map; Determining one of the first pixel and the second pixel of the greater reliability value as a third pixel; and generating a depth map using the third pixel. A method of producing a depth map according to claim 9 wherein the first structured light has a translational displacement relative to the second structured light. A method of producing a depth map according to claim 9 wherein the first structured light has an angular displacement relative to the second structured light. A method of producing a depth map according to claim 9 wherein the first structured light comprises a pattern different from the second structured light. A method for generating a depth map according to claim 9 wherein the generating the first reliability map or generating the second reliability map comprises calculating based on the following formula: ,and Where costMap (x, y, d) represents a matching cost between the first pair of images and the second pair of images, x and y represent the position of one pixel, d represents the parallax, and N represents the total number of disparity levels. The method for generating a depth map according to claim 13 , wherein generating the first reliability map or generating the second reliability map further comprises calculating according to the following formula: Where min_cost (x, y) represents the best matching disparity level for that pixel. A method for generating a depth map, the method comprising: generating a first depth map of a first pixel and a first reliability map based on a first texture pattern, the first reliability map including the first pixel Information about reliability; generating a second depth map of the second pixel and a second reliability map based on a second texture pattern, the second reliability map including information about reliability of the second pixel; Generating a third depth map of the third pixel and a third reliability map based on a third texture pattern, the third reliability map including information about reliability of the third pixel; comparing the first reliability a second reliability map and the third reliability map to identify the first pixel at the same position of the first reliability map, the second reliability map, and the third reliability map, a more reliable one of the second pixel and the third pixel; and using the one pixel to generate a depth map. A method of producing a depth map according to claim 15 wherein the first texturing pattern, the second texturing pattern and the third texturing pattern are different from each other. The method for generating a depth map according to claim 15 of the patent application, further comprising: projecting the first structured light having a first pattern onto an object to produce the first textured pattern; and projecting the second pattern The second structured light is incident on the object to produce the second textured pattern. A method of producing a depth map according to claim 17 wherein the first pattern has a translational displacement relative to the second pattern. A method of producing a depth map according to claim 17 wherein the first pattern has an angular displacement relative to the second pattern. A method of producing a depth map according to claim 17 wherein the first pattern and the second pattern are different from each other.