TWI485653B - Imaging system and method - Google Patents

Description

Imaging system and method

The present invention relates generally to the field of computer vision, and more particularly to an imaging system and method for generating a depth map.

Stereo-matching technology captures the depth of field information of an object in a scene by estimating the parallax distance between a pair of stereo images or corresponding pixels or groups of pixels in a parallel camera. Stereo matching techniques have many applications, such as three-dimensional gesture recognition, viewpoint synthesis, and stereoscopic television.

In general, imaging methods that produce depth maps fall into two categories: holistic approaches and regional approaches.

The holistic approach typically formulates the stereo matching problem as an energy function and targets the parallax function d that minimizes the overall energy. The energy function can be expressed by the following equation: E(d) = E _data (d) + λE _smooth (d) (1)

Where E _data (d) measures the degree to which the parallax function d matches the stereo image pair, and E _smooth (d) encodes the gradual assumption of the holistic method and measures the difference between the parallaxes of adjacent pixels or groups of pixels. When the formula of the energy function is determined, it can be minimized by, for example, dynamic programming, pattern segmentation, and credibility transfer.

Regional method for independent estimation of a pixel or group of pixels within a window The parallax distance, in the summation window, the matching cost of the pixel or pixel group of one of the stereo image pairs and the candidate matching pixel or pixel group of the other pair of stereo image pairs, and can find a plurality of different parallaxes The candidate matching pixel or the pixel group of the distance has the smallest matching cost to select the disparity value of the pixel or the pixel group in the disparity map.

Both of these methods have the trade-off problem of depth map quality and computational complexity. For example, for a holistic approach, the smoothness term E _smooth (d) in the energy function can be achieved by using a larger neighborhood, such as a neighborhood of 8 instead of 2, to achieve a better boundary, but optimized The computational complexity of the overall energy will be close to uncontrollable. In terms of regional methods, the computational complexity is much lower than the holistic approach, but at the expense of quality. In order to improve the quality, a higher number of bits can be used to represent a pixel or a group of pixels to obtain a more detailed disparity map, but the calculation and summation of the matching cost is performed for each pixel or group of pixels, so when each pixel When the pixel group is represented by a higher number of bits, the computational complexity will increase significantly.

Therefore, there is an urgent need to provide an imaging system and method that can improve the quality of the parallax map without imposing the complexity of the computational portion of the imaging system and method.

The present invention provides an imaging system and method for improving the quality of a parallax map by using an edge information of one of the images as a guide to identify an area on the parallax map that needs to be enhanced, so that a large amount of parallax estimation is not required. In the case of calculating the complexity of the part, the quality of the parallax map is improved. Use edge information as a guide to improve the quality of parallax maps and even reduce The complexity of small parallax estimation.

In one embodiment, the imaging system includes an edge detection module, first and second parallax estimation modules, a cross-check module, a mask refinement module, and a hole filling module. The edge detection module is configured to detect one or more edge coordinates of at least one reference line in a first image. The first and second disparity estimation modules are respectively configured to obtain a first estimated disparity map of the first image relative to a second image and a second estimated disparity map of the second image relative to the first image. A current location line. The cross-checking module is configured to cross-check the current position line of the first estimated disparity map by using the current position line of the second estimated disparity map to identify one of the current position lines of the first estimated disparity map. Pixel collection. The mask refinement module is configured to refine the masked pixel set to generate a refined masked pixel set. The hole filling module is configured to fill the refined set of masked pixels in the current position line of the first estimated disparity map to obtain a current position line of a first refinement disparity map.

In an embodiment, the occlusion refinement module includes a refinement basic register, and the occlusion refinement module is configured to: utilize the refinement basic register and the first one including the occlusion pixel set Estimating the current position line of the disparity map by identifying pixels or groups of pixels in the refinement of the current position line of the first estimated disparity map as being (by) a number of masked pixels or groups of pixels in a refinement basic region Shielding and refining the set of masked pixels, the refined basic regions being closest to the current position line of the first estimated disparity map in a first direction and a direction opposite to the first direction The edge coordinates of the pixel or pixel group in the refinement do not exceed the corresponding An edge coordinate of the current position line of the first estimated disparity map in a portion of the refinement basic region, and does not exceed the current position line of the first estimated disparity map in a direction opposite to the second direction; The current position line of the first estimated disparity map including the masked pixel set or the refined masked pixel set and information related to the detected one or more edge coordinates of the reference line of the first image, Update the refinement basic scratchpad.

In an embodiment, the hole filling module is configured to fill the thinned current pixel of the first estimated disparity map with the at least one neighboring pixel or pixel group of each masking pixel or pixel group. A set of shaded pixels.

In an embodiment, the refinement basic region further does not exceed the pixel or pixel in the refinement in the current direction line of the first estimated disparity map in the first direction and the direction opposite to the first direction, respectively. A group of pixels or groups of pixels with a predetermined distance. In an embodiment, the refinement basic region additionally does not exceed a position line of a predetermined distance from the current position line of the first estimated disparity map in the second direction.

In an embodiment, when the number of masked pixels or groups of pixels defined in the refinement basic region is greater than half of the total number of pixels or groups of pixels defined in the refinement basic region, the mask refinement module will be thin. The pixels or groups of pixels in the recognition are identified as being masked.

In another embodiment, the hole filling module includes a padding basic register, and the hole filling module is configured to utilize the padding basic register and the layer containing the refined masking pixel set. The current position line of the first estimated disparity map is transmitted based on a parallax of pixels or groups of pixels in the filled basic region The statistical information of the value fills the pixel or group of pixels in the refined set of masked pixels and fills the refined set of masked pixels in the current position line of the first estimated disparity map to obtain the First refining the current position line of the disparity map, wherein the filling basic area is closest to the first direction and the current position line of the first estimated disparity map in the direction opposite to the first direction The edge coordinates of the filled pixel or group of pixels, and the current position line of the first estimated disparity map is not exceeded in the direction opposite to the second direction; and the current position of the first refinement disparity map The line update fills the basic scratchpad.

In one embodiment, the padding base region is further predetermined in the first direction and the direction opposite to the first direction, and the pixel or pixel group that is filled in the current position line from the first estimated disparity map is predetermined. The pixel or pixel group of the distance. In an embodiment, the padding base region further does not exceed a position line of a predetermined distance from the current position line of the first estimated disparity map in the second direction. In one embodiment, the fill base region additionally does not exceed the position line containing the filled pixels or groups of pixels in a direction opposite the second direction.

In one embodiment, the hole filling module obtains statistical information by performing a multi-stage voting method, and the multi-stage voting method includes: obtaining a maximum number of pixels or pixel groups in the filled basic region associated with one of the series of disparity values And obtaining a maximum of the number of pixels or groups of pixels in the fill basic region of the series of disparity values associated with one of the disparity values.

In another embodiment, a computer system includes: storing one or more memories including program routines, and coupling to the one or more memories for control One or more processing units that perform the execution of the program routines. The program routines include an edge detection module, first and second parallax estimation modules, a cross-check module, a mask refinement module, and a hole filling module. The edge detection module is configured to detect edge coordinates in a first image. The first and second disparity estimation modules are respectively configured to obtain a first estimated disparity map of the first image relative to a second image and a second estimated disparity of the second image relative to the first image. Figure. The cross-checking module is configured to cross-check the first estimated disparity map with the second estimated disparity map to identify one of the masked pixel sets in the first estimated disparity map. The mask refinement module is configured to refine the masked pixel set to generate a refined masked pixel set. The hole filling module is configured to fill the refined masked pixel set in the first estimated disparity map to obtain a first refinement disparity map.

In an embodiment, the mask refinement module is configured to transmit at least one pixel or pixel group in the first estimated disparity map based on the number of masked pixels or groups of pixels in a refinement basic region. Recognizing as being masked, the shading pixel set is refined, and the refinement basic region does not exceed one of the pixels or pixel groups in the refinement in a first direction and a direction opposite to the first direction The edge coordinates of the pixel or pixel group closest to the refinement in the position line, and respectively in a second direction and in a direction opposite to the second direction, do not exceed the pixel corresponding to the refinement or The position line of the pixel group is at an edge coordinate of a portion of the refinement basic region.

In one embodiment, the hole filling module is configured to fill the refinement based on at least one respective neighboring pixel or group of pixels of each masking pixel or group of pixels. The shaded pixel collection.

In another embodiment, an imaging method includes: providing a first image and a second image; detecting edge coordinates in the first image; obtaining a first estimated parallax of the first image relative to the second image And a second estimated disparity map of the second image relative to the first image; the first estimated disparity map is cross-checked by the second estimated disparity map to identify one of the masked pixel sets in the first estimated disparity map And sizing the masked pixel set to generate a refined masked pixel set; and filling the refined masked pixel set in the first estimated disparity map to obtain a first refinement disparity map.

In an embodiment, the thinning the set of masked pixels comprises identifying the pixel or group of pixels in the thinned first disparity map as being based on the number of masked pixels or groups of pixels in a refinement basic region as Masked, the refinement basic regions are respectively in a first direction and a direction opposite to the first direction not exceeding a position in a line of pixels or a pixel group in the refinement that is closest to the refinement The edge coordinates of the pixel or the pixel group, and respectively in a second direction and in a direction opposite to the second direction does not exceed the position line corresponding to the pixel or pixel group containing the refinement in the refinement basic The edge coordinates of one part of the area.

In one embodiment, filling the refined set of masked pixels is based on at least one respective neighboring pixel or group of pixels of each masked pixel or group of pixels.

In one embodiment, the refinement basic region does not exceed the position line of the pixel or pixel group contained in the refinement in a direction opposite to the second direction. In an embodiment, the refinement basic regions are additionally in the first direction and the first The direction of the opposite direction does not exceed the pixel or group of pixels having a predetermined distance from the pixel or group of pixels in the thinning in the position line of the pixel or group of pixels in the refinement. In one embodiment, the refinement basic region additionally does not exceed a position line at a predetermined distance from the position line containing the pixel or pixel group in the refinement in the second direction.

In an embodiment, when the number of masked pixels or groups of pixels defined in the refinement basic region is greater than half of the total number of pixels or groups of pixels defined in the refinement basic region, the mask refinement module will be thin. The pixels or groups of pixels in the recognition are identified as being masked.

In one embodiment, the hole filling module fills the pixel that is filled in one of the refined masked pixels or groups of pixel groups based on a statistical information of a disparity value of pixels or groups of pixels in the filled basic region. a pixel group, the filling basic region in the first direction and the direction opposite to the first direction respectively not exceeding a pixel or a pixel closest to the filled pixel in a position line containing the filled pixel or pixel group The edge coordinates of the group.

In an embodiment, the filling base region is further spaced apart from the filled pixel or pixel group in a position line opposite to the first direction in a direction opposite to the first direction, respectively, in a direction line containing the filled pixel or pixel group. The pixel or group of pixels. In one embodiment, the fill base region additionally does not exceed a position line at a predetermined distance from the position line containing the filled pixel or pixel group in the second direction. In one embodiment, the fill base region additionally does not exceed the position line containing the filled pixels or groups of pixels in a direction opposite to the second direction.

In one embodiment, the hole filling module is configured to perform multi-stage voting To obtain statistical information, the multi-stage voting method includes: obtaining a maximum number of pixels or groups of pixels in the filled basic region associated with a series of disparity values; and obtaining a maximum number of pixels in the basic region of the series of disparity values Or a group of pixels associated with one of the disparity values.

In another embodiment, a non-transitory computer readable recording medium stores a computer program to cause a computer system to perform the imaging method described above.

The above description is a summary of the features and advantages of the present invention in order to provide a The technical features and advantages of the present invention are further described below to support the subject matter claimed in the claims of the present invention. Those having ordinary skill in the art should understand that the concept and specific embodiments disclosed may be directly modified or designed to achieve the same function as the present invention. Those having ordinary skill in the art should understand that the above-described equivalent design does not depart from the spirit of the invention and should be covered by the following claims.

The objects and advantages of the invention will be apparent from the description and appended claims.

1 shows a block diagram of an application of imaging system 20 in accordance with an embodiment of the present invention. Referring to FIG. 1, in the application of the imaging system 20, the stereo camera 10 includes two cameras that capture two original images 12 or video of a scene from different viewpoints. A stereo camera can be any image recording device that can capture two images of a scene from different viewpoints. Depending on the application, the camera can be integrated into one device or configured separately. Correction and correction block 14 performs correction of two original images 12 to remove lens distortion, and The two original images 12 are twisted to remove the mismatch between the coplanar and the nuclear lines, so that the output first image 16 and the second image 18 can be compared by a single line to a single line. Imaging system 20 utilizes first image 16 and second image 18 to generate depth map 22. The depth map 22 provides distance information of the object from the stereo camera 10 in the scene. Since the depth map 22 can be directly calculated from the disparity map, the disparity map is hereinafter referred to as the output of the imaging system 20, and the disparity map is obtained by measuring the parallax distance between the first image 16 and the second image 18. Application block 24 can thus utilize depth map 22 for, for example, three-dimensional gesture recognition, view synthesis, and stereoscopic television presentation.

The imaging system 20 can be implemented by a hardware, such as a Field Programmable Gate Array (FPGA), and implemented by an Application Specific Integrated Circuit (ASIC), or can be transmitted through a general computer system. It is implemented in software or in a combination of hardware and software. Hardware implementation can achieve higher performance than software implementation, but requires higher design costs. For instant applications, hardware implementations are often chosen to meet speed requirements.

2 and 3 respectively show block diagrams of an imaging system in accordance with an embodiment of the present invention. Referring to FIG. 2, in an embodiment, the imaging system 20 implemented by the sorting buffer hardware receives a first image and a second image, outputs a refined disparity map, and includes an edge detection module 204, The first and second parallax estimation modules 208 and 258, the cross-check module 212, the mask refinement module 216, and the hole filling module 220.

The edge detection module 204 is configured to receive at least one reference line in the first image and detect one or more edge coordinates of the reference line in the first image to generate an edge Figure 206. Corresponding to the position on the reference line of the first image, a current position line of the refined disparity map is generated.

The first disparity estimation module 208 is configured to receive a current position line of the first image and obtain a current position line of the first estimated disparity map 210 of the first image relative to the second image (as shown in the left part of the double arrow in the figure) display). Similarly, the second disparity estimation module 258 is configured to receive a current position line of the second image and obtain a current position line of the second image relative to the second estimated disparity map 260 of the first image (as shown by the double arrow in the figure) Shown to the section). The first or second disparity estimation module 208 or 258 can be implemented using a holistic approach or a regional approach. As noted above, the regional approach has a much lower complexity than the holistic approach, and thus, in order to achieve a higher speed, such as that required for immediate applications, in one embodiment of the invention described below, A regional approach is adopted.

Referring to FIG. 2 and FIG. 3, in an embodiment, the first disparity estimation module 208 includes a first census conversion module 2081, a first cost summation module 2083, and a first disparity selection module 2085; The module 258 includes a second census conversion module 2581, a second cost addition module 2583, and a first parallax selection module 2585.

a first census conversion module 2081 including a first census register (not shown) for receiving a current position line of the first image, and using a first census register and a current position line of the first image, Performing a census conversion to generate a corresponding line of the first image, and updating the first census register with the current position line of the first image, and outputting the census converted comparison line 2082 of the first image. In an embodiment, the size of the first census register is equal to the first shadow The product of the width, pixel or pixel group bit length and the height of the two lines. The height is determined by the neighborhood used by the census conversion. Similarly, the second census conversion module 2581 including the second census register is configured to receive a current position line of the second image, and perform a census conversion using the second census register and the current position line of the second image. To generate a second image comparison line, the second census register is updated with the current position line of the second image, and the census converted comparison line 2582 of the second image is output.

The first cost summation module 2083 including the first summation window register (not shown) is configured to respectively receive the comparison lines 2082 and 2582 of the first image and the second image; for a plurality of parallax distances, calculate The current matching cost of the comparison line 2082 of the first image with respect to the second image comparison line 2582; for the plurality of parallax distances, the first matching window buffer and the current matching cost are used to add the matching cost; Updating the first summing window register with the current matching cost; and for the plurality of disparity distances, outputting the summed matching cost 2084. In this embodiment, the comparison lines of the first and second images are obtained from the output 2082 of the first census conversion module 2081 and the output 2582 of the second census conversion module 2581, respectively. In one embodiment, the size of the first summing window register is equal to the product of the width of the first image, the length of the pixel or pixel group, and the height of the summing window (described in the description of the following section).

Similarly, a second cost summation module 2583 including a second sum total window register (not shown) is configured to receive the first and second image comparison lines 2082 and 2582, respectively; for a plurality of parallaxes Distance, calculating the current match of the second image comparison line 2582 with respect to the first image comparison line 2082 a cost; for the plurality of disparity distances, summing the matching cost using the second summing window register and the current matching cost; updating the second summing window register with the current matching cost; and for the plurality of disparities Distance, the output is summed to match the cost of 2584. It should be noted that in the present embodiment, the cost of the census converted image is added up. However, the invention is not limited to being implemented in this manner. In other embodiments, the unconverted images may be cost-added.

The first disparity selection module 2085 is configured to receive the summed matching cost 2084 of the plurality of disparity distances of the first cost summation module 2083; for each coordinate on the current position line of the first estimated disparity map 210, select the respective One of a plurality of parallax distances is used as the disparity value at the coordinates on the first estimated disparity map 210, and the current position line of the first estimated disparity map 210 is output. Similarly, the second disparity selection module 2585 is configured to receive, from the second cost summation module 2583, the aggregated matching cost 2584 of the plurality of disparity distances; each of the current position lines of the second estimated disparity map 260 A flag selects one of the plurality of parallax distances as a disparity value at a coordinate of the second estimated disparity map 260, and outputs a current position line of the second estimated disparity map 260.

The cross-check module 212 is configured to receive the current position line of the first estimated disparity map 210 and the current position line of the second estimated disparity map 260; and cross-check the first estimated disparity map 210 by using the current position line of the second estimated disparity map 260. a current position line to identify one of the current position lines of the first estimated disparity map 210 to mask the set of pixels 214, or to perform the reverse operation; and output a set of masked pixels in the current position line of the first estimated disparity map 210 214. The masked pixel set 214 can contain zero or more masked pixels or groups of pixels.

The occlusion refinement module 216 including the refinement basic register (not shown) is configured to receive the occlusion pixel set 214 in the current position line of the first estimated disparity map 210; using the refinement basic register and containing the occlusion The current position line of the first estimated disparity map 210 of the set of pixels 214 passes through the refined pixels or pixels on the current position line of the first estimated disparity map 210 based on the number of masked pixels or groups of pixels in the refinement basic region. The group is identified as being masked, and the masked pixel set 214 is refined; and the current position line of the first estimated disparity map 210 containing the masked pixel set 214 or the refined masked pixel set 218 and the edge map 206 The information about one or more edge coordinates of a reference line of an image is updated to refine the basic scratchpad. Refining the basic regions in a first direction and in a direction opposite to the first direction, not exceeding the edge coordinates of the pixel or pixel group closest to the refinement in the current position line of the first estimated disparity map 210, not exceeding The edge coordinates corresponding to a portion of the current position line of the first estimated disparity map 210 in the refinement basic region, and the current position line of the first estimated disparity map 210 are not exceeded in a direction opposite to the second direction. The information related to the detected one or more edge coordinates may be, but is not limited to, between each pixel or pixel group of the current position line of the first estimated disparity map 210 and the respective edge coordinates in the second direction. distance. For example, if the pixel or pixel group on the current position line of the first estimated disparity map 210 is located at the edge coordinate detected on the reference line of the first image, the distance is set to zero; otherwise, the pixel or the pixel group The distance will be the corresponding position of the current position line in the previous position line The distance of a pixel or group of pixels is incremented by one.

In an embodiment, the refinement basic region is further in the first direction and the direction opposite to the first direction, and does not exceed the pixel or pixel group in the refinement from the current position line of the first estimated disparity map. A pixel or a group of pixels of a predetermined distance r _r , r _l . In another embodiment, the refinement base region additionally does not exceed a position line having a predetermined distance r _u from the current position line of the first estimated disparity map in the second direction. In this embodiment, refining the size of the basic register equal to the width of the first image, the length of the bit equal to the predetermined distance r _u , and storing the detected one or more edge coordinates of the reference line of the first image The product of the length of the bit length required for the information, the bit length required to store the information is equal to log ₂ (r _u )+1 in one embodiment.

The hole filling module 220 is configured to receive the refined masked pixel set 218 in the current position line of the first estimated disparity map 210, based at least on each neighboring pixel or pixel group of each masking pixel or pixel group. The current position line of the first estimated disparity map 210 is filled with the refined masked pixel set 218 to generate a current position line of the first refinement disparity map. In one embodiment, the hole filling module 220 uses the disparity pixel of the non-masked pixel or pixel group closest to the filled pixel or pixel group in the current position line of the first estimated disparity map 210 to the filled pixel. Or pixel groups are filled. In another embodiment, a hole filling module 220 including a padding basic register (not shown) is used to utilize the padding basic register and the first estimate containing the refined masked pixel set 218. The current position line of the disparity map 210 is filled with an image that is filled in one of the thinned masked pixel sets 218 by statistical information based on a disparity value of pixels or groups of pixels in the filled basic region. Filling the refined masked pixel set 218 with the pixel or the pixel group in the current position line of the first estimated disparity map 210 to obtain the current position line of the first refinement disparity map, respectively One direction and the direction opposite to the first direction does not exceed an edge coordinate of the current position line of the first estimated disparity map 210 that is closest to the filled pixel or group of pixels, and is opposite to the second direction The current position line of the first estimated disparity map 210 is not exceeded in the direction.

In an embodiment, the padding base region further does not exceed the pixel or pixel group in the refinement from the current position line of the first estimated disparity map in the first direction and the direction opposite to the first direction, respectively. A pixel or group of pixels having a predetermined distance s _r , s _l . In addition, in an embodiment, the padding base region additionally does not exceed a position line having a predetermined distance s _u from a current position line of the first estimated disparity map in the second direction. In one embodiment, the predetermined distance s _u is equal to one position line, and in this embodiment, the size of the padding basic register is the width of the first image and is used to store the pixels or groups of pixels in the previous position line. The length of the bit required for the maximum disparity value of the difference, the bit length 1 required to mark the pixel or group of pixels of the previous position line as the edge pixel or the pixel group, and the first estimated disparity map 210 The pixel or group of pixels on the current position line is indicated as the product of the sum of the bit lengths 1 of the masked pixels or groups of pixels.

In another embodiment, imaging system 20 is implemented in software via a general computer system. The general computer system includes one or more memories (not shown) for storing information including a program routine, and is coupled to the one or more memories for controlling execution of a program routine including the software module. One or more Unit (not shown). Referring to FIG. 2, in an embodiment, the routine includes an edge detection module 204, first and second parallax estimation modules 208 and 258, a cross-check module 212, a mask refinement module 216, and a hole. The module 220 is filled.

The edge detection module 204 is configured to detect edge coordinates in the first image. The first disparity estimation module 208 is configured to obtain a first estimated disparity map 210 of the first image relative to a second image (shown in the leftward portion of the double arrow in the figure). Similarly, the second disparity estimation module 258 is configured to obtain a second estimated disparity map 260 of the second image relative to the first image (shown in the rightward portion of the double arrow in the figure). The cross-check module 212 is configured to cross-check the first estimated disparity map 210 with the second estimated disparity map 260 to identify one of the masked pixel sets 214 in the first estimated disparity map 210. The mask refinement module 216 is configured to identify at least one pixel or group of pixels in the refinement of the first estimated disparity map 210 as being masked by masking pixels or groups of pixels in a refinement basic region, The masked pixel set 214 is refined to obtain a refined shaded pixel set 218. In one embodiment, the refinement basic regions are respectively in a direction opposite to the first direction in a direction opposite to the first direction, and are closest to the refinement in a position line of the pixel or the pixel group in the refinement. Edge coordinates of the pixels or groups of pixels, and in a direction opposite to the second direction in a second direction and in a direction opposite to the second direction, respectively, corresponding to the position line corresponding to the pixel or group of pixels in the refinement The edge coordinates of the portion between the edge coordinates. The hole filling module 220 is configured to fill the refined masked pixel set 218 based on at least one respective neighboring pixel or group of pixels of each masking pixel or group of pixels to obtain a first refined disparity map.

The above-described embodiments are merely examples of the imaging system 20 of the present invention and alternative implementations are possible. For example, the order of operations in the modules can vary, and all modules are not required, as will become more apparent from the following description. The detailed operation of the module can be obtained directly from the following description of the imaging method of one embodiment of the present invention.

4 is a flow chart showing an image forming method of an embodiment of the present invention. Figure 5 is a flow chart showing the parallax estimation step implemented by the regional method in accordance with one embodiment of the present invention. Figure 6 shows a schematic diagram of the first and second images for stereo matching. Figure 7 is a schematic illustration of an edge map of one embodiment of the present invention. Figure 8 is a diagram showing the census conversion step of one embodiment of the present invention. Figure 9 is a diagram showing the cost-adding step of one embodiment of the present invention. Figure 10 is a schematic illustration of the cross-checking step of one embodiment of the present invention. Figures 11 and 12 show schematic views of the mask refinement steps of various embodiments of the present invention. Figure 13 is a diagram showing the results of the mask refinement step of one embodiment of the present invention. Figure 14 is a diagram showing the steps of filling a hole in an embodiment of the present invention.

In one embodiment, the imaging method is a computer implemented method and can be stored on a non-transitory computer readable recording medium to cause the computer system to perform an imaging method. Referring to FIG. 4, in an embodiment, the imaging method 400 includes the following steps: providing a first image and a second image (S402); detecting edge coordinates in the first image (S404); obtaining the first image Correlating a first estimated disparity map with respect to one of the second images and a second estimated disparity map of the second image with respect to one of the first images (S406); using the second estimated disparity map to cross-check the first estimated disparity map To identify a masked pixel set (S408); The set of masked pixels is refined (S410); and the refined set of masked pixels is filled (S412).

Referring to FIG. 4 and FIG. 6, in step S402, first and second images 60 and 65 are provided. As described above and as shown in FIG. 6, the first image 60 and the second image 65 have been twisted so that the horizontal lines aligned in the first image 60 and the second image 65 can be pixel or pixel pair pixels or pixels. Match of groups. In this embodiment, the first image 60 is a left view image and the second image 65 is a right view image. In another embodiment, the first image may be a right view image and the second image may be a left view image.

Referring to FIG. 4 and FIG. 7 again, in step S404, edge coordinates in the first image 60 are detected to obtain an edge map 70. In edge map 70, edge coordinates 702 corresponding to boundary pixels or groups of pixels in the first image 60 are labeled. Clear object boundaries can be obtained using appropriate edge detection techniques in the art.

Referring to FIG. 4, in step S406, a first estimated disparity map of the first image 60 relative to the second image 65 is obtained. The first estimated disparity map can be obtained using a holistic or regional approach. As described above, the holistic approach formulates the stereo matching problem as an energy function, which can be represented by equation (1). The energy function includes a graduality term E _smooth (d) that can cause the first estimated disparity map to be flat everywhere and thus blur the object boundaries and affect the quality of the disparity map.

In one embodiment, the disparity estimation step S406 is performed in a regional method including the following steps. Reference will be made hereinafter to FIGS. 5, 6, and 9. In FIG. 9, grid 90 represents a portion of a pixel or group of pixels of first image 60, and grids 952, 954, 956 represent portions of pixels or groups of pixels in second image 65. In step S504, the matching cost of the reference window (the slash indication portion in the grid 90) and the plurality of candidate matching windows (the slash indication portions in the grids 952, 954, and 956) are calculated and added, and the reference window and the first Reference pixels or groups of pixels having (x, y) coordinates in the image 60 overlap, and the plurality of candidate matching windows and the second image 65 have, for example, (x, y), (x-1, y), (x-2) , y)...(xd _max , y) The respective candidate matching pixels or groups of pixels of the coordinates overlap. Each candidate matching pixel or group of pixels having (x, y), (x-1, y), (x-2, y) ... or (xd _max , y) coordinates has relative to the reference pixel (x, y) A different parallax distance d = 0, d = -1, d = - 2 ... or d = - d _max . Next, in step S506, according to the total matching cost, the parallax distance d corresponding to one of the candidate matching pixels or the pixel group is selected as one of the coordinates of the reference pixel (x, y) on the first estimated disparity map. Difference. In one embodiment, the parallax distance corresponding to the candidate matching pixel or group of pixels having the smallest total matching cost is selected. Next, in step S508, if the processed reference pixel located at the coordinate (x, y) is the last reference pixel or pixel group in the first image 60, the first estimated disparity map is completed; otherwise, the steps are repeated. S504 and S506. The cost summing step S504 reduces the influence of the noise in the first image 602 on the accuracy of the disparity map, but blurs the boundary of the object and affects the quality of the disparity map.

In different embodiments, the total matching cost C in step S504 can be obtained by using Sum of Absolute Difference (SAD) and Sum of Difference (Sum of Absolute Difference, SAD, respectively). Squared Differences (SSD), or normalized cross-correlation (Normalized Cross Correlation, NCC) calculations.

C _SAD =Σ _{x , y} | I ₁ ( x , y )- I ₂ ( x - d , y )| (2)

C _SSD =Σ _{x , y} ( I ₁ ( x , y )- I ₂ ( x - d , y )) ² (3)

Wherein, x and y are coordinates of a reference pixel, d is a parallax distance, and I ₁ and I ₂ represent pixel intensities of the first and second images 60 and 65, respectively.

Referring to FIG. 4, FIG. 5 and FIG. 8 , in another embodiment, step S404 further includes a census conversion step S502 of performing a census conversion on the first image 60. For example, in FIG. 8, grid 80 displays the pixel intensity of a portion of the pixels or groups of pixels in the first image 60. Performing a census conversion on a pixel located in the first image 602 at coordinates (x, y) means performing a comparison of the pixel intensities of the pixels or groups of pixels and their neighboring pixels or groups of pixels to obtain, for example, a grid 80' The bit vector at the coordinates (x, y). In one embodiment, each pixel or group of pixels is compared to its immediately adjacent four pixels or groups of pixels to obtain a bit vector of length 4. In one embodiment, the bit vector length is at least one. In one embodiment, the bit vector length may be 1, 2, 3, or any higher number depending on the number of neighboring pixels or groups of pixels being compared. A pixel or group of pixels that have undergone a census conversion has a significantly shorter bit length than a pixel or group of pixels represented by the pixel intensity described above. Similarly, the second image 65 is also subjected to a census conversion. Grid 85 displays the pixel intensities of a portion of the pixels or groups of pixels in the second image 85, and grid 85' displays the pixels or groups of pixels that have undergone a general scan. For the image converted by the census, the total matching cost C of step S504 can be calculated by Hamming Distance (HD) as shown in equation (5): C _HD = Σ _{x , y} HD (bit vector ₁ ( x , y ), bit vector ₂ ( x - d , y )) (5)

Wherein, x and y are coordinates of a reference pixel, d is a parallax distance, and BitVec ₁ and BitVec ₂ respectively represent bit vectors of pixels or groups of pixels of the first and second images 60 and 65 that have been census-converted.

Next, referring to FIG. 4 and FIG. 10, in step S408, the first estimated disparity map 100 is cross-checked with the second estimated disparity map to identify a masked pixel set 102. In general, the object should have the same disparity value in the first estimated disparity map and the second estimated disparity map. Depending on the location of the viewpoint, portions of the object may be obscured in one of the images and only visible in the other image. The cross check step S408 detects the masked portion of the object and other unmatched portions. In an embodiment, the cross-checking step S408 is performed based on the constraint conditions shown below:

Wherein D and D' represent the first and second estimated disparity maps, respectively, and λ is a predetermined cross check threshold value. Pixels or groups of pixels on the first estimated disparity map 100 that do not satisfy the constraint (6) will be included in the masked pixel set 102. In addition, as shown in FIG. 10, since the quality of the estimated disparity map 100 can be affected by the gradation term in the holistic method, it can be affected by the cost summation and/or the census conversion in the regional method, so the object boundary is in the first An estimated parallax map 100 becomes blurred.

Next, referring to FIG. 4, FIG. 11, FIG. 12 and FIG. 13, in step S410, the first estimated disparity map 100 shown in FIG. 11 or FIG. 12 is obtained by using the edge coordinates 702 obtained from the edge detecting step S404. The shaded pixel set 102' or 102 in 'or 100 is refined to produce a refined shaded pixel set 102" on the first estimated disparity map 100", as shown in FIG. In Figures 11 and 12, the grid shown at the bottom of the figure is an enlarged view of the first estimated disparity map 100' or 100 at different portions of the shading refinement. In one embodiment, the masked pixel set 102' or 102 is in the refinement of the first estimated disparity map 100' or 100 by the number of masked pixels 1008 in the refined basic region 1010 (indicated by a thick line). Pixel 1006 is identified as being shaded and refined. Based on the cross-checking step S408, the blank pixels 1006 and the blank pixels 1007 in the basic region 1010 are refined into unmasked pixels or groups of pixels.

Referring to the enlarged view shown on the bottom left side of FIG. 11, in an embodiment, in the first direction (for example, the X direction), the refinement of the basic region 1010 does not exceed the edge of the pixel 1006 closest to the refinement. The coordinates (not shown) and additionally do not exceed the pixel or group of pixels having a predetermined distance r _r from the pixel 1006 in the refinement in the position line of the pixel 1006 in the refinement. In a direction opposite to the first direction, the refinement basic region 1010 does not exceed the edge coordinates 1004 of the pixel 1006 closest to the refinement and additionally does not exceed the position line of the pixel 1006 included in the refinement. The pixel 1006 has a pixel or a pixel group of a predetermined distance r _l . In the second direction (eg, the Y direction), the refinement basic region 1010 does not exceed the edge coordinates 1002 corresponding to the portion of the position line of the pixel 1006 included in the refinement in the refinement basic region 1010, and additionally does not exceed the distance The position line of the pixel 1006 contained in the refinement has a position line of a predetermined distance r _u . In a direction opposite to the second direction, the refinement basic region 1010 does not exceed an edge coordinate (not shown) corresponding to a portion of the position line of the pixel 1006 included in the refinement in the refinement basic region 1010, and additionally position of the line of pixels or groups of pixels in the thinning without departing from the contained position of the line with the predetermined distance r _d. In one embodiment, the predetermined distances r _r , r _l , r _{u ,} and r _{d are} equal to at least one position line. In the example shown on the bottom left side of FIG. 11, the refinement basic region 1010 is bounded by the edge coordinates 1004 during the edge coordinate search in the opposite direction to the first direction and the edge coordinates 1002 in the second direction, and The predetermined distances r _r and r _d are at most reached in the first direction and in the opposite direction to the second direction. Compared to the example shown on the bottom left side of FIG. 11, the refined basic region 1010 in the example shown on the bottom right side of FIG. 11 is defined by the edge coordinates 1002 in the opposite direction to the second direction, and is in the second The predetermined distance r _{u is} reached at most in the direction.

Referring to the enlarged view on the bottom left side of FIG. 12, in the present embodiment, the direction opposite to the second direction (for example, the Y direction) is not searched. Therefore, in the first direction (for example, the X direction), the refinement basic region 1010 does not exceed the edge coordinates 1004 of the pixel 1006 closest to the refinement and additionally does not exceed the position line of the pixel 1006 contained in the refinement. A pixel or group of pixels having a predetermined distance r _r from the pixel 1006 in the refinement. In a direction opposite to the first direction, the refinement base region 1010 does not exceed the edge coordinates (not shown) of the pixel 1006 closest to the refinement and additionally does not exceed the position line of the pixel 1006 contained in the refinement. A pixel or group of pixels with a predetermined distance r _l . In the second direction, the refinement basic region 1010 does not exceed the edge coordinate 1002 corresponding to the portion of the position line of the pixel 1006 included in the refinement in the refinement basic region 1010, and additionally does not exceed the distance contained in the refinement. The position line of the pixel 1006 has a position line of a predetermined distance r _u . In a direction opposite to the second direction, the refinement basic region 1010 does not exceed the position line of the pixel 1006 contained in the refinement. In the present example, the refinement basic region 1010 is bounded by the edge coordinates 1004 during the edge coordinate search in the first direction and the edge coordinates 1002 during the second direction, and reaches a predetermined distance in the opposite direction to the first direction. r _l . Compared to the example shown on the bottom left side of FIG. 12, the refined basic region 1010 in the example shown on the bottom right side of FIG. 12 is defined by the first direction and the edge coordinates of the search period in the opposite direction to the first direction. 1004 is limited.

The phrase "not exceeding" as used in the above description means the span of the area up to the specified boundary or within the specified boundary.

Referring to FIG. 4 and FIG. 11 , in an embodiment, in step S410 , the number of masked pixels 1008 defined in the thinned basic region 1010 is greater than the total number of pixels 1006 , 1007 , and 1008 defined in the refined basic region 1010 . At a predetermined percentage, the pixel 1006 in the refinement is identified as being masked. In one embodiment, the predetermined percentage is at least 0.1 to 0.9. to In the embodiment shown in FIG. 11, when it is determined whether the pixel 1006 in the refinement is marked as being masked, the masked pixel or pixel group identified during the mask refinement step S410 is obtained from the cross check step S408. The masked pixels or groups of pixels are counted together. As shown in FIG. 11, the first estimated disparity map 100' has been subjected to the mask refinement processing in the portion above and to the left of the pixel 1006 in the refinement and contains the mask from both the cross check step S408 and the mask refinement step S410. Pixel or group of pixels. In the different embodiments as shown in FIG. 12, only the masked pixels 1008 obtained from the cross-checking step S408 are counted when deciding whether to mark the pixels 1006 in the refinement as being masked.

Next, referring to FIG. 4 and FIG. 14 , in an embodiment, in step S412 , the thinned masked pixel set 102 is filled based on at least one respective neighboring pixel or pixel group of each masked pixel or group of pixels. To obtain a first refinement disparity map. In one embodiment, assuming that the padding process is performed from left to right, top to bottom, a position line containing the filled pixels 1012 is used (eg, in FIG. 14 The disparity pixel of the non-masked pixel or group of pixels closest to the filled pixel 1012 in the second column of the illustrated grid fills the filled pixel 1012. For example, the closest non-masked pixel or group of pixels may Is a pixel or group of pixels having a disparity value 36 to the left of the filled pixel 1012.

In another embodiment, the filled pixels 1012 in the refined shaded pixel set 102" are populated based on statistical information of the disparity values of the pixels or groups of pixels in the fill base region 1014 (shown in bold lines). In one embodiment, the refinement basic region 1014 does not exceed the pixel 1012 closest to the fill in the first direction (eg, the X direction) and the direction opposite to the first direction in the position line containing the filled pixel 1012, respectively. Edge coordinates 1016. As shown in FIG. 14, in an embodiment, the filling base regions 1014 are additionally received in the first direction and in the opposite direction to the first direction, respectively, beyond the position of the line containing the filled pixels 1012. The filled pixels 1012 have pixels or groups of pixels of a predetermined distance s _r , s _l . In one embodiment, the predetermined distances r _r , r _l , r _{u ,} and r _{d are} equal to at least one position line. In addition, the filled basic area 1014 Also in the second direction (e.g., Y direction) without departing from the subject containing the pixel position of the filling line position of the line 1012 with a predetermined distance s _u of the embodiment, the base region 1014 to fill a further embodiment in the second direction opposite to the Upwardly by the pixel position of the line does not exceed the filling contains 1012. In one embodiment, _u is equal to at least one predetermined position of the line from S. In the example shown at the top of the drawing, the filling of the first base region 1014 The direction and the edge coordinates 1016 during the edge coordinate search in the opposite direction to the first direction are bounded, and at most a predetermined distance s _u in the second direction and at most up to the filled direction The position line of the pixel 1012. Compared to the example shown at the top of Fig. 14, the filled basic region 1014 shown at the bottom of the figure reaches a predetermined distance s _r at most in the first direction.

Continuing with the above description, in one embodiment, statistical information is obtained by performing a multi-stage voting method. The multi-stage voting method includes: obtaining a maximum number of pixels or groups of pixels in the filled basic region 1014 associated with a series of disparity values; And obtaining a maximum number of pixels or groups of pixels in the basic region 1014 of the series of disparity values associated with one of the disparity values. In Figure 14 In the illustrated example, the maximum number of pixels or groups of pixels in the fill base region 1014 belong to a series of disparity values of 45-49, and within the range of 45-49, the disparity value 47 has the highest number of votes. Therefore, the filled pixel 1012 is filled with the disparity value 47.

In the foregoing embodiment, by using the edge coordinates of the first image as a guide to identify pixels or groups of pixels that must be enhanced and identifying at least one pixel that can serve as a reference for repairing each pixel or group of pixels that must be enhanced Or the pixel group, the mask refinement step S410 and the hole filling step S412 improve the quality of the disparity map. Therefore, the mask refinement step S410 and the hole filling step S412 can improve the quality of the disparity map without increasing the complexity of the disparity estimation step S404 in which a large amount of calculation is required. The mask refinement step S410 and the hole filling step S412 may even help reduce the complexity of the disparity estimation step S404 by allowing a more compact representation of the pixels or groups of pixels, such as in an embodiment including the census conversion step S502.

It should be noted that the embodiments of the above imaging method are merely examples of the present invention. Alternative equivalent embodiments are possible. For example, the census conversion step S502 can be performed on a region-by-region basis and included in the processing loop shown in FIG. 5; or as in the above-described pipelined hardware embodiment, the first and second estimated parallax can be generated in rows The graph, the masked pixel set, the refined masked pixel set, and the first refined disparity map.

In summary, the present invention provides an imaging system and method that generate a disparity map of a first image and a second image. Using the edge information of the first image as a guide to identify the area on the disparity map that needs to be enhanced, so as not to increase the complexity of the part that requires a large amount of calculation, such as disparity estimation, Improve the quality of the parallax map. Using edge information to refine the disparity map can even reduce the complexity of disparity estimation. By improving the quality and control complexity of the parallax map, the present invention is applicable to applications such as three-dimensional gesture recognition, viewpoint synthesis, and stereoscopic television.

The technical content and the technical details of the present invention have been disclosed above, but those skilled in the art can still make various substitutions and modifications without departing from the spirit and scope of the present invention. For example, many of the above described programs can be implemented in different ways, with alternatives to other programs, or a combination thereof.

In addition, the scope of the present invention is not limited to the specific embodiments of the procedures, machines, products, compositions, means, methods and steps described in the specification. Those having ordinary skill in the art should be able to directly appreciate the presently existing or later developed ones having the functions substantially equivalent to the embodiments of the present specification or substantially achieving the same results as the embodiments of the present specification. The program, machine, product, composition, means, method or procedure used in the present invention. Therefore, the above-mentioned procedures, machines, finished products, compositions, means, methods or steps are to be covered by the following claims.

10‧‧‧ Stereo camera

12‧‧‧ original image

14‧‧‧Correction and twist squares

16‧‧‧ first image

18‧‧‧Second image

20‧‧‧ imaging system

22‧‧‧Depth map

24‧‧‧Application block

60‧‧‧ first image

65‧‧‧Second image

70‧‧‧Edge map

80‧‧‧ grid

80'‧‧‧ grid

85‧‧‧Grid

85'‧‧‧Grid

90‧‧‧Grid

100‧‧‧First estimated disparity map

100'‧‧‧First Estimated Disparity Map

100"‧‧‧First Estimated Disparity Map

102‧‧‧Shaded pixel collection

102'‧‧‧Shaded pixel collection

102"‧‧‧shaded pixel collection

204‧‧‧Edge Detection Module

206‧‧‧Edge map

208‧‧‧First Parallax Estimation Module

210‧‧‧First estimated disparity map

212‧‧‧ cross check module

214‧‧‧shaded pixel collection

216‧‧‧shadow refinement module

218‧‧‧Refined fine pixel collection

220‧‧‧ hole filling module

258‧‧‧Second Parallax Estimation Module

260‧‧‧Second estimated disparity map

400‧‧‧ imaging methods

702‧‧‧Edge coordinates

952‧‧‧Grid

954‧‧‧Grid

956‧‧‧Grid

1002‧‧‧Edge coordinates

1004‧‧‧Edge coordinates

1006‧‧‧ blank pixels

1007‧‧‧ blank pixels

1008‧‧‧shaded pixels

1010‧‧‧Refine the basic area

1012‧‧‧filled pixels

1014‧‧‧filled basic area

1016‧‧‧Edge coordinates

2081‧‧‧First census conversion module

2082‧‧‧ alignment line

2083‧‧‧First Cost Plus Module

2084‧‧‧Additional matching cost

2085‧‧‧First parallax selection module

2581‧‧‧Second census conversion module

2582‧‧‧ alignment line

2583‧‧‧Second cost plus module

2584‧‧‧Additional matching cost

2585‧‧‧Second parallax selection module

r _l ‧‧‧ predetermined distance from the pixel or group of pixels in the refinement in the current position line of the first estimated disparity map

r _r ‧‧‧ predetermined distance from the pixel or group of pixels in the refinement in the current position line of the first estimated disparity map

r _u ‧‧‧ predetermined distance from the current position line of the first estimated disparity map

r _d ‧‧‧ predetermined distance from the pixel or group of pixels in the refinement in the current position line of the first estimated disparity map

s _r ‧‧‧ a predetermined distance from the pixel or group of pixels in the refinement in the current position line of the first estimated disparity map in the first direction

s _l ‧‧‧ a predetermined distance from the pixel or group of pixels in the refinement from the current position line of the first estimated disparity map in a direction opposite to the first direction

s _u ‧‧‧ a predetermined distance from the current position line of the first estimated disparity map in the second direction

1 is a block diagram showing an application of an imaging system according to an embodiment of the present invention; FIG. 2 is a block diagram showing an imaging system according to an embodiment of the present invention; and FIG. 3 is a block diagram showing an imaging system according to another embodiment of the present invention. 4 is a flow chart showing an imaging method according to still another embodiment of the present invention; and FIG. 5 is a flow chart showing a parallax estimation step implemented by a regional method according to an embodiment of the present invention; 6 is a schematic diagram showing first and second images for stereo matching; FIG. 7 is a schematic diagram showing an edge diagram of an embodiment of the present invention; FIG. 8 is a schematic diagram showing a census conversion step according to an embodiment of the present invention; Schematic diagram of a cost-adding step of an embodiment of the present invention; and FIG. 10 is a schematic diagram showing a cross-checking step of an embodiment of the present invention; FIGS. 11 and 12 are diagrams showing a mask refining step of different embodiments of the present invention; Figure 13 is a view showing the result of the mask refining step of one embodiment of the present invention; and Figure 14 is a view showing the step of filling the hole according to an embodiment of the present invention.

204‧‧‧Edge Detection Module

206‧‧‧Edge map

208‧‧‧First Parallax Estimation Module

210‧‧‧First estimated disparity map

212‧‧‧ cross check module

214‧‧‧shaded pixel collection

216‧‧‧shadow refinement module

218‧‧‧Refined fine pixel collection

220‧‧‧ hole filling module

258‧‧‧Second Parallax Estimation Module

260‧‧‧Second estimated disparity map

Claims (26)

An imaging system includes: an edge detection module for detecting one or more edge coordinates of at least one reference line in a first image; and first and second parallax estimation modules respectively for obtaining the first a first estimated disparity map of the image relative to a second image and a current position line of the second estimated disparity map of the second image relative to the first image; a cross check module for utilizing the second The current position line of the estimated disparity map cross-checks the current position line of the first estimated disparity map to identify one of the current position lines of the first estimated disparity map, and a masking pixel set; And the thinning filling module is configured to fill the current position line of the first estimated disparity map with the refinement The set of pixels is masked to obtain a current position line of one of the first refinement disparity maps. The imaging system of claim 1, wherein the mask refinement module comprises: a refinement basic register for utilizing the refinement basic register and the first set including the masked pixel set The current position line of the estimated disparity map is identified as a pixel or a group of pixels in the refinement of the current estimated position line of the first estimated disparity map based on the number of masked pixels or groups of pixels in a refinement basic region Masked, the shaded pixel set is refined, the refinement base The region does not exceed the edge coordinate of the pixel or pixel group in the refinement in the first direction and the direction opposite to the first direction, and does not exceed the current position line of the first estimated disparity map, and does not exceed Corresponding to an edge coordinate of the current position line of the first estimated disparity map in a portion of the refinement basic region, and not exceeding the current position line of the first estimated disparity map in a direction opposite to the second direction And the current position line of the first estimated disparity map including the masked pixel set or the refined masked pixel set and the detected one or more edge coordinates of the reference line of the first image Information to update the refinement basic scratchpad. The imaging system of claim 1, wherein the hole filling module is configured to map the first estimated disparity map based on at least one respective neighboring pixel or pixel group of each masking pixel or group of pixels. The current position line is filled with the refined masked pixel set. The imaging system of claim 1, wherein the refinement basic region additionally does not exceed a position line of a predetermined distance from the current position line of the first estimated disparity map in the second direction. The image forming system of claim 1, wherein the hole filling module comprises a filling basic register, and the hole filling module is configured to utilize the filling basic register and include the fine The current position line of the first estimated disparity map of the masked pixel set is filled by statistical information based on a disparity value of a pixel or a pixel group in the filled basic region Filling the refined set of masked pixels in the current position line of the first estimated disparity map to obtain the first refinement disparity map, wherein one of the refined masked pixel sets is filled with pixels or groups of pixels The current position line, the filled basic area is in the first direction and the direction opposite to the first direction, and the pixel that is closest to the filled pixel or pixel in the current position line of the first estimated disparity map An edge coordinate of the group, and the current position line of the first estimated disparity map is not exceeded in the direction opposite to the second direction; and the padding is updated with the current position line of the first refinement disparity map Save. The imaging system of claim 5, wherein the filling base region further does not exceed a position line of a predetermined distance from the current position line of the first estimated disparity map in the second direction. The imaging system of claim 1, wherein the first disparity estimation module comprises: a first cost summation module, comprising a first summation window register, configured to: obtain the first a comparison line of an image and the second image, for a plurality of parallax distances, calculating a current matching cost of the comparison line of the first image with respect to the comparison line of the second image, for the plurality of parallax distances Using the first summing window register and the current matching cost, summing the matching cost, and updating the first summing window register with the current matching cost; a first parallax selection module, configured to select one of the parallax distances for each pixel or group of pixels on a current position line of a first estimated disparity map as the pixel of the first estimated disparity map or One of the pixel groups has a disparity value. The imaging system of claim 7, wherein the first parallax estimation module further comprises: a first census conversion module, comprising a first census register for utilizing the first census And the current position line of the first image, performing a census conversion to generate the comparison line of the first image, and updating the first census register with the current position line of the first image, where Each census-converted pixel or group of pixels in the alignment line of the first image is represented by a bit vector having a length of at least one. A computer system comprising: storing one or more memories containing program routines, comprising: an edge detection module for detecting edge coordinates in a first image; first and second parallax estimation modes The first and second disparity estimation modules are configured to obtain a first estimated disparity map of the first image relative to a second image and a second estimate of the second image relative to the first image. a disparity map; a cross-checking module, configured to cross-check the first estimated disparity map by using the second estimated disparity map to identify one of the masked pixel sets in the first estimated disparity map; and a mask refinement module Used to fine-tune the masked pixel set Forming a refined set of masked pixels; and a hole filling module for filling the refined masked pixel set in the first estimated disparity map to obtain a first refinement disparity map; One or more processing units coupled to the one or more memories for controlling execution of the program routines. The computer system of claim 9, wherein the occlusion refinement module is configured to transmit at least one of the first estimated disparity maps based on a number of masked pixels or groups of pixels in a refinement basic region The pixels or groups of pixels in the refinement are identified as being masked, and the set of masked pixels is refined, and the refinement basic regions are respectively not included in a first direction and a direction opposite to the first direction. The edge of one of the pixel or pixel group in the position line closest to the pixel or group of pixels in the refinement, and respectively in a second direction and in a direction opposite to the second direction does not exceed The edge line of the portion of the refinement base region of the location line containing the pixel or group of pixels in the refinement. The computer system of claim 9, wherein the hole filling module is configured to fill the refined shadow pixel based on at least one adjacent pixel or pixel group of each masking pixel or pixel group. set. The computer system of claim 10, wherein the refinement basic region does not exceed the position line of the pixel or pixel group in the refinement in a direction opposite to the second direction. The computer system of claim 10, wherein the refinement basic region additionally does not exceed the distance in the second direction The position line of the pixel or group of pixels has a position line of a predetermined distance. The computer system of claim 9, wherein the hole filling module fills one of the refined masked pixel sets based on a statistical information of a disparity value of pixels or groups of pixels in the filling basic region. a padded pixel or group of pixels, the padding base region being in the first direction and in the direction opposite to the first direction, respectively, not closest to the position line containing the padded pixel or pixel group The edge coordinates of the filled pixel or group of pixels. The computer system of claim 14, wherein the filling base region further does not exceed a position line at a predetermined distance from the position line containing the filled pixel or pixel group in the second direction. . The computer system of claim 9, wherein the first disparity estimation module is configured to obtain a disparity value on the first estimated disparity map by calculating and summing a reference window and a matching cost of the plurality of candidate matching windows, wherein the reference window overlaps with one of the reference pixels or the pixel groups in the first image, and each of the plurality of candidate matching windows matches one of the candidate candidates in the second image Pixel or group of pixels overlapping, wherein each candidate matching pixel or group of pixels has a different parallax distance relative to the reference pixel or group of pixels; and selecting the disparity distance corresponding to one of the candidate matching pixels or groups of pixels as The reference pixel or pixel on the first estimated disparity map A disparity value at the coordinates of the group. The computer system of claim 16, wherein the disparity estimation module is further configured to perform a census conversion on the first image, wherein each of the comparison lines of the first image is converted into a census A pixel or group of pixels is represented by a bit vector having a length of at least one. An imaging method includes: providing a first image and a second image; detecting edge coordinates in the first image; obtaining a first estimated disparity map and a second image of the first image relative to the second image Second estimating the disparity map with respect to one of the first images; using the second estimated disparity map to cross-check the first estimated disparity map to identify one of the masked pixel sets in the first estimated disparity map; Performing refinement to generate a refined set of masked pixels; and filling the refined set of masked pixels in the first estimated disparity map to obtain a first refinement disparity map. The imaging method of claim 18, wherein the refining the masked pixel set comprises one of the first estimated disparity maps based on a number of masked pixels or groups of pixels in a refinement basic region The pixel or pixel group in the recognition is identified as being masked, and the refinement basic region does not exceed a position line of the pixel or the pixel group in the refinement in a first direction and a direction opposite to the first direction The edge coordinates closest to the pixel or group of pixels in the refinement, and respectively in the first The edge coordinates corresponding to one of the position lines of the pixel or pixel group in the refinement corresponding to the pixel or pixel group in the refinement are not exceeded in the direction of the second direction and the direction opposite to the second direction. The imaging method of claim 18, wherein filling the refined masked pixel set is based on at least one respective neighboring pixel or group of pixels of each masked pixel or group of pixels. The image forming method according to claim 19, wherein the refinement basic region does not exceed the position line of the pixel or the pixel group in the refinement in a direction opposite to the second direction. The imaging method of claim 19, wherein the refinement basic region additionally has a predetermined distance from the position line containing the pixel or pixel group in the refinement in the second direction. A position line. The imaging method of claim 18, wherein in the step of filling the refined masked pixel set, the pixel or pixel group in which one of the refined masked pixel sets is filled is based on Filling a statistical information of a disparity value of a pixel or a group of pixels in the padding region, the basic padding region not exceeding the pixel containing the padding in the first direction and the direction opposite to the first direction The edge coordinate of the pixel or pixel group closest to the filled position in one of the pixel groups. The imaging method of claim 23, wherein the filling base region further does not exceed a position of the predetermined distance from the position line containing the filled pixel or pixel group in the second direction. line. The imaging method according to claim 18, wherein in the step of obtaining a first estimated disparity map, one of the disparity values on the first estimated disparity map is obtained by the following steps: calculating and summing one Referring to the matching cost of the reference window and the plurality of candidate matching windows, the reference window overlaps with one of the reference pixels or the pixel group in the first image, and each of the plurality of candidate matching windows and one of the second images Other candidate matching pixels or groups of pixels overlapping, wherein each candidate matching pixel or group of pixels has a different parallax distance relative to the reference pixel or group of pixels; and selecting the one corresponding to one of the candidate matching pixels or groups of pixels The disparity distance is a disparity value at a coordinate of the reference pixel or the pixel group on the first estimated disparity map. The imaging method of claim 18, wherein the step of obtaining a first estimated disparity map further comprises performing a census conversion on the first image, wherein each of the first images is census converted pixels or The pixel group is represented by a bit vector having a length of at least 1.