CN106780474B - Kinect-based real-time depth map and color map registration and optimization method - Google Patents
Kinect-based real-time depth map and color map registration and optimization method Download PDFInfo
- Publication number
- CN106780474B CN106780474B CN201611232116.3A CN201611232116A CN106780474B CN 106780474 B CN106780474 B CN 106780474B CN 201611232116 A CN201611232116 A CN 201611232116A CN 106780474 B CN106780474 B CN 106780474B
- Authority
- CN
- China
- Prior art keywords
- index
- depth
- point
- scanning
- color
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000005457 optimization Methods 0.000 title claims abstract description 8
- 238000013507 mapping Methods 0.000 claims abstract description 33
- 230000002159 abnormal effect Effects 0.000 claims abstract description 10
- 230000002457 bidirectional effect Effects 0.000 claims abstract description 6
- 239000011800 void material Substances 0.000 claims description 10
- 230000009191 jumping Effects 0.000 claims 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Landscapes
- Image Analysis (AREA)
Abstract
A Kinect real-time depth map-to-color map registration and optimization method includes the steps of obtaining a depth map through Kinect, filling small holes of the depth map through a background filling method, and filling large holes of the depth map through a bidirectional linear scanning polling interpolation method to obtain a primary depth map image. Then, a coordinate mapping table from a depth map to a color map is established and calculated, abnormal coordinate parts are processed, areas with invalid ghosting are found and cut out, then, a projection image from the depth map to the color map is established, an RGB-depth map with a series of discrete depth points is obtained, then, filling of a cavity part is carried out through a linear scanning decision value-taking method, and the depth map and the color map which are obtained in the way are cut into corresponding sizes, so that a real-time and accurate RGB-depth map can be obtained. The invention provides a Kinect-based real-time depth image and color image registration and optimization method which is good in real-time performance, good in stability and high in accuracy.
Description
Technical Field
The invention relates to the fields of image processing, numerical analysis, three-dimensional reconstruction, computer science, parallel computation and the like, in particular to a method for registering and optimizing a real-time depth map and a color map based on Kinect.
Background
The depth generation technology of video is a classic problem in the field of computer vision. To date, no algorithm has been able to guarantee the accuracy of depth generation for complex scenes while guaranteeing the real-time performance of video depth generation. People therefore prefer to use external devices for simultaneous capture of depth and video to ensure real-time and accuracy of depth information.
In recent years, microsoft has introduced a light, small and cheap Kinect device, which can acquire a depth image and a color image in a scene in real time, but due to the fact that environmental factors in the scene interfere with infrared light, the depth image has many hollow points, and the Kinect device has poor practical experience under the condition of high requirements on the quality of the depth image. And the positions and resolutions of the infrared receiving camera and the color camera of the Kinect are different, so that the positions and pixel points of the depth image and the color image are not corresponding, although microsoft provides a mapping method from the depth image to the color image in the Kinect SDK, in fact, the method is based on the DIBR technology, and projects an image with low resolution to an image with high resolution, so that a large number of hollow points are generated, and the problems of ghost image, shielding and the like are caused.
Disclosure of Invention
In order to overcome the defects of poor instantaneity, poor stability and poor accuracy of the existing mapping method of the depth map and the color map, the invention provides a Kinect-based real-time depth map and color map registration and optimization method which is good in instantaneity, good in stability and high in accuracy.
Therefore, the invention adopts the following technical scheme that the method comprises the following steps:
a Kinect-based real-time depth map and color map registration and optimization method comprises the following steps:
(1) connecting the Kinect equipment by using an application programming interface Kinect SDK2.0 pushed by Microsoft;
(2) acquiring an original depth data stream and a color data stream;
(3) filling fine holes of the depth map by using the background frame;
(4) filling a large hole of the depth map by using a bidirectional linear scanning and polling interpolation method;
(5) obtaining a coordinate mapping relation m _ pColorcoordinates of the depth image and the color image through a function MapDepthFrameToColorSpace provided by Kinect SDK 2.0;
(6) map table m _ pColorCoordinates coordinate exception part handling;
(7) establishing a depth image color _ depth aligned with the color image;
(8) and cutting the color image and the aligned depth image, and subtracting the non-overlapped area of the shooting areas of the depth camera and the color camera, thereby realizing the complete alignment of the depth image and the color image.
Further, in the step (3), the method for filling the fine holes in the depth map by using the background frame is as follows:
3.1) background frame generation: first-read depth image data Dep0Updating of background frame as background frame image data: the formula is as follows
BacDep(x,y)＝Dep0(x,y) Dep0(x,y)>BacDep (x, y) +10
Wherein BacDep (x, y) is depth data of each pixel point of background frame, and Dep0(x, y) is the depth data of each pixel point of the current depth image;
3.2) filling the small holes of the current depth frame by using the background frame: for each pixel point of the depth map if and only if Dep0(x, y) < boundary value and BacDep (x, y) > boundary value, where Dep0 is the depth image and BacDep is the background image, the boundary value is used to distinguish whether it is a hole point, the following steps are performed:
3.2.1) setting the expansion number to expand to 1, and executing 3.2.2);
3.2.2) extending the point (x, y) in the Dep0 to the adjacent pixel points to obtain (x + extended, y), (x-extended, y), (x, y + extended), (x, y-extended), if there is one of the points belonging to the background frame, that is, the absolute difference between the pixel point and the corresponding pixel point of the background frame is less than 10, then Dep0(x, y) is BacDep (x, y);
3.2.3) if the expanded scanning times are limited, the expanded is increased by 1, and the jump is carried out to 3.2.1); otherwise, ending the step.
Still further, in step (4), the method for filling the large holes in the depth map by using the bidirectional linear scanning polling interpolation method is as follows:
4.1) setting endSC to be scanning times, initializing endSC to be 0, and executing 4.2);
4.2) scanning Dep from the upper right to the lower left of the image0For each pixel point (x, y), if it is a hole point, executing the third step; otherwise, skipping the point and continuing traversal, and executing 4.5) after traversal is finished;
4.3) scanning the endSC 3 points to the left from (x, y) until either (0, y) or (x-endSC 3, y) or a non-void point (index, y) is scanned, wherein x-endSC 3 ≦ index < x. When a non-hole point (index, y) is retrieved, then scan back from point (index, y) to point (x, y), there is the following formula:
Dep0(i,y)＝Dep0(index,y)index＜i≤x
after the scanning is finished, if the (x, y) is still the cavity point, executing the fourth step; otherwise return to 4.2);
4.4) scan down endSC 3 points from (x, y) until either (x, depthHeight) or (x, y + endSC 3) or a non-void point (x, index) is scanned, where y < index ≦ y + endSC 3, depthHeight being the height of the depth image. When a non-hole point (x, index) is retrieved, then (x, y) is scanned back from (x, index), with the following equation:
Dep0(x,i)＝Dep0(x,index)y＜i≤index
returning to 4.2) after the scanning is finished;
4.5) scanning the image from the lower left to the upper right to fill the cavity once according to the steps of 4.2) to 4.4), and executing 4.6) after traversing is finished;
4.6) if the endSC is not equal to the upper limit of the scanning times, the endSC is increased by 1 and jumps to 4.2); otherwise, ending the step.
Further, in step (6), the map table m _ pColorcordinates coordinate exception part processing is as follows:
6.1) coordinate left edge exception part handling
For each point (0, y) at the leftmost end of the mapping table, where 0 ≦ y < depthHeight,
depthHeight is the height of the depth map, and the following steps are performed:
6.1.1) scanning to the right from (0, y) of the mapping table, and if the scanned point is (index, y), extracting three continuous pixel points (index, y), (index +1, y), (index +2, y), extracting corresponding coordinate points A, B, C from the mapping table, and executing 6.1.2); if the index value is depthWidth, stopping scanning and finishing the step; wherein depthWidth is the width of the depth map;
6.1.2) if both A, B and B, C manhattan distances are less than a predefined threshold, then A, B, C are considered to be consecutive coordinate points without anomalies, and then 6.1.3) is performed; otherwise, return to 6.1.1); wherein the Manhattan distance formula is as follows:
d(i，j)＝|xi-xj|+|yi-yj|
wherein d (i, j) is the Manhattan distance, xi, yi are the coordinates of the first coordinate point, xj, yj are the coordinates of the second coordinate point;
6.1.3) go back from (index, y) to (0, y), for each point (i, y) in between, 0 ≦ i < index, resetting it at the value m _ pColorcordinates (i, y) of the mapping table, assuming the coordinates D, with the formula:
D.X＝A.X-(index-i)*3
D.Y＝A.Y
wherein D.X and D.Y are X and Y coordinate values of D, and A.X and A.Y are X and Y coordinate values of A;
returning to 6.1.1 after traversing is finished);
6.2) coordinate Right edge Exception part handling
Processing a rightmost point (depthWidth, y) according to the step 6.1), wherein y is more than or equal to 0 and less than depthHeight, and the depthWidth and the depthHeight are respectively the width and the height of the depth map;
6.3) marking ghost coordinate area and internal coordinate abnormal area
And establishing a ghost mark table map _ use with the size of depthWidth depthHeight, and initializing the value of each point in the map _ use to be 0, wherein the ghost mark table map _ use represents a region which is not a ghost or abnormal coordinates. For each point (0, y) at the leftmost end of the mapping table, where 0 ≦ y < depthHeight, the following steps are performed:
6.3.1) scanning to the right from (0, y) of the mapping table m _ pColorcordinates, if the scanned point is (index, y), extracting two continuous pixel points (index, y), (index +1, y), extracting corresponding coordinate points A and B from the mapping table, and executing 6.3.2); and if the index value is depthWidth, stopping scanning and ending the step.
6.3.2) if the X coordinate of A, B conforms to the relationship A.X > B.X, then B is considered to be a coordinate outlier, followed by 6.3.3); otherwise, return to 6.3.1);
6.3.3) start scanning to the right from (index +1, y), set the scanned point as (i, y) and the corresponding coordinate as C, if C.X < A.X, the value of the mark map _ user (i, y) is 1, which indicates that the corresponding point is in the ghost or abnormal point region, and then continue scanning; otherwise, the scan is stopped, and the process returns to 6.3.1).
In step (7), the process of establishing the depth image color _ depth aligned with the color map is as follows:
7.1) establishing a depth image color _ depth aligned with the color map
Initializing all point correspondence values in color _ depth to be 0, and then reversely taking a depth value according to a coordinate mapping table, wherein the formula is as follows:
color_depth(m_pColorCoordinates(i,j))＝Dep0(i,j)map_use(i,j)≠1
wherein m _ pColorcordinates (i, j) represents the pixel point coordinates in the color image corresponding to the point (i, j) in the depth image, and the value of map _ use (i, j) is 1, which represents that the point (i, j) is in the ghost image area;
7.2) filling the holes by a linear scanning polling interpolation method, comprising the following steps:
7.2.1) set endSC to scan times, initialize endSC to 0, execute 7.2.2);
7.2.2) scanning color _ depth from the upper right to the lower left of the image, and for each pixel point (x, y), executing 7.2.3 if the pixel point is a hole point); otherwise, skipping this point and continuing the traversal; jump to 7.2.5 after traversal);
7.2.3) sweep the endSC 3 points to the left from (x, y) until either (0, y) or (x-endSC 3, y) or a non-void point (index, y) is swept, where x-endSC 3 ≦ index < x. When a non-hole point (index, y) is retrieved, then (x, y) is scanned back from (index, y), with the following equation:
color_depth(i,y)＝color_depth(index,y)index＜i≤x
after the scanning is finished, if (x, y) is still the cavity point, 7.2.4 is executed; otherwise, go back to 7.2.2);
7.2.4) scan down endSC 3 points from (x, y) until either (x, colorHeight) or (x, y + endSC 3) or a non-void point (x, index) is scanned, where y < index ≦ y + endSC 3, colorHeight being the height of the color image. When a non-hole point (x, index) is retrieved, then (x, y) is scanned back from (x, index), with the following equation:
color_depth(x,i)＝color_depth(x,index)y＜i≤index
after the scanning is finished, returning to 7.2.2);
7.2.5) if endSC is not equal to the upper limit of scanning times, then endSC increases by 1 and jumps to 7.2.2); otherwise, ending the step.
The invention has the beneficial effects that: the real-time performance is good, the stability is good, and the accuracy is higher.
Drawings
Fig. 1 is a flowchart of a method for registering and optimizing a color map and a real-time depth map based on Kinect.
Fig. 2 is the resulting depth image from the final registration.
Fig. 3 is the image resulting from the final registration.
Detailed Description
The invention is further described below with reference to the accompanying drawings.
Referring to fig. 1 to 3, a method for registering and optimizing a real-time depth map and a color map based on Kinect includes the following steps:
(1) connecting the Kinect equipment by using an application programming interface Kinect SDK2.0 pushed by Microsoft;
(2) acquiring an original depth data stream and a color data stream;
(3) the depth image hole preliminary processing, utilize the background frame to fill up the tiny hole of the depth image, include the following steps:
3.1) background frame generation: first-read depth image data Dep0Updating of background frame as background frame image data: the formula is as follows
BacDep(x,y)＝Dep0(x,y) Dep0(x,y)>BacDep (x, y) +10
Wherein BacDep (x, y) is the data of each pixel point of the background frame, Dep0(x, y) is the data of each pixel point of the current depth image;
3.2) filling the tiny holes of the current depth frame by using the background frame: for each pixel point of the depth map if and only if Dep0(x,y)<50 and BacDep (x, y)>At 50, the following steps are executed:
3.2.1): setting the expansion number to be expanded to be 1, and executing 3.2.2);
3.2.2): points (x, y) in the Dep0 are respectively extended to adjacent pixels to obtain (x + extended, y), (x-extended, y), (x, y + extended), (x, y-extended), and if one of the points belongs to the background frame, that is, the absolute difference between the pixel point and the corresponding pixel point of the background frame is less than 10, then the Dep0(x, y) is BacDep (x, y);
3.2.3) if the expanded scanning times are limited, the expanded is increased by 1, and the jump is carried out to 3.2.1); otherwise, ending the step;
(4) depth map hole depth processing, a bidirectional linear scanning polling interpolation method for filling holes, comprising the following steps:
4.1) setting endSC to be scanning times, initializing endSC to be 0, and executing 4.2);
4.2) scanning Dep from the upper right to the lower left of the image0For each pixel point (x, y), if it is a hole point, executing the third step; otherwise skip this pointContinuously traversing, and executing 4.5) after traversing is finished;
4.3) scanning the endSC 3 points to the left from (x, y) until either (0, y) or (x-endSC 3, y) or a non-void point (index, y) is scanned, wherein x-endSC 3 ≦ index < x. When a non-hole point (index, y) is retrieved, then scan back from point (index, y) to point (x, y), there is the following formula:
Dep0(i,y)＝Dep0(index,y)index＜i≤x
after the scanning is finished, if the (x, y) is still the cavity point, executing the fourth step; otherwise return to 4.2);
4.4) scan down endSC 3 points from (x, y) until either (x, depthHeight) or (x, y + endSC 3) or a non-void point (x, index) is scanned, where y < index ≦ y + endSC 3, depthHeight being the height of the depth image. When a non-hole point (x, index) is retrieved, then (x, y) is scanned back from (x, index), with the following equation:
Dep0(x,i)＝Dep0(x,index)y＜i≤index
returning to 4.2) after the scanning is finished;
4.5) scanning the image from the lower left to the upper right to fill the cavity once according to the steps of 4.2) to 4.4), and executing 4.6) after traversing is finished;
4.6) if the endSC is not equal to the upper limit of the scanning times, the endSC is increased by 1 and jumps to 4.2); otherwise, ending the step.
(5) Establishing coordinate mapping relation of depth image and color image
Obtaining the coordinate mapping relation m _ pColorcordinates of the depth image and the color image through a function MapDepthFrameToColorSpace provided by Kinect SDK2.0, wherein the coordinate mapping relation m _ pColorCoordinates is stored, wherein each depth point is in a color image Col0Corresponding coordinates of (a).
(6) The mapping table m _ pColorcordinates coordinate exception part processing proceeds as follows:
6.1) coordinate left edge exception part handling
For each point (0, y) at the leftmost end of the mapping table, where 0 ≦ y < depthHeight,
depthHeight is the height of the depth map, and the following steps are performed:
6.1.1) scanning to the right from (0, y) of the mapping table, and if the scanned point is (index, y), extracting three continuous pixel points (index, y), (index +1, y), (index +2, y), extracting corresponding coordinate points A, B, C from the mapping table, and executing 6.1.2); if the index value is depthWidth, stopping scanning and finishing the step; wherein depthWidth is the width of the depth map;
6.1.2) if both A, B and B, C manhattan distances are less than a predefined threshold, then A, B, C are considered to be consecutive coordinate points without anomalies, and then 6.1.3) is performed; otherwise, return to 6.1.1); wherein the Manhattan distance formula is as follows:
d(i，j)＝|xi-xj|+|yi-yj|
wherein d (i, j) is the Manhattan distance, xi, yi are the coordinates of the first coordinate point, xj, yj are the coordinates of the second coordinate point;
6.1.3) go back from (index, y) to (0, y), for each point (i, y) in between, 0 ≦ i < index, resetting it at the value m _ pColorcordinates (i, y) of the mapping table, assuming the coordinates D, with the formula:
D.X＝A.X-(index-i)*3
D.Y＝A.Y
wherein D.X and D.Y are X and Y coordinate values of D, and A.X and A.Y are X and Y coordinate values of A;
returning to 6.1.1 after traversing is finished);
6.2) coordinate Right edge Exception part handling
Processing a rightmost point (depthWidth, y) according to the step 6.1), wherein y is more than or equal to 0 and less than depthHeight, and the depthWidth and the depthHeight are respectively the width and the height of the depth map;
6.3) marking ghost coordinate area and internal coordinate abnormal area
And establishing a ghost mark table map _ use with the size of depthWidth depthHeight, and initializing the value of each point in the map _ use to be 0, wherein the ghost mark table map _ use represents a region which is not a ghost or abnormal coordinates. For each point (0, y) at the leftmost end of the mapping table, where 0 ≦ y < depthHeight, the following steps are performed:
6.3.1) scanning to the right from (0, y) of the mapping table m _ pColorcordinates, if the scanned point is (index, y), extracting two continuous pixel points (index, y), (index +1, y), extracting corresponding coordinate points A and B from the mapping table, and executing 6.3.2); and if the index value is depthWidth, stopping scanning and ending the step.
6.3.2) if the X coordinate of A, B conforms to the relationship A.X > B.X, then B is considered to be a coordinate outlier, followed by 6.3.3); otherwise, return to 6.3.1);
6.3.3) start scanning to the right from (index +1, y), set the scanned point as (i, y) and the corresponding coordinate as C, if C.X < A.X, the value of the mark map _ user (i, y) is 1, which indicates that the corresponding point is in the ghost or abnormal point region, and then continue scanning; otherwise, the scan is stopped, and the process returns to 6.3.1).
(7) Establishing a depth image color _ depth aligned with the color image, initializing all point correspondence values in the color _ depth to be 0, and carrying out the following process:
7.1) establishing a depth image color _ depth aligned with the color map
Initializing all point correspondence values in color _ depth to be 0, and then reversely taking a depth value according to a coordinate mapping table, wherein the formula is as follows:
color_depth(m_pColorCoordinates(i,j))＝Dep0(i,j)map_use(i,j)≠1
wherein m _ pColorcordinates (i, j) represents the pixel point coordinates in the color image corresponding to the point (i, j) in the depth image, and the value of map _ use (i, j) is 1, which represents that the point (i, j) is in the ghost image area;
7.2) filling the holes by a linear scanning polling interpolation method, comprising the following steps:
7.2.1) set endSC to scan times, initialize endSC to 0, execute 7.2.2);
7.2.2) scanning color _ depth from the upper right to the lower left of the image, and for each pixel point (x, y), executing 7.2.3 if the pixel point is a hole point); otherwise, skipping this point and continuing the traversal; jump to 7.2.5 after traversal);
7.2.3) sweep the endSC 3 points to the left from (x, y) until either (0, y) or (x-endSC 3, y) or a non-void point (index, y) is swept, where x-endSC 3 ≦ index < x. When a non-hole point (index, y) is retrieved, then (x, y) is scanned back from (index, y), with the following equation:
color_depth(i,y)＝color_depth(index,y)index＜i≤x
after the scanning is finished, if (x, y) is still the cavity point, 7.2.4 is executed; otherwise, go back to 7.2.2);
7.2.4) scan down endSC 3 points from (x, y) until either (x, colorHeight) or (x, y + endSC 3) or a non-void point (x, index) is scanned, where y < index ≦ y + endSC 3, colorHeight being the height of the color image. When a non-hole point (x, index) is retrieved, then (x, y) is scanned back from (x, index), with the following equation:
color_depth(x,i)＝color_depth(x,index)y＜i≤index
after the scanning is finished, returning to 7.2.2);
7.2.5) if endSC is not equal to the upper limit of scanning times, then endSC increases by 1 and jumps to 7.2.2); otherwise, ending the step.
(8) Cropping color images and aligned depth images
And subtracting the area where the shooting areas of the depth camera and the color camera are not overlapped. Thereby achieving perfect alignment of the depth image and the color image.
(9) And outputting the registered depth image and color image, wherein fig. 2 is the final registered depth image, and fig. 2 is the final registered image (displayed in gray scale and obtained in actual processing as a color image).
Claims (4)
1. A method for registering and optimizing a real-time depth map and a color map based on Kinect is characterized by comprising the following steps: the method comprises the following steps:
(1) connecting the Kinect equipment by using an application programming interface Kinect SDK2.0 pushed by Microsoft;
(2) acquiring an original depth data stream and a color data stream;
(3) filling fine holes of the depth map by using the background frame;
(4) filling a large hole of the depth map by using a bidirectional linear scanning and polling interpolation method;
(5) obtaining a coordinate mapping relation m _ pColorcoordinates of the depth image and the color image through a function MapDepthFrameToColorSpace provided by Kinect SDK 2.0;
(6) map table m _ pColorCoordinates coordinate exception part handling; the process is as follows:
(6.1) coordinate left edge exception portion handling
For each point (0, y) at the leftmost end in the mapping table, where 0 ≦ y < depthHeight, which is the height of the depth map, the following steps are performed:
(6.1.1) scanning rightward from (0, y) of the mapping table, setting the scanned point as (index, y), extracting three continuous pixel points (index, y), (index +1, y), (index +2, y), extracting corresponding coordinate points A, B, C from the mapping table, and executing 6.1.2); if the index value is depthWidth, stopping scanning and finishing the step; wherein depthWidth is the width of the depth map;
(6.1.2) if the A, B and B, C manhattan distances are both less than a predefined threshold, then consider a, B, C to be continuous coordinate points without anomalies, and then perform (6.1.3); otherwise, returning to (6.1.1); wherein the Manhattan distance formula is as follows:
d(i，j)＝|xi-xj|+|yi-yj|
wherein d (i, j) is the Manhattan distance, xi, yi are the coordinates of the first coordinate point, xj, yj are the coordinates of the second coordinate point;
(6.1.3) go back from (index, y) to (0, y), for each point (i, y) in between, 0 ≦ i < index, resetting it to the value m _ pColorcordinates (i, y) in the mapping table, assuming the coordinates D, for the following formula:
D.X＝A.X-(index-i)*3
D.Y＝A.Y
wherein D.X and D.Y are X and Y coordinate values of D, and A.X and A.Y are X and Y coordinate values of A;
returning to (6.1.1) after traversing;
(6.2) coordinate right edge exception portion handling
Processing a rightmost point (depthWidth, y) according to the step (6.1), wherein y is more than or equal to 0 and less than depthHeight, and the depthWidth and the depthHeight are respectively the width and the height of the depth map;
(6.3) marking ghost coordinate region and internal coordinate abnormal region
Establishing a ghost mark table map _ use with the size of depthWidth depthHeight, initializing the value of each point in the map _ use to be 0, representing that the area is not a ghost or abnormal coordinates, and executing the following steps for each point (0, y) at the leftmost end in the mapping table, wherein y is more than or equal to 0 and less than depthHeight:
(6.3.1) scanning to the right from (0, y) of a mapping table m _ pColorcordinates, if the scanned point is (index, y), extracting two continuous pixel points (index, y), (index +1, y), extracting corresponding coordinate points A and B from the mapping table, and executing (6.3.2); if the index value is depthWidth, stopping scanning and finishing the step;
(6.3.2) if the X-coordinate of A, B conforms to the relationship A.X > B.X, considering B as a coordinate outlier, and then performing (6.3.3); otherwise, returning to (6.3.1);
(6.3.3) scanning to the right from (index +1, y), setting the scanned point as (i, y) and the corresponding coordinate as C, if C.X < A.X, the value of the mark map _ user (i, y) is 1, which indicates that the corresponding point is in the ghost or abnormal point region, and then continuing the scanning; otherwise, stopping scanning and returning to (6.3.1);
(7) establishing a depth image color _ depth aligned with the color image;
(8) and cutting the color image and the aligned depth image, and subtracting the non-overlapped area of the shooting areas of the depth camera and the color camera, thereby realizing the complete alignment of the depth image and the color image.
2. The method as claimed in claim 1, wherein the method for registering and optimizing the color map and the depth map in real time based on Kinect is characterized by: in the step (3), the method for filling the fine holes in the depth map by using the background frame is as follows:
(3.1) background frame generation: first-read depth image data Dep0Updating of background frame as background frame image data: the formula is as follows
BacDep(x,y)＝Dep0(x,y)Dep0(x,y)>BacDep (x, y) +10
Wherein BacDep (x, y) is the data of each pixel point of the background frame, Dep0(x, y) is the data of each pixel point of the current depth image;
(3.2) filling the small holes of the current depth frame by using the background frame: for each pixel point of the depth map if and only if Dep0(x, y) < boundary value and BacDep (x, y) > boundary value, where Dep0 is the depth image and BacDep is the background image, the boundary value is used to distinguish whether it is a hole point, the following steps are performed:
(3.2.1) setting the expansion number to expand to 1, and executing (3.2.2);
(3.2.2) extending the points (x, y) in the Dep0 to adjacent pixels to obtain (x + extended, y), (x-extended, y), (x, y + extended), (x, y-extended), respectively, and if one of the points belongs to the background frame, that is, the absolute difference between the corresponding pixel of the pixel and the background frame is less than 10, then Dep0(x, y) is BacDep (x, y);
(3.2.3) if the expanded scanning times are in an upper limit, increasing 1 for the expanded and jumping back to (3.2.1); otherwise, ending the step.
3. The Kinect-based real-time depth map and color map registration and optimization method as claimed in claim 1, wherein: in the step (4), the method for filling the large holes in the depth map by using the bidirectional linear scanning polling interpolation method is as follows:
(4.1) setting the endSC to be the scanning times, initializing the endSC to be 0, and executing (4.2);
(4.2) scanning Dep from the upper right to the lower left of the image0For each pixel point (x, y), if it is a hole point, executing the third step; otherwise, skipping the point and continuing the traversal, and executing the step (4.5) after the traversal is finished;
(4.3) scanning the endSC 3 points to the left from (x, y) until either (0, y) or (x-endSC 3, y) or the non-hole point (index, y) is scanned, where x-endSC 3 ≦ index < x, and when the non-hole point (index, y) is retrieved, scanning back from the point (index, y) to the point (x, y), with the following formula:
Dep0(i,y)＝Dep0(index,y)index＜i≤x
after the scanning is finished, if the (x, y) is still the cavity point, executing the fourth step; otherwise go back to 94.2);
(4.4) scanning down endSC 3 points from (x, y) until either (x, depthHeight) or (x, y + endSC 3) or a non-hole point (x, index) is scanned, where y < index ≦ y + endSC 3, depthHeight is the height of the depth image, and when a non-hole point (x, index) is retrieved, then scanning back to (x, y) from (x, index), with the following formula:
Dep0(x,i)＝Dep0(x,index)y＜i≤index
returning to 4.2) after the scanning is finished;
(4.5) scanning the image from the left lower part to the right upper part to fill the hole once according to the steps of (4.2) - (4.4), and executing (4.6) after traversing is finished;
(4.6) if the endSC is not equal to the upper limit of the scanning times, the endSC is increased by 1 and jumps to (4.2); otherwise, ending the step.
4. The Kinect-based real-time depth map and color map registration and optimization method as claimed in claim 1, wherein: in step (7), the process of establishing the depth image color _ depth aligned with the color map is as follows:
(7.1) establishing a depth image color _ depth aligned with the color map
Initializing all point correspondence values in color _ depth to be 0, and then reversely taking a depth value according to a coordinate mapping table, wherein the formula is as follows:
color_depth(m_pColorCoordinates(i,j))＝Dep0(i,j)map_use(i,j)≠1
wherein m _ pColorcordinates (i, j) represents the pixel point coordinates in the color image corresponding to the point (i, j) in the depth image, and the value of map _ use (i, j) is 1, which represents that the point (i, j) is in the ghost image area;
(7.2) filling the holes by a linear scanning polling interpolation method, comprising the following steps:
(7.2.1) setting the endSC to be the scanning times, initializing the endSC to be 0, and executing (7.2.2);
(7.2.2) scanning color _ depth from the upper right to the lower left of the image, and executing 7.2.3 if the pixel is a hole point for each pixel point (x, y); otherwise, skipping this point and continuing the traversal; jumping to (7.2.5) after traversing;
(7.2.3) scanning the endSC 3 points to the left from (x, y) until either (0, y) or (x-endSC 3, y) or the non-hole point (index, y) is scanned, where x-endSC 3 ≦ index < x, and when the non-hole point (index, y) is retrieved, scanning back to (x, y) from (index, y), with the following formula:
color_depth(i,y)＝color_depth(index,y)index＜i≤x
after the scanning is finished, if (x, y) is still the hole point, executing (7.2.4); otherwise, returning to (7.2.2);
(7.2.4) scanning down endSC 3 points from (x, y) until either (x, colorHeight) or (x, y + endSC 3) or a non-void point (x, index) is scanned, where y < index ≦ y + endSC 3, colorHeight being the height of the color image, and when a non-void point (x, index) is retrieved, then scanning back from (x, index) to (x, y), having the following formula:
color_depth(x,i)＝color_depth(x,index)y＜i≤index
after the scanning is finished, returning to (7.2.2);
(7.2.5) if endSC is not equal to the upper limit of scanning times, then endSC increases by 1 and jumps to (7.2.2);
otherwise, ending the step.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611232116.3A CN106780474B (en) | 2016-12-28 | 2016-12-28 | Kinect-based real-time depth map and color map registration and optimization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611232116.3A CN106780474B (en) | 2016-12-28 | 2016-12-28 | Kinect-based real-time depth map and color map registration and optimization method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106780474A CN106780474A (en) | 2017-05-31 |
| CN106780474B true CN106780474B (en) | 2020-01-10 |
Family
ID=58922544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611232116.3A Active CN106780474B (en) | 2016-12-28 | 2016-12-28 | Kinect-based real-time depth map and color map registration and optimization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106780474B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107343180A (en) * | 2017-08-15 | 2017-11-10 | 安徽大学 | A kind of more visual modalities data collecting systems and acquisition method |
| CN113256611B (en) * | 2021-06-21 | 2021-12-24 | 浙江光珀智能科技有限公司 | RGB-D registration precision testing method and device |
| CN115797426B (en) * | 2023-02-13 | 2023-05-12 | 合肥的卢深视科技有限公司 | Image alignment method, electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607584A (en) * | 2013-11-27 | 2014-02-26 | 浙江大学 | Real-time registration method for depth maps shot by kinect and video shot by color camera |
| CN104378619A (en) * | 2014-11-12 | 2015-02-25 | 合肥工业大学 | Rapid and efficient hole filling algorithm based on foreground and background gradient transition |
| CN105574838A (en) * | 2014-10-15 | 2016-05-11 | 上海弘视通信技术有限公司 | Image registration and splicing method of multi-view camera and device thereof |
-
2016
- 2016-12-28 CN CN201611232116.3A patent/CN106780474B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607584A (en) * | 2013-11-27 | 2014-02-26 | 浙江大学 | Real-time registration method for depth maps shot by kinect and video shot by color camera |
| CN105574838A (en) * | 2014-10-15 | 2016-05-11 | 上海弘视通信技术有限公司 | Image registration and splicing method of multi-view camera and device thereof |
| CN104378619A (en) * | 2014-11-12 | 2015-02-25 | 合肥工业大学 | Rapid and efficient hole filling algorithm based on foreground and background gradient transition |
Non-Patent Citations (4)
| Title |
|---|
| "Depth Hole Filling Using the Depth Distribution of Neighboring Regions of Depth Holes in the Kinect Sensor";Na-Eun Yang 等;《IEEE》;20121231;第658-670页 * |
| Motion-Depth: RGB-D Depth Map Enhancement with Motion and Depth in Complement;Tak-Wai Hui 等;《2014 IEEE Conference on Computer Vision and Pattern Recognition》;20141231;第3962-3968页 * |
| 基于Kinect的物体三维模型构建和姿态估计方法研究;石广升;《中国优秀硕士学位论文全文数据库 信息科技辑》;20160215;正文第1-50页 * |
| 深度与彩色相机的联合标定及其在增强现实中的应用;琚旋;《中国优秀硕士学位论文全文数据库 信息科技辑》;20140615(第6期);正文第1-50页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106780474A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6351238B2 (en) | Image processing apparatus, imaging apparatus, and distance correction method | |
| JP5870273B2 (en) | Object detection apparatus, object detection method, and program | |
| CN110300292B (en) | Projection distortion correction method, device, system and storage medium | |
| CN103810685B (en) | A kind of super-resolution processing method of depth map | |
| CN101853528B (en) | Hand-held three-dimensional surface information extraction method and extractor thereof | |
| KR101121034B1 (en) | System and method for obtaining camera parameters from multiple images and computer program products thereof | |
| CN106548516B (en) | Three-dimensional roaming method and device | |
| KR101870902B1 (en) | Image processing apparatus and image processing method | |
| WO2007052191A2 (en) | Filling in depth results | |
| JPWO2020179065A1 (en) | Image processing equipment, image processing methods and programs | |
| CN106780474B (en) | Kinect-based real-time depth map and color map registration and optimization method | |
| JP7387434B2 (en) | Image generation method and image generation device | |
| CN107346040B (en) | Method and device for determining grating parameters of naked eye 3D display equipment and electronic equipment | |
| CN111080776B (en) | Human body action three-dimensional data acquisition and reproduction processing method and system | |
| CN107680039B (en) | Point cloud splicing method and system based on white light scanner | |
| CN110516639B (en) | Real-time figure three-dimensional position calculation method based on video stream natural scene | |
| CN109717966B (en) | Image simulation method and image simulation device for tooth shaping | |
| KR102464271B1 (en) | Pose acquisition method, apparatus, electronic device, storage medium and program | |
| CN112351325B (en) | Gesture-based display terminal control method, terminal and readable storage medium | |
| JP2011237296A (en) | Three dimensional shape measuring method, three dimensional shape measuring device, and program | |
| US20230062973A1 (en) | Image processing apparatus, image processing method, and storage medium | |
| KR101125061B1 (en) | A Method For Transforming 2D Video To 3D Video By Using LDI Method | |
| CN104537627A (en) | Depth image post-processing method | |
| JP2005332177A (en) | Three-dimensional information processor, three-dimensional information processing method, and three-dimensional information processing program | |
| CN110288707B (en) | Three-dimensional dynamic modeling method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Feng Yuanjing Inventor after: Huang Chenchen Inventor after: Chen Feng Inventor after: Huang Liangpeng Inventor after: Li Jiajing Inventor after: Pan Shanwei Inventor after: Yang Yong Inventor after: Hu Jianqiao Inventor after: Kong Deping Inventor after: Chen Hong Inventor before: Feng Yuanjing Inventor before: Chen Feng Inventor before: Huang Liangpeng Inventor before: Li Jiajing Inventor before: Xu Zenan Inventor before: Ye Jiasheng Inventor before: Chen Wenzhou Inventor before: Li Dingbang Inventor before: Wang Zenan |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |