CN112446843B - Image reconstruction method, system, equipment and medium based on multiple depth maps - Google Patents
Image reconstruction method, system, equipment and medium based on multiple depth maps Download PDFInfo
- Publication number
- CN112446843B CN112446843B CN201910799300.3A CN201910799300A CN112446843B CN 112446843 B CN112446843 B CN 112446843B CN 201910799300 A CN201910799300 A CN 201910799300A CN 112446843 B CN112446843 B CN 112446843B
- Authority
- CN
- China
- Prior art keywords
- depth
- image
- pixel point
- coefficient
- depth image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012216 screening Methods 0.000 claims abstract description 90
- 230000004927 fusion Effects 0.000 claims description 38
- 230000008859 change Effects 0.000 claims description 4
- 238000003708 edge detection Methods 0.000 claims description 3
- 238000002366 time-of-flight method Methods 0.000 description 18
- 238000012986 modification Methods 0.000 description 12
- 230000004048 modification Effects 0.000 description 12
- 238000012545 processing Methods 0.000 description 10
- 238000001914 filtration Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000005265 energy consumption Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000010146 3D printing Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/557—Depth or shape recovery from multiple images from light fields, e.g. from plenoptic cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image reconstruction method, system, equipment and medium based on a multi-depth map, which comprises the following steps: acquiring a plurality of depth images of at least one target object, wherein the plurality of depth images have different pixel point densities; determining a screening coefficient for each pixel point in each depth image; screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points; and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image. According to the method and the device, the plurality of pixel points can be screened out from the plurality of depth images with different pixel point densities, so that the screened plurality of pixel points are fused to generate the target depth image, and therefore the pixel points with higher precision in the comprehensive plurality of depth images can be used for generating a finer depth image.
Description
Technical Field
The invention relates to image fusion, in particular to an image reconstruction method, an image reconstruction system, image reconstruction equipment and an image reconstruction medium based on multiple depth maps.
Background
The device using TOF technology for imaging is called as TOF camera (or TOF camera), and the TOF camera is similar to the common machine vision imaging process and consists of a light source, an optical component, a sensor (TOF chip), a control circuit, a processing circuit and other units. TOF technology is basically similar to the principle of a 3D laser sensor, but only the 3D laser sensor scans point by point, and the TOF camera obtains depth information of the whole image at the same time.
The TOF technology has rich application scenes and is applied to various fields such as automobiles, industry, face recognition, logistics, pacifying monitoring, health, games, entertainment, film special effects, 3D printing, robots and the like. In the automotive field, TOF sensor can be used to autopilot, through TOF technique to the driving environment perception to acquire environmental information in order to increase the security, in addition TOF can also be used to the passenger in the car and leave the detection. In the industrial field, TOF sensors can be used as HMI (Human interface: human MACHINE INTERFACE), where workers and robots need to work together at very close distances in highly automated factories, TOF devices can be used to control safe distances in various situations. In a face recognition system, the luminance image and the depth information of a TOF camera can be connected through a model, so that face matching and detection can be completed rapidly and accurately.
However, in the conventional TOF depth camera, a depth map of a target object is obtained by emitting a surface light source signal, receiving a returned light signal at a sensor, and calculating the distance of the target object. But due to signal energy constraint, imaging characteristics and limitations, the obtained depth map has the problems of poor precision, missing depth information and the like.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide an image reconstruction method, an image reconstruction system, image reconstruction equipment and an image reconstruction medium based on multiple depth maps.
The image reconstruction method based on the multi-depth map provided by the invention comprises the following steps:
step S1: acquiring a plurality of depth images of at least one target object, wherein the plurality of depth images have different pixel point densities;
step S2: determining a screening coefficient for each pixel point in each depth image;
step S3: screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points;
Step S4: and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image.
Preferably, the method further comprises the following steps:
Step S5: and obtaining the target depth image, and carrying out depth reconstruction on the target object according to the target depth image.
Preferably, the step S2 includes the steps of:
step S201: extracting an area where the target object is located and an edge contour area of a scene area where the target object is located from each depth image;
Step S202: determining depth information of an area where the target object is located and the scene area in each depth image according to the edge contour area;
step S203: and respectively determining a screening coefficient for the edge contour area, the area where the target object is located and the depth information of the scene area.
Preferably, the step S2 includes the steps of:
step S201: acquiring a plurality of infrared images which are in one-to-one correspondence with the depth images and are acquired under the same light projection environment with the corresponding depth images;
Step S202: comparing the infrared image with the corresponding depth image to determine the corresponding amplitude of each pixel point in the depth image;
step S203: and determining a screening coefficient for the depth information of each pixel point at least according to the amplitude corresponding to each pixel point in each depth image.
Preferably, when the plurality of depth images are a first depth image generated based on floodlight projection and a second depth image generated based on lattice light projection, the step S2 includes the steps of:
step S201: acquiring a first infrared image and a second infrared image, wherein the first infrared image is an infrared image acquired based on floodlight projection, and the second infrared image is an infrared image acquired based on lattice light projection;
step S202: comparing the first infrared image with the first depth image to determine the amplitude corresponding to each pixel point in the first depth image, and comparing the second infrared image with the second depth image to determine the amplitude corresponding to each pixel point in the second depth image;
Step S203: determining a first confidence coefficient for the depth information of each pixel point according to at least the amplitude corresponding to each pixel point in the first depth image, and determining a second confidence coefficient for the depth information of each pixel point according to at least the amplitude corresponding to each pixel point in the second depth image;
step S204: extracting at least the region where the target object is located and the edge contour region of the scene region where the target object is located from the first depth image;
step S205: determining a region where the target object is located and the scene region in the first depth image according to the edge contour region;
step S206: determining a first fusion coefficient for the depth information of the edge contour region, and determining a second fusion coefficient for the depth information of the region where the target object is and the scene region;
step S207: generating a first screening coefficient according to the first confidence coefficient and the first fusion coefficient for each pixel point of the edge contour area, generating a second screening coefficient according to the first confidence coefficient and the second fusion coefficient for each pixel point of the area where the target object is located and the scene area, and generating a third screening coefficient according to the corresponding second confidence coefficient for each pixel point in the second depth image.
Preferably, the plurality of depth images comprise a plurality of depth images with different pixel point densities generated by performing lattice light projection with different densities on the target object;
or the depth image comprises a depth image generated by floodlight projection to the target object and a plurality of depth images with different pixel point densities generated by different density lattice light projection to the target object.
Preferably, in step S204, an edge contour area of the area where the target object is located and the scene area where the target object is located is determined in the first depth image by an edge detection algorithm.
The invention provides an image reconstruction system based on multiple depth maps, which is used for realizing the image reconstruction method based on the multiple depth maps, and comprises the following steps:
The image acquisition module is used for acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
the screening coefficient determining module is used for determining a screening coefficient for each pixel point in each depth image respectively;
The pixel point screening module is used for screening each pixel point in each depth image through the corresponding screening coefficient so as to generate a plurality of screened pixel points;
And the image fusion module is used for fusing a plurality of screened pixel points in the depth images to generate a target depth image.
The image reconstruction device based on the multi-depth map provided by the invention comprises:
A processor;
A memory having stored therein executable instructions of the processor;
Wherein the processor is configured to perform the steps of the multi-depth map based image reconstruction method via execution of the executable instructions.
The present invention provides a computer readable storage medium storing a program which when executed implements the steps of the multi-depth map based image reconstruction method.
Compared with the prior art, the invention has the following beneficial effects:
1. According to the method, the plurality of pixel points can be screened out from the plurality of depth images with different pixel point densities, so that the screened plurality of pixel points are fused to generate the target depth image, and therefore the pixel points with higher precision in the comprehensive plurality of depth images can be used for generating a finer depth image;
2. according to the method, the screening coefficients are respectively determined for the edge contour area, the area where the target object is located and the scene area, so that the display precision of the key area in the depth map can be improved, the key area of the depth map is clearer, and the edge of the target object is clearer;
3. according to the method, the confidence coefficient is determined through comparison of the infrared image and the depth image, and different fusion coefficients are set for the edge contour region and the region where the target object is located in the first depth image with higher pixel point density, so that depth information of pixels with poor precision and weak influence in the depth image can be removed, and the robustness of the model is improved;
4. The method and the device can be suitable for the 3D camera with floodlight projection and lattice light projection, the 3D camera can improve the projection distance and the depth information precision under the conditions of low energy consumption and low cost, and the first depth image based on floodlight projection and the second depth image based on lattice light projection, which are obtained by the 3D camera through the TOF depth sensor, can be fused together to generate the depth map with higher precision and finer edge.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art. Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a flowchart illustrating steps of a multi-depth map-based image reconstruction method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating steps of a multi-depth map-based image reconstruction method according to a modification of the present invention;
FIG. 3 is a flowchart illustrating steps for determining a filter coefficient by performing region division on a depth image according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a step of determining a filtering coefficient of each pixel according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating steps for determining a screening coefficient by integrating confidence and fusion coefficients in an embodiment of the present invention;
FIG. 6 is a schematic diagram of a 3D camera projecting lattice light to a target according to an embodiment of the present invention;
Fig. 7 is a schematic diagram of a 3D camera projecting floodlight to a target in an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of an image reconstruction device based on multiple depth maps according to the present invention; and
Fig. 9 is a schematic diagram of a computer-readable storage medium according to the present invention.
In the figure:
1 is a dot matrix projection module;
2 is the spatial distribution of the lattice light;
3 is a floodlight projection module;
4 is the spatial distribution of floodlight.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
The invention provides an image reconstruction method based on multiple depth maps, which aims to solve the problems in the prior art.
The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
The image reconstruction method based on the multi-depth map provided by the embodiment of the invention can be applied to a 3D camera with floodlight projection and lattice light projection, and the 3D camera can improve the projection distance and the depth information precision under the conditions of low energy consumption and low cost.
Fig. 1 is a step flowchart of an image reconstruction method based on multiple depth maps in an embodiment of the present invention, as shown in fig. 1, the image reconstruction method based on multiple depth maps provided in the present invention includes the following steps:
step S1: a plurality of depth images of at least one target object are acquired, wherein the depth images have different pixel point densities.
In practical applications, the execution main body of the embodiment of the invention can be a device with data processing capability, such as a computer, a central processing unit, a micro control unit MCU, a singlechip, a micro processor and the like. In this embodiment, a cpu is taken as an example to describe in detail, but the specific device type for executing the method in this embodiment is not limited.
In the embodiment of the invention, a plurality of pixel point densities with different densities can be generated by projecting lattice light with different densities to a target object. The lattice light may be periodically arranged or non-periodically arranged, for example, in periodic patterns such as triangles, quadrilaterals, rectangles, circles, hexagons, pentagons, and the like. The non-periodic arrangement is such as random arrangement, space coding arrangement, quasi-lattice arrangement, etc.
Step S2: and respectively determining a screening coefficient for each pixel point in each depth image.
In the embodiment of the present invention, when a filtering coefficient is determined for each pixel point in each depth image, each depth image may be divided into a plurality of regions, and a filtering coefficient is determined for each region, where each pixel point in the region has a filtering coefficient.
When the region division is performed in the depth image, a higher screening coefficient, such as 70% to 90%, may be set for the entire image or the region having the depth change, such as the edge region, the curved surface region, the inclined surface region, the convex region, the concave region, or the like of the object. A lower screening factor, such as 10% to 50%, is set for areas without depth change, such as planar areas.
In a modification of the present invention, a lower screening coefficient may also be set for a depth image with a higher pixel density, e.g., a depth image generated based on floodlight projection, where the screening coefficient is set between 10% and 50%, for example. If a higher screening factor is set for a depth image with a lower pixel density, e.g. 10% of the pixel count, the screening factor is set, e.g. between 70% and 90%. When the number of the light beams of the lattice light is small, the precision of each pixel point is high, a high screening coefficient is set at the moment, more pixel points are selected, and therefore the precision of the generated target depth image is high.
Step S3: and screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points.
In the embodiment of the invention, when the screening of the pixel points is performed by the screening coefficient, the screening is performed by random probability, that is, when the screening coefficient of a certain area is 90%, the probability that each pixel point of the area is selected is 90%.
Step S4: and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image.
In the embodiment of the invention, a plurality of pixel points can be screened out from a plurality of depth images with different pixel point densities, so that the screened plurality of pixel points are fused to generate a target depth image, and a finer depth image can be generated by integrating the pixel points with higher precision in the plurality of depth images.
In a modification of the present invention, a plurality of pixel points screened out from a plurality of depth images are fused to generate a target depth image, specifically, the confidence coefficient of each pixel point at a corresponding position of a plurality of depth images is compared, a depth value corresponding to the maximum confidence coefficient is selected, that is, the plurality of pixel points screened out from each depth image compare the confidence coefficient according to the position of the photodetector matrix, and the depth value corresponding to the maximum confidence coefficient is selected for fusion.
Fig. 2 is a flowchart illustrating steps of an image reconstruction method based on multiple depth maps according to a modification of the present invention, and as shown in fig. 2, the image reconstruction method based on multiple depth maps provided by the present invention further includes the following steps:
Step S5: and obtaining the target depth image, and carrying out depth reconstruction on the target object according to the target depth image.
In a modification of the present invention, the target object may be reconstructed in depth by an algorithm such as filtering interpolation, the target object may be reconstructed in depth by a markov random field model (Markov Random Filed, MRF), or the target object may be reconstructed in depth by a compressed sensing algorithm (Compressed Sensing, CS).
Fig. 3 is a flowchart of a step of determining a screening coefficient by performing region division on a depth image according to an embodiment of the present invention, as shown in fig. 3, the step S2 includes the following steps:
step S201: extracting an area where the target object is located and an edge contour area of a scene area where the target object is located from each depth image;
Step S202: determining depth information of an area where the target object is located and the scene area in each depth image according to the edge contour area;
step S203: and respectively determining a screening coefficient for the edge contour area, the area where the target object is located and the depth information of the scene area.
In the embodiment of the invention, the screening coefficient of the edge contour area is larger than the screening coefficient of the area where the target object is located and the scene area, for example, the screening coefficient of the edge contour area can be set to be 0.8, so that more pixel points are selected for the edge contour area; the screening coefficient of the area where the target object is located and the scene area is 0.2, so that smaller pixel points are selected for the area where the target object is located and the scene area, and the edge of the generated target depth map can be finer.
Fig. 4 is a flowchart of a step of determining a filtering coefficient of each pixel point according to an embodiment of the present invention, as shown in fig. 4, the step S2 includes the following steps:
Step S201: and acquiring a plurality of infrared images which are in one-to-one correspondence with the depth images and are acquired under the same light projection environment with the corresponding depth images.
In the embodiment of the invention, the same light projection environment is specifically that the infrared image and the depth image are collected under floodlight projection, or the infrared image and the depth image are collected under lattice light with the same number of light beams and the same light beam arrangement structure.
Step S202: and comparing the infrared image with the corresponding depth image to determine the corresponding amplitude of each pixel point in the depth image.
Step S203: and determining a screening coefficient for the depth information of each pixel point at least according to the amplitude corresponding to each pixel point in each depth image.
In the embodiment of the invention, a first confidence coefficient is determined for the depth information of each pixel point in the corresponding depth image based on the amplitude of each region in the infrared image, when the amplitude of one region in the infrared image is higher, a higher confidence coefficient is given to the pixel point in the same region corresponding to the depth image, and when the amplitude of one region in the infrared image is lower, a lower confidence coefficient is given to the pixel point in the same region corresponding to the depth image.
More specifically, if the amplitude of a region in the infrared image is 300W/m 2, a confidence level of 3 is assigned to the pixel point in the same region corresponding to the depth image; when the amplitude of a region in the infrared image is 100W/m 2, a confidence level of 1 is given to the pixel points in the same region corresponding to the depth image.
In a modification of the present invention, the amplitude of a region in the infrared image may be represented by temperature, and when the amplitude of a region in the infrared image is 30 ℃, confidence coefficient is given to a pixel point in the same region corresponding to the depth image, where the confidence coefficient is 3; and when the amplitude of one region in the infrared image is 10 ℃, giving confidence coefficient of 1 to the pixel point in the same region corresponding to the depth image.
Fig. 5 is a flowchart of a step of determining a screening coefficient by integrating a confidence coefficient and a fusion coefficient in an embodiment of the present invention, as shown in fig. 5, when a plurality of depth images are a first depth image generated based on floodlight projection and a second depth image generated based on lattice light projection, the step S2 includes the following steps:
step S201: the method comprises the steps of obtaining a first infrared image and a second infrared image, wherein the first infrared image is an infrared image collected based on floodlight projection, and the second infrared image is an infrared image collected based on lattice light projection.
In the embodiment of the invention, the first depth image corresponds to the first infrared image, that is, the first depth image and the first infrared image are collected under the same type of light; the second depth image corresponds to the second infrared image, i.e. the second depth image and the second infrared image are acquired under the same type of light.
Step S202: and comparing the first infrared image with the first depth image to determine the amplitude corresponding to each pixel point in the first depth image, and comparing the second infrared image with the second depth image to determine the amplitude corresponding to each pixel point in the second depth image.
In the embodiment of the invention, as the infrared images feel and reflect the difference of the outward amplitudes of the targets and the backgrounds, the amplitude of each area in the first infrared image and the second infrared image can be determined;
Because the first depth image and the first infrared image are images of the same target object, each pixel point on the first depth image corresponds to an area of the first infrared image, so that the amplitude corresponding to each pixel point on the first depth image can be determined, and similarly, the amplitude corresponding to each pixel point can also be determined for the second depth image.
Step S203: determining a first confidence coefficient for the depth information of each pixel point according to at least the amplitude corresponding to each pixel point in the first depth image, and determining a second confidence coefficient for the depth information of each pixel point according to at least the amplitude corresponding to each pixel point in the second depth image;
In the embodiment of the invention, a first confidence coefficient is determined for the depth information of each pixel point in the first depth image based on the amplitude of each region in the first infrared image, when the amplitude of one region in the first infrared image is higher, a higher confidence coefficient is given to the pixel point in the same region corresponding to the first depth image, and when the amplitude of one region in the first infrared image is lower, a lower confidence coefficient is given to the pixel point in the same region corresponding to the first depth image. Similarly, for the second depth image, when the amplitude of a region in the second infrared image is higher, a higher confidence is given to the pixel point in the same region corresponding to the second depth image, and when the amplitude of a region in the second infrared image is lower, a lower confidence is given to the pixel point in the same region corresponding to the second depth image.
More specifically, if the amplitude of a region in the first infrared image is 300, a confidence level of 3 is assigned to the pixel points in the same region corresponding to the first depth image; when the amplitude of a region in the first infrared image is 100, confidence is given to the pixel point in the same region corresponding to the first depth image, wherein the confidence is 1.
In a modification of the present invention, the amplitude of a region in the first infrared image may be represented by temperature, and when the amplitude of a region in the first infrared image is 30 ℃, confidence coefficient is given to a pixel point in the same region corresponding to the first depth image, where the confidence coefficient is 3; when the amplitude of a region in the first infrared image is 10 ℃, confidence is given to the pixel point in the same region corresponding to the first depth image as 1. The amplitude for each region in the second infrared image is similarly represented by temperature.
In a modification of the present invention, the determination of the first confidence may be performed by comprehensively considering the value of the depth information of each pixel.
Step S204: and extracting at least the region where the target object is located and the edge contour region of the scene region where the target object is located from the first depth image.
In step S204, an edge contour area of the area where the target object is located and the scene area where the target object is located is determined in the first depth image by using CANNY edge detection algorithm.
Step S205: determining a region where the target object is located and the scene region in the first depth image according to the edge contour region;
step S206: determining a first fusion coefficient for the depth information of the edge contour region, and determining a second fusion coefficient for the depth information of the region where the target object is and the scene region;
In the embodiment of the invention, the first fusion coefficient is greater than the second fusion coefficient, for example, the first fusion coefficient can be set to be 0.8, so that more pixels are selected for the edge contour area, and the second fusion coefficient is 0.2, so that smaller pixels are selected for the area where the target object is located, and the edge of the generated target depth map can be finer.
Step S207: generating a first screening coefficient according to the first confidence coefficient and the first fusion coefficient for each pixel point of the edge contour area, generating a second screening coefficient according to the first confidence coefficient and the second fusion coefficient for each pixel point of the area where the target object is located and the scene area, and generating a third screening coefficient according to the corresponding second confidence coefficient for each pixel point in the second depth image.
In the embodiment of the invention, a first screening coefficient is generated according to the first confidence coefficient, the first fusion coefficient and the second fusion coefficient so as to screen out pixels with higher precision in an edge contour area of the first depth image, and a second screening coefficient is generated according to the first confidence coefficient and the second fusion coefficient so as to screen out pixels with higher precision in an area where the target object is located and the scene area, thereby excluding depth information of pixels with lower precision and weaker influence in the depth image.
In the embodiment of the present invention, the first screening coefficient, the second screening coefficient and the third screening coefficient are all greater than 0 and less than 1; if the first confidence coefficient of the first region of the first depth image is 3, the first fusion coefficient is 0.7, if the first confidence coefficient of the first region of the first depth image is 1, the first fusion coefficient is 0.2, and the confidence value range is [0,5], the first filtering coefficient can take the value of 3/5×0.7=0.42; the second filter coefficient may take on a value of 1/5 x 0.2=0.04. When the second confidence is 4, then the third filter coefficient is 4/5=0.8.
In a modification of the present invention, a coefficient may be added to the first screening coefficient, the second screening coefficient, and the third screening coefficient, for example, 0.2 may be added to each of the first screening coefficient, the second screening coefficient, and one coefficient may be added to each of the first screening coefficient, the second screening coefficient, and the third screening coefficient, for example, 0.2 may be added to each of the first screening coefficient, the second screening coefficient, and 0.1 may be added to each of the first screening coefficient, the second screening coefficient, and the third screening coefficient.
In the embodiment of the invention, a plurality of pixels are screened out from two depth images with different pixel density respectively through the first confidence coefficient, the second confidence coefficient, the first fusion coefficient and the second fusion coefficient pair, so that a target depth image is generated, and a depth image with higher precision and finer edge can be obtained.
FIG. 6 is a schematic diagram of a 3D camera projecting lattice light to a target according to an embodiment of the present invention; fig. 7 is a schematic diagram of floodlight projected to a target by a 3D camera according to an embodiment of the present invention, as shown in fig. 6 and fig. 7, the first depth image is a depth image generated by floodlight projection to the target object, and the second depth image is a depth image generated by lattice light projection to the target object;
The first infrared image is an infrared image generated by floodlight projection to the target object, and the second infrared image is an infrared image generated by lattice light projection to the target object.
In the embodiment of the invention, the 3D camera projects lattice light and floodlight to the target for alternate switching between frames for projection. The 3D camera obtains the first depth image and the second depth image through a TOF depth sensor. The number of the beams of the lattice light is between several to several tens of thousands, preferably between 1 kilo to 10 tens of thousands.
The invention provides an image reconstruction system based on multiple depth maps, which is used for realizing the image reconstruction method based on the multiple depth maps, and comprises the following steps:
The image acquisition module is used for acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
the screening coefficient determining module is used for determining a screening coefficient for each pixel point in each depth image respectively;
The pixel point screening module is used for screening each pixel point in each depth image through the corresponding screening coefficient so as to generate a plurality of screened pixel points;
And the image fusion module is used for fusing a plurality of screened pixel points in the depth images to generate a target depth image.
The embodiment of the invention also provides image reconstruction equipment based on the multi-depth map, which comprises a processor. A memory having stored therein executable instructions of a processor. Wherein the processor is configured to perform the steps of the multi-depth map based image reconstruction method via execution of the executable instructions.
As described above, in this embodiment, a plurality of pixels can be screened out from two depth images with different pixel density through the first confidence coefficient, the second confidence coefficient, the first fusion coefficient and the second fusion coefficient pair, so as to generate a target depth image, and a depth image with higher precision and finer edge can be obtained.
Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" platform.
Fig. 8 is a schematic structural view of the image reconstruction apparatus based on multiple depth maps of the present invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 8. The electronic device 600 shown in fig. 8 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 8, the electronic device 600 is in the form of a general purpose computing device. Components of electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including memory unit 620 and processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code that is executable by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above-described multi-depth map based image reconstruction method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 6201 and/or cache memory unit 6202, and may further include Read Only Memory (ROM) 6203.
The storage unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 650. Also, electronic device 600 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in fig. 8, other hardware and/or software modules may be used in connection with electronic device 600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the steps of the image reconstruction method based on the multi-depth map are realized when the program is executed. In some possible embodiments, the aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the above description of the multi-depth map based image reconstruction method section, when the program product is run on the terminal device.
As described above, when the program of the computer readable storage medium of this embodiment is executed, the present invention can screen out a plurality of pixels from two depth images with different pixel density through the first confidence, the second confidence, the first fusion coefficient and the second fusion coefficient pair, so as to generate a target depth image, and obtain a depth image with higher precision and finer edges.
Fig. 9 is a schematic structural view of a computer-readable storage medium of the present invention. Referring to fig. 9, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In the embodiment, a plurality of pixels are screened out from two depth images with different pixel density respectively through a first confidence coefficient, a second confidence coefficient, a first fusion coefficient and a second fusion coefficient pair, so that a target depth image is generated, and a depth image with higher precision and finer edge can be obtained; according to the method, the confidence coefficient is determined through comparison of the infrared image and the depth image, and different fusion coefficients are set for the edge contour region and the region where the target object is located in the first depth image with higher pixel point density, so that depth information of pixels with poor precision and weak influence in the depth image can be removed, and the robustness of the model is improved; the method and the device can be suitable for the 3D camera with floodlight projection and lattice light projection, the 3D camera can improve the projection distance and the depth information precision under the conditions of low energy consumption and low cost, and the first depth image based on floodlight projection and the first depth image based on lattice light projection, which are obtained by the 3D camera through the TOF depth sensor, can be fused together to generate the depth image with higher precision and finer edge.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the claims without affecting the spirit of the invention.
Claims (8)
1. An image reconstruction method based on a multi-depth map is characterized by comprising the following steps:
step S1: acquiring a plurality of depth images of at least one target object, wherein the plurality of depth images have different pixel point densities;
step S2: determining a screening coefficient for each pixel point in each depth image;
step S3: screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points;
step S4: fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image;
step S5: acquiring the target depth image, and carrying out depth reconstruction on the target object according to the target depth image;
The step S2 includes the steps of:
step S201: extracting an area where the target object is located and an edge contour area of a scene area where the target object is located from each depth image;
Step S202: determining depth information of an area where the target object is located and the scene area in each depth image according to the edge contour area;
Step S203: respectively determining a screening coefficient for the edge contour area, the area where the target object is located and the depth information of the scene area; when the screening coefficients are determined, dividing each depth image into a plurality of areas, and respectively determining a screening coefficient for each area, wherein each pixel point in each area has a screening coefficient; the area with depth change is provided with high screening coefficient, the area without depth change is provided with low screening coefficient, and more pixels are selected for the edge contour area.
2. The multi-depth map-based image reconstruction method according to claim 1, wherein the plurality of depth images includes a plurality of depth images having different pixel densities generated by performing lattice light projections of different densities to the target object;
or the depth image comprises a depth image generated by floodlight projection to the target object and a plurality of depth images with different pixel point densities generated by different density lattice light projection to the target object.
3. An image reconstruction method based on a multi-depth map is characterized by comprising the following steps:
step S1: acquiring a plurality of depth images of at least one target object, wherein the plurality of depth images have different pixel point densities;
step S2: determining a screening coefficient for each pixel point in each depth image;
step S3: screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points;
step S4: fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image;
step S5: acquiring the target depth image, and carrying out depth reconstruction on the target object according to the target depth image;
The step S2 includes the steps of:
step S201: acquiring a plurality of infrared images which are in one-to-one correspondence with the depth images and are acquired under the same light projection environment with the corresponding depth images;
Step S202: comparing the infrared image with the corresponding depth image to determine the corresponding amplitude of each pixel point in the depth image;
Step S203: determining a screening coefficient for the depth information of each pixel point at least according to the amplitude corresponding to each pixel point in each depth image; determining a first confidence coefficient for the depth information of each pixel point in the corresponding depth image based on the amplitude of each region in the infrared image, when the amplitude of one region in the infrared image is high, assigning a high confidence coefficient for the pixel point in the same region corresponding to the depth image, and when the amplitude of one region in the infrared image is low, assigning a low confidence coefficient for the pixel point in the same region corresponding to the depth image.
4. An image reconstruction method based on a multi-depth map is characterized by comprising the following steps:
step S1: acquiring a plurality of depth images of at least one target object, wherein the plurality of depth images have different pixel point densities;
step S2: determining a screening coefficient for each pixel point in each depth image;
step S3: screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points;
step S4: fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image;
step S5: acquiring the target depth image, and carrying out depth reconstruction on the target object according to the target depth image;
when the plurality of depth images are a first depth image generated based on floodlight projection and a second depth image generated based on lattice light projection, the step S2 includes the steps of:
step S201: acquiring a first infrared image and a second infrared image, wherein the first infrared image is an infrared image acquired based on floodlight projection, and the second infrared image is an infrared image acquired based on lattice light projection;
step S202: comparing the first infrared image with the first depth image to determine the amplitude corresponding to each pixel point in the first depth image, and comparing the second infrared image with the second depth image to determine the amplitude corresponding to each pixel point in the second depth image;
Step S203: determining a first confidence coefficient for the depth information of each pixel point according to at least the amplitude corresponding to each pixel point in the first depth image, and determining a second confidence coefficient for the depth information of each pixel point according to at least the amplitude corresponding to each pixel point in the second depth image;
step S204: extracting at least the region where the target object is located and the edge contour region of the scene region where the target object is located from the first depth image;
step S205: determining a region where the target object is located and the scene region in the first depth image according to the edge contour region;
step S206: determining a first fusion coefficient for the depth information of the edge contour region, and determining a second fusion coefficient for the depth information of the region where the target object is and the scene region;
step S207: generating a first screening coefficient according to the first confidence coefficient and the first fusion coefficient for each pixel point of the edge contour area, generating a second screening coefficient according to the first confidence coefficient and the second fusion coefficient for each pixel point of the area where the target object is located and the scene area, and generating a third screening coefficient according to the corresponding second confidence coefficient for each pixel point in the second depth image.
5. The multi-depth map-based image reconstruction method according to claim 4, wherein the edge contour regions of the region in which the target object is located and the scene region in which the target object is located are determined in the first depth image by an edge detection algorithm in step S204.
6. A multi-depth map based image reconstruction system for implementing the multi-depth map based image reconstruction method of any one of claims 1 to 5, comprising:
The image acquisition module is used for acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
the screening coefficient determining module is used for determining a screening coefficient for each pixel point in each depth image respectively;
The pixel point screening module is used for screening each pixel point in each depth image through the corresponding screening coefficient so as to generate a plurality of screened pixel points;
And the image fusion module is used for fusing a plurality of screened pixel points in the depth images to generate a target depth image.
7. An image reconstruction apparatus based on multiple depth maps, comprising:
A processor;
A memory having stored therein executable instructions of the processor;
Wherein the processor is configured to perform the steps of the multi-depth map based image reconstruction method according to any one of claims 1 to 5 via execution of the executable instructions.
8. A computer-readable storage medium storing a program, wherein the program when executed implements the steps of the multi-depth map-based image reconstruction method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910799300.3A CN112446843B (en) | 2019-08-28 | 2019-08-28 | Image reconstruction method, system, equipment and medium based on multiple depth maps |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910799300.3A CN112446843B (en) | 2019-08-28 | 2019-08-28 | Image reconstruction method, system, equipment and medium based on multiple depth maps |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112446843A CN112446843A (en) | 2021-03-05 |
| CN112446843B true CN112446843B (en) | 2024-08-06 |
Family
ID=74741819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910799300.3A Active CN112446843B (en) | 2019-08-28 | 2019-08-28 | Image reconstruction method, system, equipment and medium based on multiple depth maps |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112446843B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113671466B (en) * | 2021-08-10 | 2024-05-31 | 南京大学 | SPAD array suitable for compressed sensing |
| CN113542534A (en) * | 2021-09-17 | 2021-10-22 | 珠海视熙科技有限公司 | TOF camera control method and device and storage medium |
| CN114972142A (en) * | 2022-05-13 | 2022-08-30 | 杭州汇萃智能科技有限公司 | Telecentric lens image synthesis method under condition of variable object distance |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105374019A (en) * | 2015-09-30 | 2016-03-02 | 华为技术有限公司 | A multi-depth image fusion method and device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2980292B1 (en) * | 2011-09-16 | 2013-10-11 | Prynel | METHOD AND SYSTEM FOR ACQUIRING AND PROCESSING IMAGES FOR MOTION DETECTION |
| CN104599314A (en) * | 2014-06-12 | 2015-05-06 | 深圳奥比中光科技有限公司 | Three-dimensional model reconstruction method and system |
| CN108616742B (en) * | 2016-12-21 | 2020-04-07 | 深圳市掌网科技股份有限公司 | 3D panoramic shooting system and method |
| CN107730445B (en) * | 2017-10-31 | 2022-02-18 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, storage medium, and electronic device |
| CN108830819B (en) * | 2018-05-23 | 2021-06-18 | 青柠优视科技（北京）有限公司 | Image fusion method and device for depth image and infrared image |
| CN109658433B (en) * | 2018-12-05 | 2020-08-28 | 青岛小鸟看看科技有限公司 | Image background modeling and foreground extracting method and device and electronic equipment |
-
2019
- 2019-08-28 CN CN201910799300.3A patent/CN112446843B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105374019A (en) * | 2015-09-30 | 2016-03-02 | 华为技术有限公司 | A multi-depth image fusion method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112446843A (en) | 2021-03-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11240485B2 (en) | Methods circuits devices assemblies systems and functionally associated with computer executable code for image acquisition with depth estimation | |
| CN109214980B (en) | Three-dimensional attitude estimation method, three-dimensional attitude estimation device, three-dimensional attitude estimation equipment and computer storage medium | |
| CN112446843B (en) | Image reconstruction method, system, equipment and medium based on multiple depth maps | |
| JP2016502704A (en) | Image processing method and apparatus for removing depth artifacts | |
| CN107592449B (en) | Three-dimensional model establishing method and device and mobile terminal | |
| CN111027407B (en) | Color image hand posture estimation method for shielding situation | |
| CN113378605B (en) | Multi-source information fusion method and device, electronic equipment and storage medium | |
| CN112651881A (en) | Image synthesis method, apparatus, device, storage medium, and program product | |
| CN112446842B (en) | Sparse and dense depth map-based image reconstruction method, system, equipment and medium | |
| You et al. | MISD‐SLAM: multimodal semantic SLAM for dynamic environments | |
| CN114186007A (en) | High-precision map generation method and device, electronic equipment and storage medium | |
| CN112509126A (en) | Method, device, equipment and storage medium for detecting three-dimensional object | |
| CN116844129A (en) | Road side target detection method, system and device for multi-mode feature alignment fusion | |
| Xu et al. | Robotic cross-platform sensor fusion and augmented visualization for large indoor space reality capture | |
| CN115761126A (en) | Three-dimensional reconstruction method and device based on structured light, electronic equipment and storage medium | |
| CN112824934A (en) | TOF multi-path interference removal method, system, equipment and medium based on modulated light field | |
| Yang et al. | MonoGAE: Roadside monocular 3D object detection with ground-aware embeddings | |
| CN109523570A (en) | Beginning parameter transform model method and device | |
| Jia et al. | Depth measurement based on a convolutional neural network and structured light | |
| KR20150003573A (en) | Method and apparatus for extracting pattern of image | |
| CN112085842B (en) | Depth value determining method and device, electronic equipment and storage medium | |
| CN116520348A (en) | Depth imaging system, method, equipment and medium based on modulated light field | |
| CN114882046B (en) | Panoramic segmentation method, device, equipment and medium for three-dimensional point cloud data | |
| KR20230006628A (en) | method and device for processing image, electronic equipment, storage medium and computer program | |
| US20220058779A1 (en) | Inpainting method and apparatus for human image, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |