CN112446843A - Image reconstruction method, system, device and medium based on multiple depth maps - Google Patents
Image reconstruction method, system, device and medium based on multiple depth maps Download PDFInfo
- Publication number
- CN112446843A CN112446843A CN201910799300.3A CN201910799300A CN112446843A CN 112446843 A CN112446843 A CN 112446843A CN 201910799300 A CN201910799300 A CN 201910799300A CN 112446843 A CN112446843 A CN 112446843A
- Authority
- CN
- China
- Prior art keywords
- depth
- image
- pixel point
- depth image
- target object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012216 screening Methods 0.000 claims abstract description 80
- 230000004927 fusion Effects 0.000 claims description 38
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000003708 edge detection Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 abstract description 2
- 238000002366 time-of-flight method Methods 0.000 description 19
- 238000010586 diagram Methods 0.000 description 10
- 238000012545 processing Methods 0.000 description 10
- 238000001914 filtration Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000005265 energy consumption Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000010146 3D printing Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/557—Depth or shape recovery from multiple images from light fields, e.g. from plenoptic cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Abstract
The invention provides an image reconstruction method, a system, equipment and a medium based on a multi-depth map, which comprises the following steps: acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities; determining a screening coefficient for each pixel point in each depth image respectively; screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points; and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image. According to the depth image processing method and device, a plurality of pixel points can be screened out from a plurality of depth images with different pixel point densities, so that the screened out pixel points are fused to generate a target depth image, and therefore the pixel points with higher precision in the plurality of depth images can be synthesized to generate a more precise depth image.
Description
Technical Field
The present invention relates to image fusion, and in particular, to a method, system, device, and medium for image reconstruction based on a multi-depth map.
Background
The imaging device using TOF technique is called TOF camera (or TOF camera), and TOF camera is similar to general machine vision imaging process, and comprises several units, such as light source, optical component, sensor (TOF chip), control circuit and processing circuit. TOF technology is basically similar to the principle of 3D laser sensors, except that the 3D laser sensor scans point by point, while the TOF camera obtains depth information of the whole image simultaneously.
The TOF technology has rich application scenes and is applied to a plurality of fields such as automobiles, industry, face recognition, logistics, pacifying monitoring, health, games, entertainment, movie special effects, 3D printing, robots and the like. In the automotive filed, the TOF sensor can be used for automatic driving, and the driving environment is sensed through the TOF technology, so that the environmental information is acquired to increase the safety, and the TOF can also be used for passenger off-position detection in the automobile. In the industrial field, TOF sensors may be used as HMI (Human Machine Interface), and in highly automated factories, workers and robots need to work in cooperation at a close distance, and TOF apparatuses may be used to control a safe distance in various situations. In a face recognition system, the brightness image and the depth information of a TOF camera can be connected through a model, and face matching and detection can be rapidly and accurately completed.
However, in the conventional TOF depth camera, a depth map of a target object is obtained by emitting a surface light source signal, receiving a returned light signal at a sensor end, and calculating the distance of the target object. However, due to signal energy constraint, imaging characteristics and limitations, the obtained depth map has the problems of poor precision, depth information loss and the like.
Disclosure of Invention
In view of the defects in the prior art, the present invention aims to provide a method, a system, a device and a medium for image reconstruction based on multiple depth maps.
The image reconstruction method based on the multi-depth map provided by the invention comprises the following steps:
step S1: acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
step S2: determining a screening coefficient for each pixel point in each depth image respectively;
step S3: screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points;
step S4: and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image.
Preferably, the method further comprises the following steps:
step S5: and acquiring the target depth image, and performing depth reconstruction on the target object according to the target depth image.
Preferably, the step S2 includes the steps of:
step S201: extracting the area where the target object is located and the edge contour area of the scene area where the target object is located in each depth image;
step S202: determining the depth information of the region where the target object is located and the scene region in each depth image according to the edge contour region;
step S203: and respectively determining a screening coefficient for the depth information of the edge contour region, the region where the target object is located and the scene region.
Preferably, the step S2 includes the steps of:
step S201: acquiring a plurality of infrared images which are in one-to-one correspondence with the depth images and are acquired under the same light projection environment with the corresponding depth images;
step S202: comparing the infrared image with the corresponding depth image to determine the corresponding amplitude of each pixel point in the depth image;
step S203: and determining a screening coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in each depth image.
Preferably, when a plurality of the depth images are a first depth image generated based on floodlight projection and a second depth image generated based on dot matrix light projection, the step S2 includes the steps of:
step S201: acquiring a first infrared image and a second infrared image, wherein the first infrared image is an infrared image acquired based on floodlight projection, and the second infrared image is an infrared image acquired based on dot matrix light projection;
step S202: comparing the first infrared image with the first depth image to determine the amplitude corresponding to each pixel point in the first depth image, and comparing the second infrared image with the second depth image to determine the amplitude corresponding to each pixel point in the second depth image;
step S203: determining a first confidence coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in the first depth image, and determining a second confidence coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in the second depth image;
step S204: extracting at least the area where the target object is located and the edge contour area of the scene area where the target object is located from the first depth image;
step S205: determining the region where the target object is located and the scene region in the first depth image according to the edge contour region;
step S206: determining a first fusion coefficient for the depth information of the edge contour region, and determining a second fusion coefficient for the depth information of the region where the target object is located and the scene region;
step S207: and generating a first screening coefficient for each pixel point of the edge contour region according to the first confidence degree and the first fusion coefficient, generating a second screening coefficient for each pixel point of the region where the target object is located and the scene region according to the first confidence degree and the second fusion coefficient, and generating a third screening coefficient for each pixel point in the second depth image according to the corresponding second confidence degree.
Preferably, the plurality of depth images include a plurality of depth images with different pixel point densities generated by performing dot matrix light projection with different densities on the target object;
or the depth image comprises a depth image generated by floodlight projection to the target object and another depth image generated by different density lattice light projection to the target object and having different pixel point densities.
Preferably, in step S204, an edge detection algorithm is used to determine the area where the target object is located and an edge contour area of the scene area where the target object is located in the first depth image.
The image reconstruction system based on the multiple depth maps is used for realizing the image reconstruction method based on the multiple depth maps, and comprises the following steps:
the image acquisition module is used for acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
the screening coefficient determining module is used for determining a screening coefficient for each pixel point in each depth image;
the pixel point screening module is used for screening each pixel point in each depth image through the corresponding screening coefficient so as to generate a plurality of screened pixel points;
and the image fusion module is used for fusing a plurality of screened pixel points in the depth images to generate a target depth image.
The invention provides an image reconstruction device based on a multi-depth map, which comprises:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the multi-depth map based image reconstruction method via execution of the executable instructions.
The present invention provides a computer readable storage medium for storing a program which, when executed, implements the steps of the multi-depth map based image reconstruction method.
Compared with the prior art, the invention has the following beneficial effects:
1. according to the depth image processing method, a plurality of pixel points can be screened out from a plurality of depth images with different pixel point densities, so that the screened out pixel points are fused to generate a target depth image, and therefore the pixel points with higher precision in the plurality of depth images can be synthesized to generate a finer depth image;
2. according to the method, the screening coefficients are respectively determined for the edge contour area, the area where the target object is located and the scene area, so that the display precision of a key area in the depth map can be improved, the key area of the depth map is clearer, and the edge of the target object is clearer;
3. according to the method, the confidence coefficient is determined by comparing the infrared image with the depth image, and different fusion coefficients are set for the edge contour region in the first depth image with higher pixel point density and the region where the target object is located, so that the depth information of the pixel points with poorer precision and weaker influence in the depth image can be removed, and the robustness of the model is improved;
4. the method is applicable to the 3D camera with floodlight projection and dot matrix light projection, the 3D camera can improve the projection distance and the depth information precision under the conditions of low energy consumption and low cost, and the first depth image based on floodlight projection and the second depth image based on dot matrix light projection, which are obtained by the 3D camera through the TOF depth sensor, can be fused together to generate the depth map with higher precision and finer edge.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts. Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flowchart illustrating steps of a multi-depth-map-based image reconstruction method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating the steps of a method for reconstructing an image based on a multi-depth map according to a variation of the present invention;
FIG. 3 is a flowchart illustrating the steps of determining a filter coefficient by performing region segmentation on a depth image according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating the steps of determining the filtering coefficients of each pixel point according to the amplitude in the embodiment of the present invention;
FIG. 5 is a flowchart illustrating the steps for determining a screening coefficient by integrating confidence and fusion coefficients in an embodiment of the present invention;
FIG. 6 is a schematic diagram of a 3D camera projecting a lattice light to a target according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a 3D camera projecting floodlight towards a target according to an embodiment of the present invention;
FIG. 8 is a block diagram of a multi-depth map based image reconstruction system according to the present invention;
FIG. 9 is a schematic structural diagram of an image reconstruction apparatus based on multiple depth maps according to the present invention; and
fig. 10 is a schematic structural diagram of a computer-readable storage medium according to the present invention.
In the figure:
1 is a dot matrix projection module;
2 is the spatial distribution of the lattice light;
3 is a floodlight projection module;
and 4, the space distribution of floodlight.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical solution of the present invention will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
The invention provides an image reconstruction method based on a multi-depth map, and aims to solve the problems in the prior art.
The following describes the technical solutions of the present invention and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
The image reconstruction method based on the multi-depth map provided by the embodiment of the invention can be suitable for a 3D camera with floodlight projection and dot matrix light projection, the 3D camera can improve the projection distance and the depth information precision under the conditions of low energy consumption and low cost, and the depth image based on floodlight projection and the depth image based on dot matrix light projection, which are obtained by the 3D camera through a TOF depth sensor, can be fused together to generate the depth map with higher precision and finer edge.
Fig. 1 is a flowchart illustrating steps of an image reconstruction method based on a multi-depth map according to an embodiment of the present invention, and as shown in fig. 1, the image reconstruction method based on a multi-depth map according to the present invention includes the following steps:
step S1: the method comprises the steps of obtaining a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities.
In practical application, the execution main body of the embodiment of the invention can be a computer, a central processing unit, a Micro Control Unit (MCU), a single chip microcomputer, a microprocessor and other devices with data processing capability. In this embodiment, a cpu is taken as an example for detailed description, but the specific device type for executing the method in this embodiment is not limited.
In the embodiment of the invention, a plurality of dot matrix lights with different pixel point densities can be generated by projecting dot matrix lights with different densities to a target object. The lattice light may be arranged periodically, such as in a periodic pattern of triangles, quadrilaterals, rectangles, circles, hexagons, and pentagons, or non-periodically. The non-periodic arrangement is, for example, a random arrangement, a spatially coded arrangement, a quasi-lattice arrangement, etc.
Step S2: and respectively determining a screening coefficient for each pixel point in each depth image.
In the embodiment of the present invention, when a screening coefficient is determined for each pixel point in each depth image, each depth image may be divided into a plurality of regions, and a screening coefficient is determined for each region, so that each pixel point in the region has a screening coefficient.
When the area division is performed in the depth image, a higher screening coefficient, for example, 70% to 90%, may be set for an edge area, a curved area, a slope area, a convex area, a concave area, and the like of the entire image or the object having a depth variation. A lower screening factor, e.g. 10% to 50%, is set for areas without depth variation, e.g. flat areas.
In the modified example of the present invention, a lower screening coefficient may be set for a depth image with a higher pixel density, and for example, the screening coefficient is set between 10% and 50% in a depth image generated based on flood projection. If a higher screening coefficient is set for a depth image with a lower pixel density, for example, the number of pixels occupies 10% of the photodetector array, the screening coefficient is set to be, for example, 70% to 90%. When the number of the light beams of the dot matrix light is small, the precision of each pixel point is higher, a higher screening coefficient is set at the moment, more pixel points are selected, and therefore the generated target depth image is higher in precision.
Step S3: and screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points.
In the embodiment of the present invention, when the pixel points are screened by the screening coefficient, the screening is performed by the random probability, that is, when the screening coefficient of a certain region is 90%, the probability that each pixel point of the region is selected is 90%.
Step S4: and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image.
In the embodiment of the invention, a plurality of pixel points can be screened from a plurality of depth images with different pixel point densities, so that the screened pixel points are fused to generate a target depth image, and a more precise depth image can be generated by synthesizing the pixel points with higher precision in the depth images.
In the modified example of the present invention, a plurality of pixel points selected from a plurality of depth images are fused to generate a target depth image, specifically, the confidence of each pixel point at a corresponding position of the plurality of depth images is compared, and a depth value corresponding to the maximum confidence is selected, that is, the confidence of the plurality of pixel points selected from each depth image is compared according to the position of the light detector matrix, and the depth value corresponding to the maximum confidence is selected for fusion.
Fig. 2 is a flowchart of steps of an image reconstruction method based on a multi-depth map according to a variation of the present invention, and as shown in fig. 2, the image reconstruction method based on a multi-depth map according to the present invention further includes the following steps:
step S5: and acquiring the target depth image, and performing depth reconstruction on the target object according to the target depth image.
In the modification of the present invention, the target object may be deeply reconstructed by using an algorithm such as filtering interpolation, the target object may be deeply reconstructed by using a Markov Random field Model (MRF), or the target object may be deeply reconstructed by using a Compressed Sensing (CS) algorithm.
Fig. 3 is a flowchart of a step of determining a filtering coefficient by performing region division on a depth image according to an embodiment of the present invention, and as shown in fig. 3, the step S2 includes the following steps:
step S201: extracting the area where the target object is located and the edge contour area of the scene area where the target object is located in each depth image;
step S202: determining the depth information of the region where the target object is located and the scene region in each depth image according to the edge contour region;
step S203: and respectively determining a screening coefficient for the depth information of the edge contour region, the region where the target object is located and the scene region.
In the embodiment of the present invention, the screening coefficient of the edge contour region is greater than the screening coefficients of the region where the target object is located and the scene region, for example, the screening coefficient of the edge contour region may be set to 0.8, so as to select more pixel points for the edge contour region; the screening coefficient of the region where the target object is located and the screening coefficient of the scene region are 0.2, so that smaller pixel points are selected for the region where the target object is located and the scene region, and the edge of the generated target depth map can be finer.
Fig. 4 is a flowchart of a step of determining a filtering coefficient of each pixel point according to the amplitude in the embodiment of the present invention, and as shown in fig. 4, the step S2 includes the following steps:
step S201: and acquiring a plurality of infrared images which are in one-to-one correspondence with the depth images and are acquired under the same light projection environment with the corresponding depth images.
In the embodiment of the present invention, the same light projection environment is specifically that both the infrared image and the depth image are collected under flood projection, or the infrared image and the depth image are collected under lattice light with the same light beam number and light beam arrangement structure.
Step S202: and comparing the infrared image with the corresponding depth image to determine the corresponding amplitude of each pixel point in the depth image.
Step S203: and determining a screening coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in each depth image.
In the embodiment of the invention, a first confidence coefficient is determined based on the depth information of each pixel point in the depth image corresponding to the amplitude of each region in the infrared image, when the amplitude of one region in the infrared image is higher, a higher confidence coefficient is given to the pixel point in the same region corresponding to the depth image, and when the amplitude of one region in the infrared image is lower, a lower confidence coefficient is given to the pixel point in the same region corresponding to the depth image.
More specifically, for example, when the amplitude of an area in the infrared image is 300W/m2Giving confidence coefficient of 3 to the pixel points in the same region corresponding to the depth image; when the amplitude of an area in the infrared image is 100W/m2And if so, giving confidence 1 to the pixel points in the same region corresponding to the depth image.
In the modified example of the present invention, the amplitude of a region in the infrared image can also be represented by temperature, and when the amplitude of a region in the infrared image is 30 ℃, a confidence of 3 is given to the pixel point in the same region corresponding to the depth image; and when the amplitude of one region in the infrared image is 10 ℃, giving a confidence coefficient of 1 to the pixel point in the same region corresponding to the depth image.
Fig. 5 is a flowchart illustrating a step of determining a filtering coefficient by combining confidence and fusion coefficients according to an embodiment of the present invention, where, as shown in fig. 5, when a plurality of depth images are a first depth image generated based on floodlight projection and a second depth image generated based on lattice light projection, the step S2 includes the following steps:
step S201: the method comprises the steps of obtaining a first infrared image and a second infrared image, wherein the first infrared image is an infrared image collected based on floodlight projection, and the second infrared image is an infrared image collected based on dot matrix light projection.
In the embodiment of the present invention, the first depth image corresponds to the first infrared image, that is, the first depth image and the first infrared image are collected under the same type of light; the second depth image corresponds to the second infrared image, that is, the second depth image and the second infrared image are collected under the same type of light.
Step S202: comparing the first infrared image with the first depth image to determine the amplitude corresponding to each pixel point in the first depth image, and comparing the second infrared image with the second depth image to determine the amplitude corresponding to each pixel point in the second depth image.
In the embodiment of the invention, because the infrared image senses and reflects the difference of the outward amplitudes of the target and the background, the amplitude of each area in the first infrared image and the second infrared image can be determined;
because the first depth image and the first infrared image are imaging carried out on the same target object, each pixel point on the first depth image corresponds to one area of the first infrared image, so that the amplitude corresponding to each pixel point on the first depth image can be determined, and similarly, the amplitude corresponding to each pixel point can be determined for the second depth image.
Step S203: determining a first confidence coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in the first depth image, and determining a second confidence coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in the second depth image;
in the embodiment of the present invention, a first confidence is determined for the depth information of each pixel point in the first depth image based on the amplitude of each region in the first infrared image, when the amplitude of one region in the first infrared image is higher, a higher confidence is given to the pixel point in the same region corresponding to the first depth image, and when the amplitude of one region in the first infrared image is lower, a lower confidence is given to the pixel point in the same region corresponding to the first depth image. Similarly, for the second depth image, when the amplitude of a region in the second infrared image is higher, a higher confidence is given to the pixel points in the same region corresponding to the second depth image, and when the amplitude of a region in the second infrared image is lower, a lower confidence is given to the pixel points in the same region corresponding to the second depth image.
More specifically, if the amplitude of a region in the first infrared image is 300, the confidence coefficient of a pixel point in the same region corresponding to the first depth image is 3; and when the amplitude of one region in the first infrared image is 100, giving a confidence coefficient of 1 to the pixel point in the same region corresponding to the first depth image.
In the modified example of the present invention, the amplitude of a region in the first infrared image may also be represented by temperature, and when the amplitude of a region in the first infrared image is 30 ℃, a confidence of 3 is given to a pixel point in the same region corresponding to the first depth image; and when the amplitude of one region in the first infrared image is 10 ℃, giving a confidence coefficient of 1 to the pixel point in the same region corresponding to the first depth image. The amplitude for each region in the second infrared image is expressed in terms of temperature in the same way.
In the modification of the present invention, the value of the depth information of each pixel point may be comprehensively considered to determine the first confidence.
Step S204: and at least extracting the area where the target object is located and the edge contour area of the scene area where the target object is located from the first depth image.
In step S204, determining the area where the target object is located and the edge contour area of the scene area where the target object is located in the first depth image by using a CANNY edge detection algorithm.
Step S205: determining the region where the target object is located and the scene region in the first depth image according to the edge contour region;
step S206: determining a first fusion coefficient for the depth information of the edge contour region, and determining a second fusion coefficient for the depth information of the region where the target object is located and the scene region;
in the embodiment of the present invention, the first fusion coefficient is greater than the second fusion system, and if the first fusion coefficient is set to be 0.8, more pixel points are selected for the edge contour region, and the second fusion coefficient is 0.2, so that smaller pixel points are selected for the region where the target object is located, so that the edge of the generated target depth map can be made finer.
Step S207: and generating a first screening coefficient for each pixel point of the edge contour region according to the first confidence degree and the first fusion coefficient, generating a second screening coefficient for each pixel point of the region where the target object is located and the scene region according to the first confidence degree and the second fusion coefficient, and generating a third screening coefficient for each pixel point in the second depth image according to the corresponding second confidence degree.
In the embodiment of the invention, a first screening coefficient is generated according to the first confidence coefficient, the first fusion coefficient and the second fusion coefficient so as to screen out pixel points with higher precision in the edge contour region of the first depth image, and a second screening coefficient is generated according to the first confidence coefficient and the second fusion coefficient, so that pixel points with higher precision are screened out in the region where the target object is located and the scene region, and the depth information of the pixel points with poorer precision and weaker influence in the depth image is eliminated.
In the embodiment of the present invention, the first screening coefficient, the second screening coefficient, and the third screening coefficient are all greater than 0 and less than 1; if the first confidence coefficient of the first depth image first region is 3 and the first fusion coefficient is 0.7, and if the first confidence coefficient of the first depth image first region is 1 and the first fusion coefficient is 0.2, and the confidence coefficient is in the range of [0,5], the first filtering coefficient may be 3/5 × 0.7 — 0.42; the second screening coefficient may take the value of 1/5 × 0.2 — 0.04. When the second confidence is 4, the third screening coefficient is 4/5 ═ 0.8.
In the modification of the present invention, a first screening coefficient, for example, 0.2 may be added to the first screening coefficient, the second screening coefficient, and the third screening coefficient, and then screening may be performed, or a first screening coefficient, a second screening coefficient, and a third screening coefficient, for example, 0.2 may be added to the first screening coefficient and the second screening coefficient, and 0.1 may be added to the first screening coefficient.
In the embodiment of the invention, a plurality of pixel points are screened out in two depth images with different pixel point densities respectively through the first confidence degree, the second confidence degree, the first fusion coefficient and the second fusion coefficient pair, and then a target depth image is generated, so that a depth image with higher precision and finer edge can be obtained.
FIG. 6 is a schematic diagram of a 3D camera projecting a lattice light to a target according to an embodiment of the present invention; fig. 7 is a schematic diagram of a 3D camera projecting floodlight to a target in an embodiment of the present invention, as shown in fig. 6 and 7, the first depth image is a depth image generated by performing floodlight projection to the target object, and the second depth image is a depth image generated by performing lattice light projection to the target object;
the first infrared image is an infrared image generated by floodlight projection on the target object, and the second infrared image is an infrared image generated by dot matrix light projection on the target object.
In the embodiment of the invention, the 3D camera projects dot light and floodlight to the target by switching between frames alternately. The 3D camera obtains the first depth image and the second depth image through a TOF depth sensor. The number of the light beams of the lattice light is between several beams and tens of thousands of beams, and preferably, the number of the light beams of the lattice light is between 1 thousand beams and 10 thousands of beams.
Fig. 8 is a schematic block diagram of an image reconstruction system based on a multi-depth map in the present invention, and as shown in fig. 8, the image reconstruction system based on a multi-depth map provided in the present invention is configured to implement the image reconstruction method based on a multi-depth map, and includes:
the image acquisition module is used for acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
the screening coefficient determining module is used for determining a screening coefficient for each pixel point in each depth image;
the pixel point screening module is used for screening each pixel point in each depth image through the corresponding screening coefficient so as to generate a plurality of screened pixel points;
and the image fusion module is used for fusing a plurality of screened pixel points in the depth images to generate a target depth image.
The embodiment of the invention also provides image reconstruction equipment based on the multi-depth map, which comprises a processor. A memory having stored therein executable instructions of the processor. Wherein the processor is configured to perform the steps of the multi-depth map based image reconstruction method via execution of executable instructions.
As described above, in this embodiment, a plurality of pixel points can be screened out through the first confidence, the second confidence, the first fusion coefficient and the second fusion coefficient pair in two depth images with different pixel point densities, so as to generate a target depth image, and a depth map with higher precision and finer edge can be obtained.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
Fig. 8 is a schematic structural diagram of the image reconstruction apparatus based on multiple depth maps according to the present invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 8. The electronic device 600 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code which can be executed by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention as described in the above-mentioned multi-depth map based image reconstruction method section of the present specification. For example, processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in FIG. 8, other hardware and/or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program realizes the steps of the image reconstruction method based on the multi-depth map when being executed. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the section of the description above for a multi-depth map based image reconstruction method when the program product is run on the terminal device.
As described above, when the program of the computer-readable storage medium of this embodiment is executed, the present invention can screen out a plurality of pixel points through the first confidence, the second confidence, the first fusion coefficient, and the second fusion coefficient pair in two depth images with different pixel point densities, and further generate a target depth image, so as to obtain a depth image with higher precision and finer edge.
Fig. 9 is a schematic structural diagram of a computer-readable storage medium of the present invention. Referring to fig. 9, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In this embodiment, in two depth images with different pixel density, a plurality of pixels are screened out through the first confidence, the second confidence, the first fusion coefficient and the second fusion coefficient pair, so as to generate a target depth image, and a depth image with higher precision and finer edge can be obtained; according to the method, the confidence coefficient is determined by comparing the infrared image with the depth image, and different fusion coefficients are set for the edge contour region in the first depth image with higher pixel point density and the region where the target object is located, so that the depth information of the pixel points with poorer precision and weaker influence in the depth image can be removed, and the robustness of the model is improved; the method can be suitable for the 3D camera with floodlight projection and dot matrix light projection, the 3D camera can improve the projection distance and the depth information precision under the conditions of low energy consumption and low cost, and the first depth image based on floodlight projection and the first depth image based on dot matrix light projection, which are obtained by the 3D camera through the TOF depth sensor, can be fused together to generate the depth map with higher precision and finer edge.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.
Claims (10)
1. An image reconstruction method based on multiple depth maps is characterized by comprising the following steps:
step S1: acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
step S2: determining a screening coefficient for each pixel point in each depth image respectively;
step S3: screening each pixel point in each depth image through the corresponding screening coefficient to generate a plurality of screened pixel points;
step S4: and fusing a plurality of pixel points screened out from the plurality of depth images to generate a target depth image.
2. The multi-depth map-based image reconstruction method according to claim 1, further comprising the steps of:
step S5: and acquiring the target depth image, and performing depth reconstruction on the target object according to the target depth image.
3. The multi-depth-map-based image reconstruction method according to claim 1, wherein the step S2 comprises the steps of:
step S201: extracting the area where the target object is located and the edge contour area of the scene area where the target object is located in each depth image;
step S202: determining the depth information of the region where the target object is located and the scene region in each depth image according to the edge contour region;
step S203: and respectively determining a screening coefficient for the depth information of the edge contour region, the region where the target object is located and the scene region.
4. The multi-depth-map-based image reconstruction method according to claim 1, wherein the step S2 comprises the steps of:
step S201: acquiring a plurality of infrared images which are in one-to-one correspondence with the depth images and are acquired under the same light projection environment with the corresponding depth images;
step S202: comparing the infrared image with the corresponding depth image to determine the corresponding amplitude of each pixel point in the depth image;
step S203: and determining a screening coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in each depth image.
5. The multi-depth-map-based image reconstruction method of claim 1, wherein when the plurality of depth images are a first depth image generated based on floodlight projection and a second depth image generated based on lattice light projection, the step S2 comprises the steps of:
step S201: acquiring a first infrared image and a second infrared image, wherein the first infrared image is an infrared image acquired based on floodlight projection, and the second infrared image is an infrared image acquired based on dot matrix light projection;
step S202: comparing the first infrared image with the first depth image to determine the amplitude corresponding to each pixel point in the first depth image, and comparing the second infrared image with the second depth image to determine the amplitude corresponding to each pixel point in the second depth image;
step S203: determining a first confidence coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in the first depth image, and determining a second confidence coefficient for the depth information of each pixel point at least according to the magnitude of the amplitude corresponding to each pixel point in the second depth image;
step S204: extracting at least the area where the target object is located and the edge contour area of the scene area where the target object is located from the first depth image;
step S205: determining the region where the target object is located and the scene region in the first depth image according to the edge contour region;
step S206: determining a first fusion coefficient for the depth information of the edge contour region, and determining a second fusion coefficient for the depth information of the region where the target object is located and the scene region;
step S207: and generating a first screening coefficient for each pixel point of the edge contour region according to the first confidence degree and the first fusion coefficient, generating a second screening coefficient for each pixel point of the region where the target object is located and the scene region according to the first confidence degree and the second fusion coefficient, and generating a third screening coefficient for each pixel point in the second depth image according to the corresponding second confidence degree.
6. The multi-depth-map-based image reconstruction method according to claim 3, wherein the plurality of depth images include a plurality of depth images with different pixel densities generated by performing dot matrix light projection with different densities on the target object;
or the depth image comprises a depth image generated by floodlight projection to the target object and another depth image generated by different density lattice light projection to the target object and having different pixel point densities.
7. The method for image reconstruction based on multi-depth map of claim 5, wherein in step S204, the area of the target object and the edge contour area of the scene area where the area of the target object is located are determined in the first depth image by an edge detection algorithm.
8. A multi-depth map-based image reconstruction system for implementing the multi-depth map-based image reconstruction method according to any one of claims 1 to 7, comprising:
the image acquisition module is used for acquiring a plurality of depth images of at least one target object, wherein the depth images have different pixel point densities;
the screening coefficient determining module is used for determining a screening coefficient for each pixel point in each depth image;
the pixel point screening module is used for screening each pixel point in each depth image through the corresponding screening coefficient so as to generate a plurality of screened pixel points;
and the image fusion module is used for fusing a plurality of screened pixel points in the depth images to generate a target depth image.
9. An image reconstruction apparatus based on a multi-depth map, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the multi-depth map based image reconstruction method of any one of claims 1 to 7 via execution of the executable instructions.
10. A computer-readable storage medium storing a program, wherein the program is configured to implement the steps of the multi-depth map-based image reconstruction method according to any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910799300.3A CN112446843A (en) | 2019-08-28 | 2019-08-28 | Image reconstruction method, system, device and medium based on multiple depth maps |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910799300.3A CN112446843A (en) | 2019-08-28 | 2019-08-28 | Image reconstruction method, system, device and medium based on multiple depth maps |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112446843A true CN112446843A (en) | 2021-03-05 |
Family
ID=74741819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910799300.3A Pending CN112446843A (en) | 2019-08-28 | 2019-08-28 | Image reconstruction method, system, device and medium based on multiple depth maps |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112446843A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113542534A (en) * | 2021-09-17 | 2021-10-22 | 珠海视熙科技有限公司 | TOF camera control method and device and storage medium |
| CN113671466A (en) * | 2021-08-10 | 2021-11-19 | 南京大学 | SPAD array suitable for compressed sensing |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013038089A1 (en) * | 2011-09-16 | 2013-03-21 | Prynel | Method and system for acquiring and processing images for the detection of motion |
| CN104599314A (en) * | 2014-06-12 | 2015-05-06 | 深圳奥比中光科技有限公司 | Three-dimensional model reconstruction method and system |
| CN105374019A (en) * | 2015-09-30 | 2016-03-02 | 华为技术有限公司 | A multi-depth image fusion method and device |
| CN107730445A (en) * | 2017-10-31 | 2018-02-23 | 广东欧珀移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
| CN108616742A (en) * | 2016-12-21 | 2018-10-02 | 深圳市掌网科技股份有限公司 | A kind of 3D panoramic shooting systems and method |
| CN108830819A (en) * | 2018-05-23 | 2018-11-16 | 青柠优视科技（北京）有限公司 | A kind of image interfusion method and device of depth image and infrared image |
| CN109658433A (en) * | 2018-12-05 | 2019-04-19 | 青岛小鸟看看科技有限公司 | Image background modeling and foreground extracting method, device and electronic equipment |
-
2019
- 2019-08-28 CN CN201910799300.3A patent/CN112446843A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013038089A1 (en) * | 2011-09-16 | 2013-03-21 | Prynel | Method and system for acquiring and processing images for the detection of motion |
| CN104599314A (en) * | 2014-06-12 | 2015-05-06 | 深圳奥比中光科技有限公司 | Three-dimensional model reconstruction method and system |
| CN105374019A (en) * | 2015-09-30 | 2016-03-02 | 华为技术有限公司 | A multi-depth image fusion method and device |
| CN108616742A (en) * | 2016-12-21 | 2018-10-02 | 深圳市掌网科技股份有限公司 | A kind of 3D panoramic shooting systems and method |
| CN107730445A (en) * | 2017-10-31 | 2018-02-23 | 广东欧珀移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
| CN108830819A (en) * | 2018-05-23 | 2018-11-16 | 青柠优视科技（北京）有限公司 | A kind of image interfusion method and device of depth image and infrared image |
| CN109658433A (en) * | 2018-12-05 | 2019-04-19 | 青岛小鸟看看科技有限公司 | Image background modeling and foreground extracting method, device and electronic equipment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113671466A (en) * | 2021-08-10 | 2021-11-19 | 南京大学 | SPAD array suitable for compressed sensing |
| CN113542534A (en) * | 2021-09-17 | 2021-10-22 | 珠海视熙科技有限公司 | TOF camera control method and device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Romero-Ramirez et al. | Speeded up detection of squared fiducial markers | |
| US11216971B2 (en) | Three-dimensional bounding box from two-dimensional image and point cloud data | |
| US11240485B2 (en) | Methods circuits devices assemblies systems and functionally associated with computer executable code for image acquisition with depth estimation | |
| US20140240467A1 (en) | Image processing method and apparatus for elimination of depth artifacts | |
| CN108352074A (en) | Quasi- parameter optical flow estimation | |
| JP2015079505A (en) | Noise identification method and noise identification device of parallax depth image | |
| CN107592449B (en) | Three-dimensional model establishing method and device and mobile terminal | |
| CN110853085B (en) | Semantic SLAM-based mapping method and device and electronic equipment | |
| CN112446843A (en) | Image reconstruction method, system, device and medium based on multiple depth maps | |
| CN112446842A (en) | Image reconstruction method, system, device and medium based on sparse and dense depth maps | |
| CN112824934A (en) | TOF multi-path interference removal method, system, equipment and medium based on modulated light field | |
| CN104483712A (en) | Method, device and system for detecting invasion of foreign objects in power transmission line | |
| CN112824935A (en) | Depth imaging system, method, device and medium based on modulated light field | |
| CN112509126A (en) | Method, device, equipment and storage medium for detecting three-dimensional object | |
| CN115861601A (en) | Multi-sensor fusion sensing method and device | |
| US9842402B1 (en) | Detecting foreground regions in panoramic video frames | |
| CN114697521A (en) | TOF camera motion blur detection method, system, equipment and storage medium | |
| CN115294234B (en) | Image generation method and device, electronic equipment and storage medium | |
| CN113762001B (en) | Target detection method and device, electronic equipment and storage medium | |
| CN115661577B (en) | Method, apparatus and computer readable storage medium for object detection | |
| CN112435293B (en) | Method and device for determining structural parameter representation of lane line | |
| Ghaneizad et al. | A fast bottom-up approach toward three-dimensional human pose estimation using an array of cameras | |
| US20220301127A1 (en) | Image processing pipeline for optimizing images in machine learning and other applications | |
| CN114724097A (en) | Positioning image processing method, medium, equipment and system based on video compression | |
| Peng | Enhancing Depth Estimation in Adverse Lighting Scenarios for Autonomous Driving |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |