CN110349196A - The method and apparatus of depth integration - Google Patents
The method and apparatus of depth integration Download PDFInfo
- Publication number
- CN110349196A CN110349196A CN201910261747.5A CN201910261747A CN110349196A CN 110349196 A CN110349196 A CN 110349196A CN 201910261747 A CN201910261747 A CN 201910261747A CN 110349196 A CN110349196 A CN 110349196A
- Authority
- CN
- China
- Prior art keywords
- depth
- map
- depth map
- fusion
- related information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 136
- 230000010354 integration Effects 0.000 title claims abstract description 37
- 230000004927 fusion Effects 0.000 claims abstract description 72
- 238000007500 overflow downdraw method Methods 0.000 claims description 38
- 238000002366 time-of-flight method Methods 0.000 claims description 29
- 238000012805 post-processing Methods 0.000 claims description 9
- 238000013459 approach Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 5
- 230000005611 electricity Effects 0.000 claims description 2
- 239000007787 solid Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 description 74
- 230000006870 function Effects 0.000 description 39
- 238000000059 patterning Methods 0.000 description 22
- 238000005516 engineering process Methods 0.000 description 10
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000004438 eyesight Effects 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 2
- 239000011469 building brick Substances 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000000429 assembly Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 239000003990 capacitor Substances 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005034 decoration Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000002844 melting Methods 0.000 description 1
- 230000008018 melting Effects 0.000 description 1
- 239000003607 modifier Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Abstract
The disclosure discloses a kind of depth integration method and relevant apparatus.Depth integration method includes receiving multiple different types of sensor signals from multiple sensors；The first depth-related information of scene and the second depth-related information of scene are generated based on multiple sensor signals；And the first depth-related information and the second depth-related information are merged to generate the fusion depth map of scene.Depth integration method of the invention and relevant apparatus can reduce cost.
Description
[cross reference]
The disclosure requires the U.S. Patent application 62/651,813 and 2019 year submitted on April 3rd, 2018 to mention March 20
The priority of the U.S. Patent application 16/359,713 of friendship, entire contents are also incorporated in this together.
[technical field]
The present disclosure relates generally to computer stereo visions, and more particularly, to by one or more accurate and complete
Range depth merges (full-range depth fusion) and detection technology carries out space or depth perception sensing.
[background technique]
Unless otherwise indicated herein, otherwise method described in this section is not showing for scope of the claims listed below
There is technology, and is not included in this section as the prior art.
There are many technologies for range sensing and estimation of Depth at present to realize computer stereo vision.For example, some
The prior art include structure light, passive stereo matching (passive stereo), active stereo matching (active stereo) and
Flight time (time-of-flight).But some of which technology can be combined without a kind of technology.In addition, using
Infrared (IR) projector and IR camera carry out in the application of depth sense, and the component used is usually costly.It is expected that by use
Ready-made and relatively cheap component provides the solution for the computer stereo vision for allowing to have Stereo matching.
[summary of the invention]
According to the first aspect of the invention, a kind of depth integration method is disclosed, including multiple from the reception of multiple sensors
Different types of sensor signal；The first depth-related information and the scene of scene are generated based on multiple sensor signal
Second depth-related information；And first depth-related information and second depth-related information are merged to generate the scene
Merge depth map.
According to the second aspect of the invention, a kind of depth integration device, including control circuit are disclosed, be coupled to from
Multiple sensors receive multiple different types of sensor signals, so that during operation, it includes following which, which executes,
The operation of operation: the first depth-related information of scene and the second depth phase of the scene are generated based on multiple sensor signal
Close information；And first depth-related information and second depth-related information are merged to generate the fusion depth of the scene
Figure.
After having read the following detailed description book of the illustrated preferred embodiment in various charts and figure, this hair
These and other bright purposes undoubtedly will be apparent for those of ordinary skills.
[Detailed description of the invention]
Fig. 1 shows the exemplary scene realized according to the disclosure.
Fig. 2 shows the active stereo matched exemplary scenes according to the realization of the disclosure.
Fig. 3 shows the exemplary scene of the realization according to the disclosure.
Fig. 4 shows the exemplary scene of the realization according to the disclosure.
Fig. 5 shows the exemplary scene of the realization according to the disclosure.
Fig. 6 shows the exemplary scene of the depth integration of the realization according to the disclosure.
Fig. 7 shows the example fusion method according to the realization of the disclosure.
Fig. 8 shows the example fusion method according to the realization of the disclosure.
Fig. 9 shows the exemplary device of the realization according to the disclosure.
Figure 10 shows the instantiation procedure of the realization according to the disclosure.
Figure 11 shows the instantiation procedure of the realization according to the disclosure.
Figure 12 shows the instantiation procedure of the realization according to the disclosure.
[specific embodiment]
Certain terms have been used through following description and claims, have been related to specific element.As belonging to
As field tool usually intellectual will be understood that, electronics manufacturer may refer to element through different titles.This public affairs
It opens and be not intended to different in those titles but distinguished between functionally different elements.In following description
And in detail in the claims, term " including " and " including " are carried out in an open-ended fashion using and therefore should being construed as
Mean " including, but are not limited to ... ".Equally, term " combination " be intended to mean that or indirectly or be directly electrically connected
It connects.Therefore, if a device is bound to another device, this connection can pass through direct electrical connection, or through by it
The indirect electrical connection of his device and connection.
In the various technologies for realizing computer stereo vision with estimation of Depth for range sensing, such as structure light
(structured-light), active twin-lens Stereo matching (also referred to as active stereo matching) (Active Stereo) and fly
Row time (TOF), electromagnetic wave (for example, laser or structure light) are launched or otherwise project scene, then can be with
By executing Stereo matching using the projector-camera corresponding relationship and/or camera-camera corresponding relationship.Every kind of estimation of Depth
Technology provides its advantage respectively.It, can be by the fusion method that is proposed under the scheme according to the various propositions of the disclosure
To provide the depth information of accurate and gamut.More specifically, can use the difference of sensor (for example, camera) and the projector
Device configuration allows and is fused together the advantages of different depth estimation technique.
Fig. 1 shows the exemplary scene 100 realized according to the disclosure.Scene 100 shows being permitted for sensor and the projector
More distinct device configurations.In the part (A) of scene 100, the configuration of the first example apparatus can be related to two IR cameras, an IR
The projector and RGB (RGB) camera.In the part (B) of scene 100, the configuration of the second example apparatus can be related to one
IR camera, an IR projector and one include the RGB camera for the pixel that can receive pure IR light (here shown as " RGB-IR
Camera ").In the part (C) of scene 100, the configuration of third example apparatus can be related to a RGB camera, an IR projector
With a RGB-IR camera.In the part (D) of scene 100, the 4th device configuration can be related to two RGB cameras, an IR
The projector (or TOF projector) and an IR camera (or TOF camera).
In the configuration of each example devices, the physical distance between two sensor/cameras is expressed as baseline.Each
In example apparatus configuration, the RGB camera as auxiliary RGB camera can provide the colouring information for the depth map to be generated.It may
Need to calibrate a pair of of camera and camera and the projector.It, can be good smart using usually having for a pair of of projector and camera
The structure light or TOF method of degree.For a pair of of camera, stereo algorithm (being usually good at integrality) can be applied to estimating depth.
Under the scheme proposed, these results can be fused together to generate accurate gamut depth or depth map.
It is worth noting that, in the device configuration and its any modification in scene 100, each RGB camera and RGB-IR
Camera can be replaced by following one: monochrome cameras (monochrome camera) is (here shown as " grayscale camera (mono
Camera) ")), a kind of RGB camera has biobelt bandpass filtering (DB-DualBand) energy relative to visible light and IR light
Power (is denoted herein as " RGB-DB camera "), and a kind of monochrome cameras comprising that can receive the pixel of pure IR light (is denoted herein as
" grayscale IR (mono-IR) camera ") and with grayscale camera (this about visible light and the biobelt bandpass filtering ability of IR light
In be expressed as " grayscale DB video camera ").In addition, IR camera, RGB camera, RGB-IR camera, RGB-DB camera, grayscale camera,
Grayscale IR camera and grayscale DB it is magazine each be known as electromagnetism (EM) wave sensor with can be interchanged because each such
Camera can sense the EM wave in visible and/or invisible (for example, IR) spectrum.
The matched IR feature of active stereo
Under the scheme according to the proposition of the disclosure, the structuring IR light that is emitted by the IR projector or otherwise projected
(structured IR light) (also referred to as patterning IR light (patterned IR light))) one or more can be met
Feature request (characteristic requirements).That is, the one or more features of patterning IR light can be with
By utilizing relatively cheap component (such as two cameras and an IR projector) to match for active stereo.It therefore, can be with
By active stereo matching, cost of implementation is saved in computer stereo vision, without by relatively expensive component.
Fig. 2 shows the active stereo matched exemplary scenes 200 according to the realization of the disclosure.It, can in scene 200
Active stereo matching is executed to use two cameras and an IR projector.Two it is magazine each can be IR phase
Machine, RGB-IR camera, RGB-DB camera, grayscale IR camera or grayscale DB camera.
In operation, the IR projector can emit towards scene or otherwise projection patterns IR light, and two
It is magazine each can be with the respective image of capturing scenes (for example, the left camera and capturing scenes of the left image of capturing scenes
Right image right camera).As shown in Fig. 2, can be to the given pixel or picture in the specified or predefined window in left image
The respective pixel or block of pixels execution active stereo matching in specified or predefined window in plain block and right image.It is actively vertical
The matched result of body can be used for generating depth map.
Fig. 3 shows the exemplary scene 300 according to the realization of the disclosure.It provides with reference to Fig. 3 about for active stereo
The scheme of matched IR feature proposed is described below.
Under the scheme according to the proposition of the disclosure, to patterning IR light in IR pattern shape can there is no limit or
Constraint.That is, IR pattern can be formed by multiple IR pixels, be formed as (such as, but not limited to) one or more points, one
A or multiple lines, one or more circles, one or more oval, one or more polygons, one or more stars or its group
It closes.IR pattern may be different because of equipment (for example, from an IR projector to another IR projector).In other words, Mei Geshe
It is standby can be different from other equipment.The example of different IR modes is shown in the part (A) of scene 300.
Under the scheme according to the proposition of the disclosure, the density for patterning the IR pattern of IR light can be sufficiently high, so that often
A block of pixels can be differentiable.Density can be expressed as (pixel/unit area quantity or appearance (occurrence of
Pixels/unit area)), and unit area can be expressed as (widthxhieght pixel2).The part of reference scene 300
(B), the density of IR pattern of interest can be specified or pre- in the IR image with multiple IR patterns relative to capture
Determine the density (for example, from IR camera, RGB-IR camera, RGB-DB camera, grayscale IR camera or grayscale DB camera) of window.
The part (B) of scene 300 is also shown for the matched direction of search of active stereo.
Therefore, under the scheme proposed, the pattern of patterning IR light may include with the density for meeting density requirements
Multiple pixels, it is as follows: (sum of all pixels in predefined window in quantity/capture IR image of IR pixel)) >=first
Threshold value.Here, first threshold (or threshold value 1) can be used for constraining the density of the IR pattern in the given window of IR image.In addition, threshold
Value 1 can be determined by the quality of output depth map.The value of threshold value 1 can be such as 0.2, and unit is 1/ pixel (a unit of
1/pixel)。
Under the scheme according to the proposition of the disclosure, in the case where given pattern is repeated quickly and easily as many times as required (here shown as " weight
Multiple pattern (repeated pattern) "), it can be with along the repetition period of the repeat patterns of the matched direction of search of active stereo
Greater than the matched opereating specification of active stereo.Opereating specification can be, and such as, but not limited to, be equal to the part (B) of scene 300
Shown in specified window predefined window.It for illustrative purpose rather than limits, the part (C) of scene 300 shows weight
The repetition period of multiple pattern is less than the example of opereating specification.
Under the scheme according to the proposition of the disclosure, along the fuzziness in the search range of the direction of search of Stereo matching
It may be relatively low.The cost function (cost function) of definition can be used to calculate the mould of each pixel or block of pixels
Paste degree.And minimum cost value (cost value) is values of ambiguity relative to the value of the second minimum cost value.Values of ambiguity is answered
Lower than threshold value (such as 0.8).
The cost function of definition can be used to calculate the fuzziness of each pixel or block of pixels.Fig. 4 is shown according to this
The exemplary scene 400 of disclosed realization.In scene 400, along each of in the search range of the direction of search of Stereo matching
The values of ambiguity of pixel or each block of pixels is less than or equal to second threshold (or threshold value 2), which for example can be 0.8.Example
Such as, when executing the Stereo matching of left image and right image, including use LOOK LEFT left image and LOOK RIGHT right image it
Between the cost function of value at cost (or matching value at cost) of difference calculate values of ambiguity.Cost function can use mathematical expression
Are as follows:
Here,It can indicate the brightness (or color response) of the current pixel in the block (patch) in left image
(it is expressed as " PL"), andIt can indicate the current pixel in the reference block during Stereo matching in right image
Brightness (or color response) (is expressed as " PR").It, can when use cost function calculates values of ambiguity under the scheme proposed
By the way that the minimum cost value from cost function is calculated fuzziness divided by the second minimum cost value from cost function
Value.
Under the scheme according to the proposition of the disclosure, the inclination (tilt) or rotation of IR pattern can use
(rotation) angle reduces the repeatabilities of the repeat patterns along Stereo matching direction, to follow the pact of low fuzziness
Beam.The absolute value for tilting or rotating angle can be greater than 0 ° and less than 90 °.Fig. 5 shows the example field according to the realization of the disclosure
Scape 500.In the part (A) of scene 500, the repetition direction of repeat patterns is consistent or parallel with the direction of search of Stereo matching.
In the part (B) of scene 500, the repetition direction of repeat patterns is rotated relative to the direction of search of Stereo matching.In scene 500
Part (C) in, inclination/rotation angle absolute value can be greater than 0 ° and less than 90 °.
Depth integration (depth fusion)
Under the scheme according to the proposition of the disclosure, it can be matched by fusion from such as structure light, passive stereo
(passive stereo), active stereo match the depth information of the different depth estimation technique of (active stereo) and TOF
To obtain accurate and gamut depth information.Fig. 6 shows the exemplary scene 600 of the depth integration of the realization according to the disclosure.
Scene 600 can be related to the EM wave projector and two sensors.The EM wave projector can be used for transmitting or otherwise projection figure
Case.The EM wave projector can be such as IR projector or the TOF projector (for example, light reaches (light detection and
Ranging is abbreviated as LiDAR) projector).Two sensors can be a pair of of camera or a camera adds a TOF sensing
Device.As camera, each sensor can be RGB camera, RGB-IR camera, RGB-DB camera, grayscale camera, grayscale IR phase
Machine or grayscale DB camera.
In operation, the EM wave projector can emit or otherwise project pattern, and can be by structure light side
Method or TOF method obtain depth map and confidence map using the capture pattern of the first camera from two sensors
(confidence map).Furthermore it is possible to by stereoscopic approach (for example, active stereo and/or passive stereo) using from the
The capture pattern of one camera and the capture pattern of the second camera from two sensors obtain depth map and confidence map.So
Afterwards, the depth map from structure light/TOF method depth map and confidence map and from stereoscopic approach and confidence map can pass through
Depth integration is fused together to generate and merge depth map (fused depth map).In scene 600, utilization can be passed through
Fusion method 700 and fusion method 800 described below execute depth integration.
Fig. 7 shows the example fusion method 700 according to the realization of the disclosure.In fusion method 700, it can weigh first
Depth map of the new mappings from one of method (for example, structure light/TOF method), then by considering structure light/TOF method
Confidence map and the confidence map of stereoscopic approach to merge with the depth map from other methods (for example, stereoscopic approach), to provide
Fusion results.It is then possible to execute post-processing to fusion results to generate fusion depth map.Due to the baseline between distinct methods
It is different, it is therefore desirable to remap from one of method depth map.
In fusion method 700, the confidence map of method, confidence level (initial spike ratio (peak ratio are given)) can be expressed as follows:
CPKRN=the second minimum cost value/minimum cost value
Here it is possible to by algorithm manufacturing cost value, such as obtain the absolute difference between two capture images
(absolute difference), can be expressed as follows:
Herein, I indicates image intensity, PLIndicate the pixel (or block of pixels) in left image, PRIndicate the pixel in right image
(or block of pixels).For post-processing, the post-processing in fusion method may relate to edge perception filtering and segmentation.In addition, depth
Fusion can be expressed as follows:
D (p)=argmaxd(Conf(stereo(p)),Conf(structured light(p)))
Here, p can indicate each pixel in given depth figure, and Conf () can indicate confidence map.
Fig. 8 shows the example fusion method 800 according to the realization of the disclosure.Fusion method 800, which can be similar to, to be had
The fusion method 700 of some differences.For example, can estimate in cost amount (cost volume) stage in fusion method 800
Fusion.It can be combined by the cost volume that each of method of structured light and stereoscopic approach generate and be optimized to obtain more
Correct depth map.In fusion method 800, depth integration can be expressed as follows:
Cost (p, d)=weightConf(stereo)x coststereo(p,d)+weightConf(structured light)x coststructured light(p,d)
Under the scheme proposed, fusion method 700 and fusion method 800 can be exploited separately for different situations and answer
With.For having the whole realization there are two sensor and a projector, when covering a random component, depth quality may
Dramatically different, this is important ambiguity.
Fig. 9 shows the exemplary device 900 according to the realization of the disclosure.Device 900 can be performed various functions to realize
It is as described herein to sense related process, scheme, technology, processing and method with space or depth perception, have accurate and gamut deep
Degree is merged and is sensed and for the matched IR pattern feature of active stereo, including above with respect toWhat is described is each
Kind process, scene, scheme, solution, concept and technology and process described below 1000,1100 and 1200.
Device 900 can be electronic equipment, portable or mobile equipment, wearable device, wireless telecom equipment or calculating
A part of equipment.For example, device 900 can be in smart phone, smartwatch, personal digital assistant, digital camera or such as
It is realized in the calculating equipment of tablet computer, laptop computer or laptop.In addition, device 900 can also be machine type
A part of type device can be such as non-moving or fixed device, home devices, wire communication device or computing device
IoT or NB-IoT device.For example, device 900 can be in intelligent thermostat, intelligent refrigerator, intelligent door lock, wireless speaker or family
It is realized in the control centre of front yard.Alternatively, device 900 can be realized in the form of one or more integrated circuits (IC) chip, such as
But be not limited to, one or more single core processors, one or more multi-core processor, one or more reductions instruction set calculate
(reduced-instruction-set-computing is abbreviated as RISC) processor or one or more complex instruction set meters
Calculate (CISC) processor.
Device 900 may include at least some of those components shown in Fig. 9, such as control circuit 910, Duo Gechuan
Sensor 920 (1)~920 (N) and at least one EM wave projector 930, wherein N is greater than 1 positive integer.Device 900 can be with
Including one or more other assemblies unrelated with the scheme of the disclosure proposed (for example, internal electric source, show equipment and/
Or user interface apparatus), it is not shown this component of equipment 900, and therefore, in Fig. 9 in order to simply and for purpose of brevity, below
Also it is not described.
In one aspect, control circuit 910 can be to realize in the form of including the electronic circuit of various electronic components.It is optional
Ground, control circuit 910 can be with one or more single core processors, one or more multi-core processors, one or more RISC
The form of processor or one or more cisc processors is realized.That is, even if used here as singular references " processor "
Control circuit 910 is referred to, control circuit 910 may include multiple processors in some implementations, and according to the disclosure
Other realizations in may include single processor.On the other hand, device 910 can with electronic building brick hardware (with
And optionally, firmware) form realize, the electronic building brick include, but not limited to, e.g. one or more transistors, one or
Multiple diodes, one or more capacitors, one or more resistors, one or more inductors, one or more memristors
Device and/or one or more varactors, are configured and arranged to realize the specific purpose according to the disclosure.In other words
It says, at least some embodiments, control circuit 910 is special purpose machinery, specially designs, is arranged and disposed to execute and regard
Feel the related particular task of depth sense, there is accurate and gamut depth integration and sensing and IR pattern feature, is used for root
According to the active stereo matching of the various embodiments of the disclosure.In some embodiments, control circuit 910 may include having
The electronic circuit of hardware component, the hardware component realize one or more of the scheme of the various propositions according to the disclosure.
Alternatively, control circuit 910 can also utilize software code and/or instruction in addition to hardware components other than hardware component
To realize there is the space or depth perception sensing of accurate and gamut depth integration and sensing and IR pattern feature to be used for according to this public affairs
The active stereo for the various embodiments opened matches.
Each of multiple sensor 920 (1)~920 (N) can be camera or TOF sensor.In the background of camera
Under, corresponding sensor can be IR camera, RGB camera, grayscale camera, RGB-IR camera, grayscale IR camera, RGB-BD phase
Machine or grayscale BD camera.The EM wave projector can be the IR projector or the TOF projector.
Under the scheme according to the various propositions of the disclosure, about the view with accurate and gamut depth integration and sensing
Feel depth sense, control circuit 910 can receive different types of multiple sensors from multiple sensor 920 (1)~920 (N)
Signal.In addition, control circuit 910 can be generated based on multiple sensor signals scene the first depth-related information and scene the
Two depth-related informations.In addition, control circuit 910 can merge the first depth-related information and the second depth-related information with life
At the fusion depth map of scene.
In some embodiments, different types of multiple sensors are being received from multiple sensor 920 (1)~920 (N)
When signal, control circuit 910 can two or more from following receive multiple sensor signals: RGB camera, grayscale phase
Machine, IR camera, RGB-IR camera, grayscale infrared camera, RGB-DB camera, grayscale DB camera and TOF sensor.
In some implementations, when generating the first depth-related information and the second depth-related information, control circuit 910 can
To execute multiple operations.For example, control circuit 910 can be based on multiple sensor signals 920 (1)~920 (N) of the first kind
In at least the first sensor signal generate the first depth map and the first confidence map.In addition, control circuit 910 can be based on the
At least second sensor signal in multiple sensor signals 920 (1)~920 (N) of one different types of Second Type generates
Second depth map and the second confidence map.
In some embodiments, in generating the first depth map and the first confidence map, structure is can be used in control circuit 910
Light method or TOF method generate the first depth map and the first confidence map.In some embodiments, generate the second depth map and
In second confidence map, active stereo method or passive stereo method can be used to generate the second depth map and second for control circuit 910
Confidence map.
In some embodiments, the first depth-related information and the second depth-related information are being merged to generate fusion deeply
When degree figure, if dry run can be performed in control circuit 910.For example, control circuit 910 can remap relative to the second depth map
First depth map is to generate the first depth map to remap.In addition, control circuit 910 can merge first to remap deeply
Degree figure, the second depth map, the first confidence map and the second confidence map are to provide fusion results.In addition, control circuit 910 can be to melting
It closes result and executes post-processing to generate fusion depth map.
Alternatively, when merging the first depth-related information and the second depth-related information to generate fusion depth map,
Control circuit 910 can execute other operations.For example, control circuit 910 can remap first relative to the second depth map
Depth map is to generate the first depth map to remap.In addition, control circuit 910 can estimate and generate the first depth map and
The associated cost amount of first confidence map.In addition, control circuit 910 can merge the first depth map to remap, second deeply
Degree figure, the first confidence map, the second confidence map and cost amount are to provide fusion results.In addition, control circuit 910 can tie fusion
Fruit executes post-processing to generate fusion depth map.In addition, when generating the first depth map and the first confidence map, control circuit 910
Method of structured light can be used or TOF method generates the first depth map and the first confidence map.
In some embodiments, in estimated cost amount, control circuit 910 can be associated with stereoscopic approach by calculating
Weighted cost carry out estimated cost amount with the combination of weighted cost associated with method of structured light.
In some embodiments, the first depth-related information and the second depth-related information are being merged to generate fusion deeply
When degree figure, if dry run can be performed in control circuit 910.For example, control circuit 910 can be used the first fusion method or second melt
Conjunction method determines whether to merge the first depth-related information and the second depth-related information.Then, based on the definitive result, control
The first fusion method can be used in circuit 910 or the second fusion method merges the first depth-related information and the second depth correlation letter
Breath.First fusion method, which may include: (a1), to be remapped relative to the second depth map first depth map that remaps with generating
The first depth map；(b1) the first depth map, the second depth map, the first confidence map and the second confidence map to remap is merged
To provide fusion results；(c1) fusion results are post-processed, generates fusion depth map.Second fusion method may include:
(a2) the first depth map is remapped relative to the second depth map to generate the first depth map to remap；(b2) estimation and life
At the first depth map and the associated cost amount of the first confidence map；(c2) the first depth map to remap, the second depth are merged
Figure, the first confidence map, the second confidence map and cost amount are to provide fusion results；(d2) fusion results are post-processed, is generated
Merge depth map.In some embodiments, in the second fusion method, method of structured light or TOF method can be used to generate the
One depth map and the first confidence map.
In some embodiments, control circuit 910 can control the EM wave projector 930 to emit electromagnetic wave towards scene.
The EM wave projector 930 may include the IR projector or the TOF projector.
In some embodiments, control circuit 910 can calibrate a pair in multiple sensor 920 (1)~920 (N)
One in sensor or multiple sensors 920 (1)~920 (N) and the EM wave projector 930.
Under the scheme according to the proposition of the disclosure, about for the matched IR pattern feature of active stereo, control circuit
910, which can control the EM wave projector 920 (for example, IR projector), patterns IR light to project.In addition, control circuit 910 can be with
From first camera (for example, sensor 920 (1)) receive scene left image the first data, and from second camera (for example,
Sensor 920 (2)) receive scene right image the second data.In addition, control circuit 910 can execute left image and right figure
The Stereo matching (for example, active stereo matching) of picture is to generate the depth map of scene.Patterning IR light can satisfy one or more
A feature request.
In some embodiments, the pattern for patterning IR light may include multiple IR pixels, form one or more
Point, one or one or more line, one or more circles, one or more ellipses, one or more polygons,
One or more star-shaped shape or combinations thereof.
In some embodiments, the pattern for patterning IR light may include multiple pixels, and density meets density requirements,
So that the quantity of IR pixel is greater than or equal to the first threshold divided by the sum of all pixels in the predefined window in left image or right image
Value.
In some embodiments, first threshold can be 0.2.
In some embodiments, patterning IR light may include multiple examples of repeat patterns.In this case, along
The repetition period of the direction of search repeat patterns of Stereo matching can be greater than the opereating specification of Stereo matching.
In some embodiments, the repetition direction of repeat patterns can be tilted more than relative to the direction of search of Stereo matching
0 ° and the angle less than 90 °.
In some embodiments, the mould of each pixel in the search range in Stereo matching direction or each block of pixels
Paste angle value may be less than or equal to second threshold.
In some embodiments, in the Stereo matching for executing left image and right image, control circuit 910 be can be used
The cost function of the value at cost of difference between left image and right image calculates values of ambiguity.In some implementations, cost function
It can mathematically be expressed as follows:
Here,It can indicate the brightness of the current pixel in the block in left image, andIt can be with
Indicate the brightness of the current pixel in the reference block during Stereo matching in right image.
In some embodiments, when use cost function calculates values of ambiguity, control circuit 910 can be by that will come from
The minimum cost value of cost function calculates values of ambiguity divided by the second minimum cost value from cost function.
In some embodiments, second threshold can be 0.8.
In some embodiments, in the Stereo matching for executing left image and right image, control circuit 910 is executable
The matching of the active stereo of left image and right image.
Under the scheme proposed according to another of the disclosure, about the matched IR pattern feature of active stereo is used for, control
Circuit 910 processed can control the EM wave projector 930 (for example, IR projector) and pattern IR light to project.In addition, control circuit
910 can receive the first data of the left image of scene from first camera (for example, sensor 920 (1)), and from second camera
(for example, sensor 920 (2)) receives the second data of the right image of scene.In addition, control circuit 910 can execute left image
It is matched with the active stereo of right image, to generate the depth map of scene.Patterning IR light can satisfy in multiple feature requests
It is one or more.In some embodiments, multiple feature requests may include: the pattern of (1) patterning IR light comprising more
A pixel, density meet density requirements, so that the quantity of IR pixel is divided by the predefined window in left image or right image
Sum of all pixels be greater than or equal to first threshold；(2) IR light, the example including multiple repeat patterns, so that along vertical are patterned
The repetition period of the repeat patterns of the matched direction of search of body is greater than the opereating specification of Stereo matching；(3) repetition of repeat patterns
Direction rotates angle of the absolute value greater than 0 ° and less than 90 ° relative to the direction of search of Stereo matching.
In some embodiments, the pattern for patterning IR light may include multiple IR pixels, form one or more
Point, one or one or more line, one or more circles, one or more ellipses, one or more polygons,
One or more star-shaped shape or combinations thereof.
In some embodiments, the mould of each pixel in the search range in Stereo matching direction or each block of pixels
Paste angle value may be less than or equal to second threshold.
In some embodiments, in the Stereo matching for executing left image and right image, control circuit 910 be can be used
The cost function of the value at cost of difference between left image and right image calculates values of ambiguity.In some implementations, cost function
It can mathematically be expressed as follows:
Here,It can indicate the brightness of the current pixel in the block in left image, andIt can be with
Indicate the brightness of the current pixel in the reference block during Stereo matching in right image.In addition, first threshold can be 0.2,
Second threshold can be 0.8.
In some embodiments, when use cost function calculates values of ambiguity, control circuit 910 can be by that will come from
The minimum cost value of cost function calculates values of ambiguity divided by the second minimum cost value from cost function.
Figure 10 shows the instantiation procedure 1000 according to the realization of the disclosure.Process 1000 (either partially or complete)
Can be about according to the various processes that there is the space or depth perception of accurate and gamut depth integration and sensing to sense of the disclosure,
The example implementation of scene, conception, solution, concept and technology or combinations thereof.Process 1000 can indicate the feature of device 900
Realization one aspect.Process 1000 may include one or more operations, movement or function, such as block 1010,1020 and
Shown in one or more of 1030.It, can be by each of process 1000 while shown as discrete area, but according to desired realization
Kind block is divided into extra block, is combined into less piece or eliminates.In addition, the block of process 1000 can be according to shown in Figure 10
Sequence executes, or can be executed in different order.In addition, one or more blocks of process 1000 can be repeated once or more
It is secondary.Process 1000 can be realized by device 900 or its any modification.It merely for illustrative purpose rather than limits, is filling below
Set 900 process of described in the text up and down 1000.Process 1000 can start at block 1010.
At 1010, it is more from the reception of multiple sensor 920 (1)~920 (N) that process 1000 can be related to control circuit 910
The different sensor signal of seed type.Process 1000 can proceed to 1020 from 1010.
At 1020, process 1000 can be related to control circuit 910 and generate the first of scene based on multiple sensor signals
Second depth-related information of depth-related information and scene.Process 1000 can proceed to 1030 from 1020.
At 1030, process 1000 can be related to control circuit 910 and merge the first depth-related information and the second depth phase
Information is closed to generate the fusion depth map of scene.
In some embodiments, different types of multiple sensors are being received from multiple sensor 920 (1)~920 (N)
When signal, process 1000 can be related to control circuit 910 and receive multiple sensor letters from two or more following sensors
Number: RGB camera, grayscale camera, IR camera, RGB-IR camera, grayscale IR camera, RGB-DB camera, grayscale DB camera and TOF are passed
Sensor.
In some implementations, the first depth-related information is being generated and when the second depth-related information, process 1000 can be with
It is related to control circuit 910 and executes multiple operations.For example, process 1000 can be related to control circuit 910 based on the more of the first kind
At least the first sensor signal in a sensor signal 920 (1)~920 (N) generates the first depth map and the first confidence map.Separately
Outside, process 1000 can be related to multiple sensor signals 920 of the control circuit 910 based on the Second Type different from the first kind
(1)~920 at least second sensor signal in (N) generates the second depth map and the second confidence map.
In some embodiments, in generating the first depth map and the first confidence map, process 1000 can be related to control electricity
Road 910 generates the first depth map and the first confidence map using method of structured light or TOF method.In some embodiments, it is producing
In raw second depth map and the second confidence map, it is using active stereo method or passively vertical that process 1000 can be related to control circuit 910
Body method generates the second depth map and the second confidence map.
In some embodiments, the first depth-related information and the second depth-related information are being merged to generate fusion deeply
When degree figure, process 1000 can be related to control circuit 910 and execute multiple operations.For example, process 1000 can be related to control circuit
910 remap the first depth map relative to the second depth map to generate the first depth map to remap.In addition, process 1000
It can be related to control circuit 910 and merge the first depth map, the second depth map, the first confidence map and the second confidence map to remap
To provide fusion results.Fusion results execution post-processing is melted with generating in addition, process 1000 can be related to control circuit 910
Close depth map.
Alternatively, when merging the first depth-related information and the second depth-related information to generate fusion depth map, process
1000, which can be related to control circuit 910, executes other operations.For example, process 1000 can be related to control circuit 910 relative to
Two depth maps remap the first depth map to generate the first depth map to remap.In addition, process 1000 can be related to controlling
The estimation of circuit 910 cost amount associated with the first depth map of generation and the first confidence map processed.In addition, process 1000 can be related to
Control circuit 910 merges the first depth map, the second depth map, the first confidence map, the second confidence map and the cost amount to remap
To provide fusion results.In addition, process 1000, which can be related to control circuit 910, executes post-processing to fusion results to generate fusion
Depth map.In addition, process 1000 can be related to control circuit 910 and use knot when generating the first depth map and the first confidence map
Structure light method or TOF method generate the first depth map and the first confidence map.
In some implementations, in estimated cost amount, process 1000 can be related to control circuit 910 by calculating and solid
The combination of the associated weighted cost of method and weighted cost associated with method of structured light carrys out estimated cost amount.
In some embodiments, the first depth-related information and the second depth-related information are being merged to generate fusion deeply
When degree figure, process 1000 can be related to control circuit 910 and execute multiple operations.For example, process 1000 can be related to control circuit
910 determine whether that merging the first depth-related information and the second depth correlation believes using the first fusion method or the second fusion method
Breath.Then, based on the definitive result, process 1000 can be related to the first fusion method of use of control circuit 910 or the second fusion
Method merges the first depth-related information and the second depth-related information.First fusion method may include: (a1) relative to
Two depth maps remap the first depth map to generate the first depth map to remap；(b1) it is deep to merge first to remap
Degree figure, the second depth map, the first confidence map and the second confidence map are to provide fusion results；(c1) after being carried out to fusion results
Reason generates fusion depth map.Second fusion method may include: (a2) to remap the first depth map relative to the second depth map
To generate the first depth map to remap；(b2) estimation cost associated with the first depth map of generation and the first confidence map
Amount；(c2) the first depth map, the second depth map, the first confidence map, the second confidence map and the cost amount that fusion is remapped are to mention
For fusion results；(d2) fusion results are post-processed, generates fusion depth map.In some embodiments, melt second
In conjunction method, method of structured light or TOF method can be used to generate the first depth map and the first confidence map.
In some embodiments, process 1000 can control the electromagnetic wave projector further to control circuit 910
(electromagnetic wave projector) is to emit electromagnetic wave towards scene.The electromagnetic wave projector may include that IR is thrown
Emitter or the TOF projector.
In some embodiments, process 1000 can also include that control circuit 910 calibrates a pair in multiple sensors
A sensor in sensor or multiple sensors adds the electromagnetic wave projector.
Figure 11 shows the instantiation procedure 1100 according to the realization of the disclosure.Process 1100 can be about according to the disclosure
The matched IR pattern feature of active stereo various processes partially or completely, scene, conception, solution, concept and skill
The example implementation of art or combinations thereof.Process 1100 can indicate the one aspect of the realization of the feature of device 900.Process 1100 can
To include one or more operations, movement or function, as shown in one or more of block 1110,1120 and 1130.Although showing
Be out discrete area, but according to desired realization, can by various pieces of process 1100 be divided into extra block, be combined into it is less
Block or elimination.In addition, the block of process 1100 can be according to sequentially executing shown in Figure 11, or it can be in a different order
It executes.In addition, one or more blocks of process 1100 can be repeated one or more times.Process 1100 can by device 900 or its
Any modification is realized.It merely for illustrative purpose rather than limits, below in the process of described in the text up and down 1100 of device 900.
Process 1100 can start at block 1110.
At 1110, process 1100 can be related to control circuit 910 and control the electromagnetic wave projector 930 (for example, IR is projected
Device) with projection patterning IR light.Process 1100 can proceed to 1120 from 1110.
At 1120, process 1100 can be related to control circuit 910 and receive from first camera (for example, sensor 920 (1))
First data of the left image of scene, and receive from second camera (for example, sensor 920 (2)) scene right image the
Two data.Process 1100 can proceed to 1130 from 1120.
At 1130, process 1100 can be related to the Stereo matching (example that control circuit 910 executes left image and right image
Such as, active stereo matches) to generate the depth map of scene.Patterning IR light can satisfy one or more features requirement.
In some embodiments, the pattern for patterning IR light may include multiple IR pixels, form one or more
Point, one or one or more line, one or more circles, one or more ellipses, one or more polygons,
One or more star-shaped shape or combinations thereof.
In some embodiments, the pattern for patterning IR light may include multiple pixels, and density meets density requirements,
So that the quantity of IR pixel is greater than or equal to the first threshold divided by the sum of all pixels in the predefined window in left image or right image
Value.
In some embodiments, first threshold can be 0.2.
In some embodiments, patterning IR light may include multiple examples of repeat patterns.In this case, along
The repetition period of the direction of search repeat patterns of Stereo matching can be greater than the opereating specification of Stereo matching.
In some embodiments, the repetition direction of repeat patterns can rotate relative to the direction of search of Stereo matching and be greater than
0 ° of absolute value and the angle less than 90 °.
In some embodiments, the mould of each pixel in the search range in Stereo matching direction or each block of pixels
Paste angle value may be less than or equal to second threshold.
In some embodiments, in the Stereo matching for executing left image and right image, control circuit 910 be can be used
The cost function of the value at cost of difference between left image and right image calculates values of ambiguity.In some implementations, cost function
It can mathematically be expressed as follows:
Here,It can indicate the brightness of the current pixel in the block in left image, andIt can be with
Indicate the brightness of the current pixel in the reference block during Stereo matching in right image.
In some embodiments, when use cost function calculates values of ambiguity, process 1100 can be related to control circuit
910 by calculating fuzziness divided by the second minimum cost value from cost function for the minimum cost value from cost function
Value.
In some embodiments, second threshold can be 0.8.
In some embodiments, in the Stereo matching for executing left image and right image, process 1100 can be related to control
Circuit 910 executes the active stereo matching of left image and right image.
Figure 12 shows the instantiation procedure 1200 according to the realization of the disclosure.Process 1200 can be about according to the disclosure
The matched IR pattern feature of active stereo various processes partially or completely, scene, conception, solution, concept and skill
The example implementation of art or combinations thereof.Process 1200 can indicate the one aspect of the realization of the feature of device 900.Process 1200 can
To include one or more operations, movement or function, as shown in one or more of block 1210,1220 and 1230.Although showing
Be out discrete area, but according to desired realization, can by various pieces of process 1200 be divided into extra block, be combined into it is less
Block or elimination.In addition, the block of process 1200 can be according to sequentially executing shown in Figure 12, or it can be in a different order
It executes.In addition, one or more blocks of process 1200 can be repeated one or more times.Process 1200 can by device 900 or its
Any modification is realized.It merely for illustrative purpose rather than limits, below in the process of described in the text up and down 1200 of device 900.
Process 1200 can start at block 1210.
At 1210, process 1200 can be related to control circuit 910 and control the electromagnetic wave projector 930 (for example, IR is projected
Device) with projection patterning IR light.Process 1200 can proceed to 1220 from 1210.
At 1220, process 1200 can be related to control circuit 910 and connect from first camera (for example, sensor 920 (1))
First data of the left image of end scape, and the right image of scene is received from second camera (for example, sensor 920 (2))
Second data.Process 1200 can proceed to 1230 from 1220.
At 1230, process 1200 can be related to the active stereo matching that control circuit 910 executes left image and right image
To generate the depth map of scene.Patterning IR light can satisfy one or more of multiple feature requests.
In some embodiments, it includes multiple that the multiple feature request, which may include: the pattern of (1) patterning IR light,
Pixel, density meet density requirements, so that the quantity of IR pixel is divided by the predefined window in left image or right image
Sum of all pixels is greater than or equal to first threshold；(2) patterning IR light includes multiple examples of repeat patterns, so that along three-dimensional
The repetition period for the direction of search repeat patterns matched is greater than the opereating specification of Stereo matching；(3) the repetition direction phase of repeat patterns
Angle of the absolute value greater than 0 ° and less than 90 ° is rotated for the direction of search of Stereo matching.
In some embodiments, the pattern for patterning IR light may include multiple IR pixels, form one or more
Point, one or one or more line, one or more circles, one or more ellipses, one or more polygons,
One or more star-shaped shape or combinations thereof.
In some embodiments, the mould of each pixel in the search range in Stereo matching direction or each block of pixels
Paste angle value may be less than or equal to second threshold.
In some embodiments, in the Stereo matching for executing left image and right image, process 1200 can be related to control
Circuit 910 calculates values of ambiguity using the cost function of the value at cost of the difference between left image and right image.In some realizations
In, cost function can be mathematically expressed as follows:
Here,It can indicate the brightness of the current pixel in the block in left image, andIt can be with
Indicate the brightness of the current pixel in the reference block during Stereo matching in right image.In addition, first threshold can be 0.2,
Second threshold can be 0.8.
In some embodiments, when use cost function calculates values of ambiguity, process 1200 can be related to control circuit
910 by calculating fuzziness divided by the second minimum cost value from cost function for the minimum cost value from cost function
Value.
The theme of described in the text is shown sometimes to be included in other different components or connects from other different components
Different components.It is understood that the framework described in this way is only exemplary, also, can actually implement to realize identical function
Many other frameworks of energy.On conceptual sense, realize the component of identical function any arrangement be effectively " associated ",
So that realizing desired function.Therefore, any two component for specific function being recombined to obtain in text can be considered as that
This " associated ", to realize desired function, but regardless of framework or intermediate member how.Similarly, associated in this way
Two components of anticipating may be viewed as " being operably connected " each other or " being operatively coupled to ", to realize desired function
Can, further, it is possible to which any two component associated in this way may be viewed as " to couple in operation " each other, to realize the phase
The function of prestige.The specific example of " can be coupled in operation " including but not limited to: can physically be coupled and/or physically mutually,
The component of effect, and/or it can wirelessly interact and/or the component that wirelessly interacts, and/or logically interact
And/or logically interactive component.
In addition, the use about plural number substantially any in text and/or singular references, as long as context and/or answering
Be it is suitable, those skilled in the art can pluralize by complex transform singularization, and/or by odd number transformation.
It will be understood to those of skill in the art that in general, term as used herein, especially in appended claims (example
Such as, the main body in appended claims) used in term be generally intended to as " opening " term (for example, term " packet
Containing " it should be interpreted " including but unlimited dry ", term " having " should be interpreted " at least having " that term "comprising" should
It is interpreted " including but unlimited dry " etc.).It will further be appreciated by those of ordinary skill in the art that if being intended to introduced claims state
The specific number of object, then it is such to be intended to clearly to state in the claims, the case where lacking such statement
Under, such intention is not present.For example, appended claims may include using introductory phrase " at least in order to help to understand
One " and " one or more " introduce scope of the claims enonce.However, the use of such phrase should not be by
It explains are as follows: will be wanted comprising the right introduced in this way with the introduction of the claims state object of indefinite article " one (a or an) "
Asking any scope of the claims of enonce to be limited to only includes the invention of such enonce, even if in same power
Sharp claimed range includes introductory phrase " one or more " or "at least one" and such as " one (a) " or " one
Etc (an) " (for example, " one (a) " and/or " one (an) " should be generally interpreted as anticipating in the case where indefinite article
Taste "at least one" or " one or more ") also so；It is above-mentioned to introduce claims state object to definite article
Situation is equally applicable.In addition, even if clearly set forth the specific number of the claims state object of introduction, but this field skill
Art personnel will also recognize that: the number that such statement should be typically interpreted to mean at least to be stated is (for example, only
" two enonces " without the statement of other modifiers generally mean that at least two enonces or two or more
Enonce).In addition, using the idiom for being similar to " at least one of A, B and C etc. ", it is usually such
Structure is intended to the meaning of the idiom understood by one of ordinary skill in the art (for example, " at least one of A, B and C
System " will including but not limited to risen with individual A, individual B, individual C, A and B-, A and C together, B and C together and/
Or the system etc. of A, B and C together).Using the idiom for being similar to " at least one of A, B or C etc. ", usually
Such structure is intended to the meaning of the idiom understood by one of ordinary skill in the art (for example, " at least one in A, B or C
A system " will including but not limited to risen with individual A, individual B, individual C, A and B-, A and C together, B and C mono-
Rise and/or A, B and C system etc. together).Those skilled in the art understand Jin mono- Walk, regardless of in specification, claim
Still in the accompanying drawings, indicate two or more interchangeable terms almost arbitrarily extract word and/or phrase should be understood to
Consider include one in term, there are two any of term or institutes a possibility that term.For example, phrase " A or B " should
It is understood as that a possibility that including " A ", " B " or " A and B ".
Although some illustrative skills have been described and illustrated using different methods, equipment and system in the text
Art, but skilled person would appreciate that: it can be carried out in the case where not departing from theme claimed each
The other modifications of kind and progress equivalent replacement.In addition, can be carried out in the case where not departing from the central scope of described in the text
Many modifications are so that specific situation is adapted to the introduction of theme claimed.It is therefore intended that theme claimed
It is not limited to disclosed particular example, and such claimed theme can also be comprising falling in appended claims
All implementations and their equivalent in range.
The foregoing is merely the preferred embodiments of the invention, all equivalent changes done according to claims of the present invention with repair
Decorations, should all belong to the covering scope of the present invention.
Claims (20)
1. a kind of depth integration method, comprising:
Multiple different types of sensor signals are received from multiple sensors；
The first depth-related information of scene and the second depth-related information of the scene are generated based on multiple sensor signal；
And
First depth-related information and second depth-related information are merged to generate the fusion depth map of the scene.
2. depth integration method as described in claim 1, which is characterized in that receive multiple type not from multiple sensor
With sensor signal include from RGB camera, grayscale camera, infrared camera, RGB infrared camera, grayscale infrared camera,
With double frequency band-pass filtering RGB camera, with double frequency band-pass filtering grayscale camera and time-of-flight sensor in two or
More receive multiple sensor signal.
3. depth integration method as described in claim 1, which is characterized in that generate first depth-related information and this second
Depth-related information includes:
At least the first sensor signal in multiple sensor signal based on the first kind generates the first depth map and first
Confidence map；And
At least second sensor signal in multiple sensor signal based on the Second Type different from the first kind, it is raw
At the second depth map and the second confidence map.
4. depth integration method as claimed in claim 3, which is characterized in that generate first depth map and first confidence map
Including using method of structured light or pulsed-beam time-of-flight methods to generate first depth map and first confidence map, and wherein, generate
Second depth map and second confidence map include generating second depth map using active stereo method or passive stereo method
With second confidence map.
5. depth integration method as claimed in claim 3, which is characterized in that merge first depth-related information and this second
Depth-related information includes: to generate the fusion depth map
First depth map is remapped relative to second depth map to generate the first depth map to remap；It is heavy to merge this
First depth map of new mappings, second depth map, first confidence map and second confidence map are to provide fusion results；And
Post-processing is executed to generate the fusion depth map to the fusion results.
6. depth integration method as claimed in claim 3, which is characterized in that merge first depth-related information and this second
Depth-related information includes: to generate the fusion depth map
First depth map is remapped relative to second depth map to generate the first depth map to remap；Estimation and life
At first depth map and the associated cost amount of first confidence map；
Merge first depth map to remap, second depth map, first confidence map, second confidence map and the cost
Amount is to provide fusion results；And
The fusion results are post-processed to generate the fusion depth map,
Wherein, it generates first depth map and first confidence map includes generating to be somebody's turn to do using method of structured light or pulsed-beam time-of-flight methods
First depth map and first confidence map.
7. depth integration method as claimed in claim 6, which is characterized in that estimate that the cost amount includes by calculating and solid
The cost amount is estimated in the combination of the associated weighted cost of method and weighted cost associated with method of structured light.
8. depth integration method as claimed in claim 3, which is characterized in that merge first depth-related information and this second
Depth-related information includes: to generate the fusion depth map
Determine whether to merge first depth-related information and second depth using the first fusion method or the second fusion method
Relevant information；And
Based on the definitive result, using first fusion method or second fusion method merge first depth-related information and
Second depth-related information,
Wherein, which includes:
First depth map is remapped relative to second depth map to generate the first depth map to remap；It is heavy to merge this
First depth map of new mappings, second depth map, first confidence map and second confidence map are to provide fusion results；And
The fusion results are post-processed to generate the fusion depth map,
Wherein, which includes:
First depth map is remapped relative to second depth map to generate first depth map to remap；
Estimation cost amount associated with generation first depth map and first confidence map；
Merge first depth map to remap, second depth map, first confidence map, second confidence map and the cost
Amount is to provide fusion results；And
The fusion results are post-processed to generate the fusion depth map, and
Wherein, in second fusion method, first depth map is generated using method of structured light or pulsed-beam time-of-flight methods and is somebody's turn to do
First confidence map.
9. depth integration method as described in claim 1, further includes:
It controls the electromagnetic wave projector and emits electromagnetic wave to the scene,
Wherein the electromagnetic wave projector includes infrared projector or the flight time projector.
10. depth integration method as claimed in claim 9, further includes:
It calibrates a pair of sensors in multiple sensor or one in multiple sensor adds the electromagnetic wave projector.
11. a kind of depth integration device, comprising:
Control circuit is coupled to receive multiple different types of sensor signals from multiple sensors, so that in the operation phase
Between, which executes the operation including following operation:
The first depth-related information of scene and the second depth-related information of the scene are generated based on multiple sensor signal；
And
First depth-related information and second depth-related information are merged to generate the fusion depth map of the scene.
12. depth integration device as claimed in claim 11, which is characterized in that further include:
Multiple sensor, including RGB camera, grayscale camera, infrared camera, RGB infrared camera, the infrared phase of grayscale
Machine, the RGB camera with double frequency band-pass filtering, two in grayscale camera and time-of-flight sensor with double frequency band-pass filtering
It is a or more.
13. depth integration device as claimed in claim 11, which is characterized in that generate first depth-related information and this
When two depth-related informations, which executes the operation comprising following operation:
At least the first sensor signal in multiple sensor signal based on the first kind generates the first depth map and first
Confidence map；And
At least second sensor signal in multiple sensor signal based on the Second Type different from the first kind, it is raw
At the second depth map and the second confidence map.
14. depth integration device as claimed in claim 13, which is characterized in that generating first depth map and this first is set
When letter figure, which generates first depth map and first confidence map using method of structured light or pulsed-beam time-of-flight methods,
And wherein, when generating second depth map and second confidence map, the control circuit is using active stereo method or passively
Stereoscopic approach generates second depth map and second confidence map.
15. depth integration device as claimed in claim 13, which is characterized in that merging first depth-related information and be somebody's turn to do
When second depth-related information is to generate the fusion depth map, which executes the operation comprising following operation:
First depth map is remapped relative to second depth map to generate the first depth map to remap；It is heavy to merge this
First depth map of new mappings, second depth map, first confidence map and second confidence map are to provide fusion results；And
Post-processing is executed to generate the fusion depth map to the fusion results.
16. depth integration device as claimed in claim 13, which is characterized in that merging first depth-related information and be somebody's turn to do
When second depth-related information is to generate the fusion depth map, which executes the operation comprising following operation:
First depth map is remapped relative to second depth map to generate the first depth map to remap；Estimation and life
At first depth map and the associated cost amount of first confidence map；
Merge first depth map to remap, second depth map, first confidence map, second confidence map and the cost
Amount is to provide fusion results；And
The fusion results are post-processed to generate the fusion depth map,
Wherein, it generates first depth map and first confidence map includes generating to be somebody's turn to do using method of structured light or pulsed-beam time-of-flight methods
First depth map and first confidence map.
17. depth integration device as claimed in claim 16, which is characterized in that when estimating the cost amount, the control circuit
Estimated by calculating the combination of weighted cost associated with stereoscopic approach and weighted cost associated with method of structured light
The cost amount.
18. depth integration device as claimed in claim 13, which is characterized in that merging first depth-related information and be somebody's turn to do
When second depth-related information is to generate the fusion depth map, which executes the operation comprising following operation:
Determine whether to merge first depth-related information and second depth using the first fusion method or the second fusion method
Relevant information；And
Based on the definitive result, using first fusion method or second fusion method merge first depth-related information and
Second depth-related information,
Wherein, which includes:
First depth map is remapped relative to second depth map to generate the first depth map to remap；It is heavy to merge this
First depth map of new mappings, second depth map, first confidence map and second confidence map are to provide fusion results；And
The fusion results are post-processed to generate the fusion depth map,
Wherein, which includes:
First depth map is remapped relative to second depth map to generate first depth map to remap；
Estimation cost amount associated with generation first depth map and first confidence map；
Merge first depth map to remap, second depth map, first confidence map, second confidence map and the cost
Amount is to provide fusion results；And
The fusion results are post-processed to generate the fusion depth map, and
Wherein, in second fusion method, first depth map is generated using method of structured light or pulsed-beam time-of-flight methods and is somebody's turn to do
First confidence map.
19. depth integration device as claimed in claim 11, further includes:
The electromagnetic wave projector；And
Multiple sensor,
Wherein, during operation, which more controls the electromagnetic wave projector and emits electromagnetic wave, the wherein electricity to the scene
The magnetic wave projector includes infrared projector or the flight time projector.
20. depth integration device as claimed in claim 19, which is characterized in that during operation, which more calibrates
One in a pair of sensors or multiple sensor in multiple sensor adds the electromagnetic wave projector.
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201862651813P | 2018-04-03 | 2018-04-03 | |
| US62/651,813 | 2018-04-03 | ||
| US16/359,713 | 2019-03-20 | ||
| US16/359,713 US10958897B2 (en) | 2018-04-02 | 2019-03-20 | Method and apparatus of depth fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110349196A true CN110349196A (en) | 2019-10-18 |
| CN110349196B CN110349196B (en) | 2024-03-29 |
Family
ID=68174668
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910261747.5A Active CN110349196B (en) | 2018-04-03 | 2019-04-02 | Depth fusion method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110349196B (en) |
| TW (1) | TWI734092B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113341425A (en) * | 2020-02-17 | 2021-09-03 | 联发科技股份有限公司 | Distance sensing method and related device |
| JP2022538947A (en) * | 2020-05-27 | 2022-09-07 | 北京小米移動軟件有限公司南京分公司 | Image processing method and apparatus, camera assembly, electronic device, storage medium |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090292468A1 (en) * | 2008-03-25 | 2009-11-26 | Shunguang Wu | Collision avoidance method and system using stereo vision and radar sensor fusion |
| US20110025827A1 (en) * | 2009-07-30 | 2011-02-03 | Primesense Ltd. | Depth Mapping Based on Pattern Matching and Stereoscopic Information |
| US20120056982A1 (en) * | 2010-09-08 | 2012-03-08 | Microsoft Corporation | Depth camera based on structured light and stereo vision |
| US8355565B1 (en) * | 2009-10-29 | 2013-01-15 | Hewlett-Packard Development Company, L.P. | Producing high quality depth maps |
| US20130070060A1 (en) * | 2011-09-19 | 2013-03-21 | Pelican Imaging Corporation | Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion |
| US8447098B1 (en) * | 2010-08-20 | 2013-05-21 | Adobe Systems Incorporated | Model-based stereo matching |
| US20130142415A1 (en) * | 2011-12-01 | 2013-06-06 | Gazi Ali | System And Method For Generating Robust Depth Maps Utilizing A Multi-Resolution Procedure |
| CN104899870A (en) * | 2015-05-15 | 2015-09-09 | 清华大学深圳研究生院 | Depth estimation method based on light-field data distribution |
| US20150359462A1 (en) * | 2013-01-16 | 2015-12-17 | Van De Velde Nv | Fitting room mirror |
| US20160005179A1 (en) * | 2012-12-17 | 2016-01-07 | Lsi Corporation | Methods and apparatus for merging depth images generated using distinct depth imaging techniques |
| US20160189419A1 (en) * | 2013-08-09 | 2016-06-30 | Sweep3D Corporation | Systems and methods for generating data indicative of a three-dimensional representation of a scene |
| US20160212411A1 (en) * | 2015-01-20 | 2016-07-21 | Qualcomm Incorporated | Method and apparatus for multiple technology depth map acquisition and fusion |
| US20160292497A1 (en) * | 2015-04-06 | 2016-10-06 | The Texas A&M University System | Fusion of inertial and depth sensors for movement measurements and recognition |
| CN106612387A (en) * | 2015-10-15 | 2017-05-03 | 杭州海康威视数字技术股份有限公司 | Combined depth map acquisition method and depth camera |
| CA3008922A1 (en) * | 2015-12-21 | 2017-06-29 | Koninklijke Philips N.V. | Processing a depth map for an image |
| US20170302910A1 (en) * | 2016-04-19 | 2017-10-19 | Motorola Mobility Llc | Method and apparatus for merging depth maps in a depth camera system |
| CN107277491A (en) * | 2012-11-01 | 2017-10-20 | 谷歌公司 | Generate the method and corresponding medium of the depth map of image |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9560343B2 (en) * | 2012-11-23 | 2017-01-31 | Samsung Electronics Co., Ltd. | Apparatus and method for calibrating multi-layer three-dimensional (3D) display |
| CN106504284B (en) * | 2016-10-24 | 2019-04-12 | 成都通甲优博科技有限责任公司 | A kind of depth picture capturing method combined based on Stereo matching with structure light |
-
2019
- 2019-04-02 CN CN201910261747.5A patent/CN110349196B/en active Active
- 2019-04-03 TW TW108111790A patent/TWI734092B/en active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090292468A1 (en) * | 2008-03-25 | 2009-11-26 | Shunguang Wu | Collision avoidance method and system using stereo vision and radar sensor fusion |
| US20110025827A1 (en) * | 2009-07-30 | 2011-02-03 | Primesense Ltd. | Depth Mapping Based on Pattern Matching and Stereoscopic Information |
| US8355565B1 (en) * | 2009-10-29 | 2013-01-15 | Hewlett-Packard Development Company, L.P. | Producing high quality depth maps |
| US8447098B1 (en) * | 2010-08-20 | 2013-05-21 | Adobe Systems Incorporated | Model-based stereo matching |
| US20120056982A1 (en) * | 2010-09-08 | 2012-03-08 | Microsoft Corporation | Depth camera based on structured light and stereo vision |
| US20130070060A1 (en) * | 2011-09-19 | 2013-03-21 | Pelican Imaging Corporation | Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion |
| US20130142415A1 (en) * | 2011-12-01 | 2013-06-06 | Gazi Ali | System And Method For Generating Robust Depth Maps Utilizing A Multi-Resolution Procedure |
| CN107277491A (en) * | 2012-11-01 | 2017-10-20 | 谷歌公司 | Generate the method and corresponding medium of the depth map of image |
| US20160005179A1 (en) * | 2012-12-17 | 2016-01-07 | Lsi Corporation | Methods and apparatus for merging depth images generated using distinct depth imaging techniques |
| US20150359462A1 (en) * | 2013-01-16 | 2015-12-17 | Van De Velde Nv | Fitting room mirror |
| US20160189419A1 (en) * | 2013-08-09 | 2016-06-30 | Sweep3D Corporation | Systems and methods for generating data indicative of a three-dimensional representation of a scene |
| US20160212411A1 (en) * | 2015-01-20 | 2016-07-21 | Qualcomm Incorporated | Method and apparatus for multiple technology depth map acquisition and fusion |
| CN107113415A (en) * | 2015-01-20 | 2017-08-29 | 高通股份有限公司 | The method and apparatus for obtaining and merging for many technology depth maps |
| US20160292497A1 (en) * | 2015-04-06 | 2016-10-06 | The Texas A&M University System | Fusion of inertial and depth sensors for movement measurements and recognition |
| CN104899870A (en) * | 2015-05-15 | 2015-09-09 | 清华大学深圳研究生院 | Depth estimation method based on light-field data distribution |
| CN106612387A (en) * | 2015-10-15 | 2017-05-03 | 杭州海康威视数字技术股份有限公司 | Combined depth map acquisition method and depth camera |
| CA3008922A1 (en) * | 2015-12-21 | 2017-06-29 | Koninklijke Philips N.V. | Processing a depth map for an image |
| US20170302910A1 (en) * | 2016-04-19 | 2017-10-19 | Motorola Mobility Llc | Method and apparatus for merging depth maps in a depth camera system |
Non-Patent Citations (12)
| Title |
|---|
| YUTONG ZHONG，ET AL: "Depth image interpolation algorithm based on confidence map", 《PROCEEDINGS OF SPIE》 * |
| YUTONG ZHONG，ET AL: "Depth image interpolation algorithm based on confidence map", 《PROCEEDINGS OF SPIE》, 8 March 2017 (2017-03-08), pages 1 - 8 * |
| ZHAOXINLI，ET AL: "Multi-view stereo via depth map fusion: A coordinate decent optimization method", 《NEUROCOMPUTING》 * |
| ZHAOXINLI，ET AL: "Multi-view stereo via depth map fusion: A coordinate decent optimization method", 《NEUROCOMPUTING》, 20 February 2016 (2016-02-20), pages 46 - 61 * |
| 刘娇丽等: "基于TOF与立体匹配相融合的高分辨率深度获取", 《信息技术》 * |
| 刘娇丽等: "基于TOF与立体匹配相融合的高分辨率深度获取", 《信息技术》, no. 12, 25 December 2016 (2016-12-25), pages 198 - 201 * |
| 刘怡光等: "基于置信度的深度图融合", 《四川大学学报(工程科学版)》 * |
| 刘怡光等: "基于置信度的深度图融合", 《四川大学学报(工程科学版)》, no. 04, 13 June 2016 (2016-06-13), pages 104 - 109 * |
| 孙哲等: "基于置信度的TOF与双目系统深度数据融合", 北京航空航天大学学报, no. 08, pages 199 - 206 * |
| 李英等: "基于深度图像绘制技术的Criminisi算法的改进", 计算机工程与设计, no. 05, pages 1287 - 1291 * |
| 肖鹏等: "基于置信图自适应融合的视觉目标跟踪", 《无线电工程》 * |
| 肖鹏等: "基于置信图自适应融合的视觉目标跟踪", 《无线电工程》, no. 09, 5 September 2013 (2013-09-05), pages 24 - 27 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113341425A (en) * | 2020-02-17 | 2021-09-03 | 联发科技股份有限公司 | Distance sensing method and related device |
| JP2022538947A (en) * | 2020-05-27 | 2022-09-07 | 北京小米移動軟件有限公司南京分公司 | Image processing method and apparatus, camera assembly, electronic device, storage medium |
| JP7321187B2 (en) | 2020-05-27 | 2023-08-04 | 北京小米移動軟件有限公司南京分公司 | Image processing method and apparatus, camera assembly, electronic device, storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| TWI734092B (en) | 2021-07-21 |
| CN110349196B (en) | 2024-03-29 |
| TW202001802A (en) | 2020-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105814875B (en) | Selecting camera pairs for stereo imaging | |
| CN107004273A (en) | For colored method, equipment and the media synchronous with deep video | |
| US20180012064A1 (en) | Image Processing Apparatus, Image Processing Method, and Image Communication System | |
| US9001192B2 (en) | Storage medium having stored therein stereoscopic image display program, stereoscopic image display device, stereoscopic image display system, and stereoscopic image display method | |
| KR20210154814A (en) | Head-mounted display with pass-through imaging | |
| US20120056992A1 (en) | Image generation system, image generation method, and information storage medium | |
| JP2016537903A (en) | Connecting and recognizing virtual reality content | |
| US20120068980A1 (en) | Storage medium having stored thereon display control program, display control apparatus, display control system, and display control method | |
| US11017587B2 (en) | Image generation method and image generation device | |
| US11477428B2 (en) | Separable distortion disparity determination | |
| US9420263B2 (en) | Information processor and information processing method | |
| US11240481B2 (en) | Creation and user interactions with three-dimensional wallpaper on computing devices | |
| US11699259B2 (en) | Stylized image painting | |
| US11461883B1 (en) | Dirty lens image correction | |
| US20210306608A1 (en) | Multi-dimensional rendering | |
| CN110349196A (en) | The method and apparatus of depth integration | |
| CN109660748B (en) | Image processing method and system for eyeball sight correction | |
| CN113870213A (en) | Image display method, image display device, storage medium, and electronic apparatus | |
| US20230412779A1 (en) | Artistic effects for images and videos | |
| CN110349198A (en) | Solid matching method and corresponding Stereo matching device | |
| CN106331672A (en) | Method, apparatus and system for obtaining viewpoint image | |
| US20230028976A1 (en) | Display apparatus, image generation method, and program | |
| EP3267682A1 (en) | Multiview video encoding | |
| JP2012227625A (en) | Stereoscopic image photographing device, portable electronic equipment, stereoscopic image photographing control method, stereoscopic image photographing control program and computer readable recording medium | |
| CN109922331A (en) | A kind of image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |