CN108280807A - Monocular depth image collecting device and system and its image processing method - Google Patents
Monocular depth image collecting device and system and its image processing method Download PDFInfo
- Publication number
- CN108280807A CN108280807A CN201710007466.8A CN201710007466A CN108280807A CN 108280807 A CN108280807 A CN 108280807A CN 201710007466 A CN201710007466 A CN 201710007466A CN 108280807 A CN108280807 A CN 108280807A
- Authority
- CN
- China
- Prior art keywords
- image
- depth
- monocular
- monocular depth
- depth image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 37
- 238000003384 imaging method Methods 0.000 claims abstract description 32
- 238000003331 infrared imaging Methods 0.000 claims abstract description 32
- 239000011159 matrix material Substances 0.000 claims description 54
- 230000004927 fusion Effects 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 7
- 238000000034 method Methods 0.000 claims description 7
- 238000002156 mixing Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 11
- 238000006243 chemical reaction Methods 0.000 description 10
- 230000003287 optical effect Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 208000017983 photosensitivity disease Diseases 0.000 description 4
- 231100000434 photosensitization Toxicity 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000003708 edge detection Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000005693 optoelectronics Effects 0.000 description 2
- 238000007500 overflow downdraw method Methods 0.000 description 2
- 238000011946 reduction process Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 230000000392 somatic effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
One monocular depth image collecting device and system and its image processing method, wherein the monocular depth image capturing system includes：One infrared imaging unit；One depth converting unit；One speckle projecting unit；One color imaging member and a control unit, the wherein described speckle projecting unit is for projecting infrared light, the infrared imaging unit receives the infrared light of reflection, obtain Infrared Image Information, the Infrared Image Information is converted to monocular depth image information by the depth converting unit, the color imaging member acquires RGB image, and the RGB image and the monocular depth image co-registration are obtained colored monocular depth image by described control unit.
Description
Technical field
The present invention relates to Image Acquisition and image processing fields, further, are related to a monocular depth image collecting device
With system and its image processing method.
Background technology
With the development of optical measuring technique and computer vision, optical three-dimensional measurement technology is gradually ripe, gradually
Applied to fields such as gesture control, 3D modeling, car radar and robotic vision systems, also become in current optical field
Research hotspot.
Infrared speckle monocular depth technology is that a kind of technology more outstanding is showed in many optical three-dimensional measurement technologies.Its
Basic principle be infrared projector launch regular coding infrared light reflected by testee after captured by infrared phase, thus may be used
Testee is calculated at a distance from infrared speckle monocular depth camera.It is infrared compared with other optical three-dimensional measurement technologies
Speckle monocular depth technology has many advantages, such as that calculation amount is small, real-time, precision is high.
But based on the basic principle of traditional monocular depth Image Acquisition, existing infrared speckle monocular depth camera acquisition
Image be all gray level image, be not coloured image.That is, the image that this mode obtains can not be as common flat
Face image equally passes through color image reproduction testee.Although this mode obtains depth information, there is space multistory sense,
It is to have lost color information, the resolution ratio of obtained image is relatively low, and the clarity of image is poor.This effect makes user experience
It is poor, therefore also limit the application range of infrared speckle monocular depth camera.
On the other hand, hole is usually contained in traditional infrared speckle monocular depth image, picture quality is poor.
Invention content
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
Wherein monocular depth image capturing system acquisition monocular depth image and RGB image, and by monocular depth image and RGB image
It is merged, forms colored monocular depth image.
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
The wherein described monocular depth image collecting device includes an at least color imaging member, in order to acquire RGB image.
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
The wherein described monocular depth image capturing system includes that an infrared imaging unit and a speckle join in unit, in order to cooperate
Collect depth image.
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
Wherein during image procossing, first collected monocular depth image and RGB image are respectively processed, to improve
The precision of monocular depth image and RGB image synthesis.
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
Wherein described monocular depth image co-registration RGB image information, improves the resolution ratio of independent degree of depth image, improves image
Clarity.
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
The wherein described RGB image supplements monocular depth image, forms colored monocular depth image, reduces traditional infrared speckle monocular
Hole in depth image.
It is an object of the present invention to provide a monocular depth image collecting device and system and its image processing method,
Wherein by benefit value, go the technologies such as flying spot, denoising, edge detection to improve the resolution ratio of infrared speckle monocular depth image, pole
The earth improves the clarity of the RGB-D images after fusion.
In order to realize that the above at least purpose, one aspect of the present invention provide a monocular depth image capturing system comprising：
One infrared imaging unit；One depth converting unit；One speckle projecting unit；One color imaging member and a control unit, wherein
The speckle projecting unit receives the infrared light of reflection, obtains infrared image for projecting infrared light, the infrared imaging unit
The Infrared Image Information is converted to monocular depth image information, the colour imaging list by information, the depth converting unit
The RGB image and the monocular depth image co-registration are obtained colored monocular depth by member acquisition RGB image, described control unit
Image.
According to some embodiments, the monocular depth image capturing system, wherein described control unit is to the monocular
Depth image goes flying spot to handle.
According to some embodiments, the monocular depth image capturing system, wherein described control unit scheme the RGB
After carrying out denoising, by the RGB image and the monocular depth image co-registration.
According to some embodiments, the monocular depth image capturing system, wherein described control unit be equipped with to
A few output interface exports the color depth image in order to communicate to connect an electronic equipment to the electronic equipment.
According to some embodiments, the monocular depth image capturing system, wherein described control unit are to described infrared
Imaging unit, the color imaging member, speckle projecting unit transmission control signal, to control the infrared imaging list
Member, the work of the speckle projecting unit and the color imaging member.
According to some embodiments, the monocular depth image capturing system, wherein described control unit are to described infrared
Imaging unit, the color imaging member, the speckle projecting unit transmit synchronizing signal, to synchronize the infrared imaging list
First, the described color imaging member and the speckle projecting unit.
According to some embodiments, the monocular depth image capturing system, wherein described control unit is in image co-registration
During, the pixel to losing depth information in monocular depth image assigns depth initial value.
According to some embodiments, the monocular depth image capturing system, wherein the pixel to losing depth information
The mode for assigning initial value is to take the depth mean value of its surrounding pixel point as its depth value.
According to some embodiments, the monocular depth image capturing system, wherein described control unit is in blending image
During, monocular depth image and RGB image are integrated to obtain an entire image, entire image is decomposed into limited a image
Block (each tile size is m × m), according to formulaFind out away from
From image block PrK-1 nearest image block, wherein IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,It is average
Depth, α and β are the weight of color and depth respectively.
According to some embodiments, the monocular depth image capturing system, wherein described control unit is in blending image
During, by image block PrAnd this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt isIdeal matrix, the object edge in coloured image and monocular depth image is detected to obtain the RGB- of image block
D structural relations estimate matrix using parabolic regression modelOrder r.
According to some embodiments, described control unit can divide matrix according to the value of r during blending image
Solution：It is rightSubmatrix A and B, wherein W are obtained using change algorithm
It is mask code matrix, the element position for losing information is 0, remaining element is 1.
According to some embodiments, the monocular depth image capturing system, wherein described control unit is in image co-registration
During, matrix A is multiplied to obtain with BIts each row is the corresponding depth of k image block.
According to some embodiments, the monocular depth image capturing system, wherein described control unit is in image co-registration
During, to each pixel p, depth of the depth mean value of all image blocks of covering p as the pixel is taken, is thus obtained whole
The depth map of width image.
Another aspect of the present invention provides a monocular depth image processing method comprising step：
(A) an at least RGB image is acquired；
(B) an at least monocular depth image is acquired；With
(C) it merges the RGB image and forms a color depth image with the monocular depth image.
According to some embodiments, the monocular depth image processing method, wherein the step (B) includes step：It adopts
Collect an infrared image, the infrared image is converted into the monocular depth image.
According to some embodiments, the monocular depth image processing method, wherein the step (C) includes step：It is right
The RGB image carries out denoising, to obtain the RGB image of high quality.
According to some embodiments, the monocular depth image processing method, wherein the step (C) includes step：It is right
The monocular depth image carries out flying spot and handles.
According to some embodiments, the monocular depth image processing method, wherein the step (C) includes step：It is right
The pixel that depth information is lost in monocular depth image assigns initial value.
According to some embodiments, the monocular depth image processing method, wherein the mode for assigning initial value is：It takes
The depth mean value of the pixel surrounding pixel point of depth information is lost as its depth value.
According to some embodiments, the monocular depth image processing method, wherein the step (C) includes step：
(C1) it finds out K-1 nearest image block of range image block Pr and constitutes matrix M；
(C2) ideal matrix of estimation matrix MOrder r；
(C3) sub- square A, B are found out；With
(C4) matrix decomposition is run, depth is obtained.
According to some embodiments, the monocular depth image processing method, wherein in the step (C1), by monocular depth
Degree image and RGB image are integrated to obtain an entire image, and entire image is decomposed into limited a image block (each tile size
For m × m), according to formulaFind out range image block PrNearest
K-1 image block, wherein IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,It is mean depth, α and β are to be respectively
The weight of color and depth.
According to some embodiments, the monocular depth image processing method, wherein in the step (C2), by image block
PrAnd this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt isIdeal matrix, it is right
Object edge in coloured image and monocular depth image is detected to obtain the RGB-D structural relations of image block, uses parabolic
Line regression model estimates matrixOrder r.
According to some embodiments, the monocular depth image processing method, wherein in the step (C3), according to r's
Value can decompose matrix：It is rightIt is obtained using change algorithm
Submatrix A and B, wherein W is mask code matrix, and the element position for losing information is 0, remaining element is 1.
According to some embodiments, the monocular depth image processing method, wherein in the step (C4), by matrix A
It is multiplied to obtain with BIts each row is the corresponding depth of k image block.
According to some embodiments, the monocular depth image processing method, wherein the step (C) includes step：It is right
Each pixel p takes depth of the depth mean value of all image blocks of covering p as the pixel, thus obtains the depth of entire image
Degree figure.
Another aspect of the present invention provides a monocular depth image collecting device, wherein the monocular depth image collector
It sets and is run in a manner of monocular depth image capturing system above-mentioned.
Description of the drawings
Fig. 1 is the operation principle block diagram of monocular depth image capturing system according to a preferred embodiment of the present invention.
Fig. 2 is that the signal transmission work of the monocular depth image capturing system of above preferred embodiment according to the present invention is former
Reason figure.
Fig. 3 be above preferred embodiment according to the present invention monocular depth image capturing system monocular depth image and
RGB image fusion method block diagram.
Fig. 4 is the monocular depth image processing method block diagram of above preferred embodiment according to the present invention.
Specific implementation mode
It is described below for disclosing the present invention so that those skilled in the art can realize the present invention.It is excellent in being described below
Embodiment is selected to be only used as illustrating, it may occur to persons skilled in the art that other obvious modifications.It defines in the following description
The present invention basic principle can be applied to other embodiments, deformation scheme, improvement project, equivalent program and do not carry on the back
Other technologies scheme from the spirit and scope of the present invention.
It will be understood by those skilled in the art that the present invention exposure in, term " longitudinal direction ", " transverse direction ", "upper",
The orientation of the instructions such as "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom" "inner", "outside" or position are closed
System is to be based on the orientation or positional relationship shown in the drawings, and is merely for convenience of description of the present invention and simplification of the description, without referring to
Show or imply that signified device or element must have a particular orientation, with specific azimuth configuration and operation, therefore above-mentioned art
Language is not considered as limiting the invention.
Referring to figs. 1 to Fig. 3, depth image acquisition system 100 according to a preferred embodiment of the invention, the monocular depth
Degree image capturing system 100 acquires monocular depth image and coloured image respectively, by monocular depth image and color depth image
It is merged, obtains colored monocular depth image (RGB-D images).The monocular depth image capturing system 100 can be answered
For different equipment and different field, such as monocular depth camera, gesture control, 3D modeling, car radar and machine
The fields such as human visual system.
The monocular depth image capturing system 100 includes an infrared imaging unit 101, a color imaging member 102, one
Speckle projecting unit 103, a depth converting unit 107 and a control unit (MCU) 104.The infrared imaging unit 101, institute
It states color imaging member 102, the speckle projecting unit 103 and the depth converting unit 107 and is communicatively coupled to the control
Unit 104 processed.
The infrared imaging unit 101 and institute's speckle projecting unit 103 match acquisition infrared image, and by described
The infrared image is converted to monocular depth image by depth converting unit 107.In other words, the infrared imaging unit 101,
The speckle projecting unit 103 and the depth converting unit 107 match realization acquisition monocular depth image.It is described it is colored at
As unit 102 is for acquiring RGB image.Described control unit 104 is for handling image information.
Specifically, in the process of work, the speckle projecting unit 103 projects on infrared speckle a to target object,
The infrared imaging unit 101 obtains the real-time infrared image with infrared speckle characteristics, and image information is sent to institute
It states depth converting unit 107 and obtains monocular depth image.The color imaging member 102 obtains the cromogram of target object
Picture.Monocular depth image and coloured image are transferred into described control unit 104, and described control unit 104 is deep by the monocular
Degree image and the coloured image blend to obtain colored monocular depth image.
More specifically, the speckle projecting unit 103 projects infrared speckle light to target object, and supplement is provided and is shone
It is bright.After the RGB image sensor photosensitization of the color imaging member 102, opto-electronic conversion is carried out, to described control unit
104 output RGB images.The IR imaging sensors of the infrared imaging unit 101 respond the speckle reflected by the object
The infrared speckle light that projecting unit 103 projects carries out photosensitization and obtains Infrared Image Information, and by Infrared Image Information
It is sent to the depth converting unit 107.Infrared Image Information is changed into monocular depth by the depth image converting unit 107
Image information, and export deep image information to described control unit 104.Described control unit 104 first by it is described it is infrared at
As the monocular depth image that unit 101 is converted by the depth image converting unit 107 is handled, high-precision monocular is obtained
Depth image, and the RGB image of color imaging member output is handled, then by monocular depth image and described
RGB image carries out fusion treatment and obtains colored monocular depth image.
Citing ground, the infrared imaging unit 101 can be infrared photography module, and the color imaging member 102 can be with
For a natural light camera module, the speckle projecting unit 103 can be an infrared ray diffraction speckle projector, described control unit
104 can be an embeded processor.
It is noted that for traditional monocular depth camera, it is typically only capable to acquisition monocular depth image, monocular depth
Image reproducing is gray level image, and does not have color information, but the only resolution ratio of the monocular depth image of half-tone information
Relatively low, image clarity is poor, and is susceptible to hole in the image acquired, and picture quality is poor.And it will according to the present invention
The Color Image Fusion of acquisition obtains the monocular depth image with color information, i.e. RGB-D images in monocular depth image,
To improve the resolution ratio of monocular depth image, the clarity of image is improved, and compensates for traditional monocular depth image and adopts
There are the defects of hole in collection image, and more good user experience is provided for user.
Further, described control unit 104 is provided with an output interface 105, such as USB interface, such as USB2.0, USB3.0
Deng so as to which the monocular depth image is exported by the output interface 105.It such as, will be described by the USB interface
Image capturing system is connected to an electronic equipment, for example, desktop computer, laptop, tablet computer, personal digital assistant device,
Cell phone apparatus, wearable device, vehicle console, somatic sensation television game equipment etc., to realize image by the electronic equipment
It reproduces, and coordinates the electronic device works.
With reference to Fig. 2, described control unit 104 to the infrared imaging unit 101, the speckle projecting unit 103 and
The signal of transmission control in real time of the color imaging member 102, to control the infrared imaging unit 101, speckle projection list
Member 107 and the color imaging member 102 work, for example control the infrared imaging unit 101 and the colour imaging list
102 acquisition of member respectively acquires image information, and the Infrared Image Information of acquisition and the color image information are sent to institute
State control unit 104.Described control unit 104 is to the infrared imaging unit 101, the color imaging member 102 and institute
It states speckle projecting unit 103 and transmits synchronizing signal, to synchronize the infrared imaging unit 101,102 and of the color imaging member
The speckle projecting unit 103 improves Image Acquisition precision.That is, in the infrared imaging unit 101, the colour
Before imaging unit 102 and the speckle projecting unit 103 work, first to the infrared imaging unit 101, the colour imaging
Unit 102 is synchronous with the speckle projecting unit 103 progress signal.
As shown in figure 3, described control unit 104 illustrates the monocular depth image and the Color Image Fusion method
Figure.During image co-registration, first the monocular depth image that the infrared imaging unit 101 acquires is carried out at flying spot
Reason, that is, flying spot is carried out to infrared speckle monocular depth figure and is handled, removal depth has the picture of larger difference with adjacent pixels
Element.
Further, initial depth value is assigned to the pixel for losing depth information in monocular depth image, that is, take picture around it
The depth mean value of vegetarian refreshments is as its depth value.
Further, described control unit 104 carries out denoising to the RGB image, improves picture quality.
Further, monocular depth image and RGB image are integrated to obtain an entire image, entire image is decomposed into limited
A image block (each tile size is m × m), according to formula
Find out range image block PrK-1 nearest image block.Wherein, IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,
It is mean depth, α and β are the weights for being color and depth respectively (default value is respectively 0.4 and 30).
By image block PrAnd this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt isIdeal matrix (do not include noise and each pixel and do not lose depth information).It is deep to coloured image and monocular
Object edge in degree image is detected to obtain the RGB-D structural relations of image block, is estimated using parabolic regression model
MatrixOrder r.
Further, matrix can be decomposed according to the value of r：It is right
Submatrix A and B are obtained using change algorithm.Wherein W is mask code matrix, and the element position for losing information is 0, remaining element is 1.
Further, matrix A is multiplied to obtain with BIts each row is the corresponding depth of k image block.
Further, to each pixel p, depth of the depth mean value of all image blocks of covering p as the pixel is taken, finally
Obtain the depth map of entire image.
Thus it obtains, by the colored monocular depth image of RGB image and monocular depth image co-registration.
During the image co-registration of the present invention, by benefit value, the technologies such as flying spot, denoising, edge detection is gone to improve
The resolution ratio of infrared speckle monocular depth image greatly improves the clarity of the RGB-D images after fusion.
According to the abovementioned embodiments of the present invention, the present invention provides a monocular depth image collecting device comprising one is infrared
Photographing module, a colored photographing module, a speckle projector, a depth conversion module and a processor.The infrared photography mould
Block is used to infrared image being changed into depth image, the colored camera shooting for obtaining infrared image, the depth conversion module
Module is for obtaining RGB image, and the speckle projector is for projecting infrared speckle light, and the processor is for handling image
Information.
In this embodiment of the present invention, the monocular depth image collecting device includes a shell, the infrared photography
Module and the color acquisition photographing module and the speckle projector are installed in the shell respectively, in order to pass through
It states shell and fixed position is provided so that the infrared photography module, the colored photographing module and the speckle projector quilt
Stablize and fixes.The processor is arranged in the shell.The depth conversion module is integrated in the infrared photography module.
The infrared photography module and the colored photographing module are arranged at the both sides of the speckle projector respectively.It is special
Not, the both sides of the speckle projector are set to the infrared photography module and the colour imaging module symmetry.Certainly,
In other embodiments of the invention, the infrared imaging module, the colored photographing module and the speckle projector may be used also
To be other layouts, such as integrated setting.
In this embodiment in accordance with the invention, the depth conversion module and the infrared photography module are integrally disposed.
That is the depth conversion module and the infrared photography module integrate a module, it is co-located.The present invention's
In another embodiment, the infrared imaging module, the depth conversion module and the colour imaging module are all integrally disposed.
Further, the processor is an integrated circuit board, the inside of the shell is arranged at, to protect the processing
Device.
The monocular depth image collecting device is provided with an at least output interface, such as USB interface, so as to pass through
The output interface exports the RGB-D images.Such as, described image acquisition system is connected to one by the USB interface
Electronic equipment, for example, desktop computer, laptop, tablet computer, personal digital assistant device, cell phone apparatus, wearable device,
Vehicle console etc. to realize the reproduction of image by the electronic equipment, and coordinates the electronic device works.It is described defeated
Outgoing interface is communicatively coupled to described image processor.
During the monocular depth image collecting device works, the speckle projector projects infrared speckle to one
On target object, the infrared photography module obtains the real-time infrared image with infrared speckle characteristics.The colored camera shooting mould
Block obtains the coloured image of target object.The infrared image is transferred into the depth conversion module and obtains monocular depth figure
Picture.The monocular depth image and the coloured image are transferred into the processor, and the processor is by the monocular depth
Image and the coloured image blend to obtain RGB-D images.
Specifically, the speckle projector projects infrared speckle light to target object, and provides additional illumination.It is described
After the RGB image sensor photosensitization of colored photographing module, opto-electronic conversion is carried out, RGB image is exported to the processor.Institute
The IR imaging sensors for stating infrared photography module respond the infrared speckle for the speckle projector projection reflected by the object
Light carries out photosensitization, obtains infrared image, and infrared image is sent to the depth conversion module, obtain monocular
Depth image.The processor is first handled the infrared image of the infrared photography module, obtains the RGB figures of high quality
Picture, and handling monocular depth image, obtains high-precision monocular depth image, then by monocular depth image and RGB
Image carries out fusion treatment.
It is noted that in traditional monocular depth image camera, it is typically only capable to sampling depth image, and depth map
It is gray level image as reproducing, and does not have color information, and the resolution ratio of the only depth image of half-tone information is relatively low, image
Clarity it is poor, and usually there is hole in the image acquired, image quality is poor.And monocular depth figure according to the present invention
As harvester, by the Color Image Fusion of acquisition in monocular depth image, the monocular depth image with color information is obtained,
That is RGB-D images improve the clarity of image to improve the resolution ratio of depth image, and it is deep to compensate for traditional monocular
There are the defects of hole in degree image, and more good user experience is provided for user.
The processor is controlled to the infrared photography module, the speckle projector and the colored photographing module transmission
Signal processed to control the infrared photography module and the colored photographing module work, for example controls the infrared photography module
With the respective acquisition image information of the colored photographing module acquisition, and by the Infrared Image Information of acquisition and the coloured image
Information is sent to the processor.The processor is to the infrared photography module, the colored photographing module and described dissipates
The spot projector transmits synchronizing signal, to synchronize the infrared photography module, the colored photographing module and the speckle projector,
Improve Image Acquisition precision.That is, in the infrared photography module, the colored photographing module and the speckle projector
Before work, it is synchronous that signal first is carried out with the speckle projector to the infrared photography module, the colored photographing module.
With reference to Fig. 3, during image co-registration, first the infrared image that the infrared photography module acquires fly
Point processing, that is, carrying out flying spot to infrared speckle monocular depth figure is handled, and removal depth has larger difference with adjacent pixels
Pixel.
Further, initial depth value is assigned to the pixel for losing depth information in monocular depth image, that is, take picture around it
The depth mean value of vegetarian refreshments is as its depth value.
Further, the processor carries out noise reduction process to the RGB image, improves picture quality.
Further, monocular depth image and RGB image are integrated to obtain an entire image, entire image is decomposed into limited
A image block (each tile size is m × m), according to formula
Find out range image block PrK-1 nearest image block.Wherein, IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,
It is mean depth, α and β are the weights for being color and depth respectively (default value is respectively 0.4 and 30).
By image block PrAnd this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt isIdeal matrix (do not include noise and each pixel and do not lose depth information).It is deep to coloured image and monocular
Object edge in degree image is detected to obtain the RGB-D structural relations of image block, is estimated using parabolic regression model
MatrixOrder r.
Further, matrix can be decomposed according to the value of r：It is right
Submatrix A and B are obtained using change algorithm.Wherein W is mask code matrix, and the element position for losing information is 0, remaining element is 1.
Further, matrix A is multiplied to obtain with BIts each row is the corresponding depth of k image block.
Further, to each pixel p, depth of the depth mean value of all image blocks of covering p as the pixel is taken, finally
Obtain the depth map of entire image.
Thus it obtains, by the colored monocular depth image of RGB image and monocular depth image co-registration.
With reference to Fig. 4, according to the abovementioned embodiments of the present invention, the present invention provides a monocular depth image processing method 1000,
The method 1000 includes step：
1001：Acquire an at least RGB image；
1002：Acquire an at least monocular depth image；With
1003：It merges the RGB image and forms a color depth image with the monocular depth image.
The wherein described step 1003 includes step：Noise reduction process is carried out to the RGB image, to obtain high quality
The RGB image.
The wherein described step 1002 includes step：An infrared image is obtained, and infrared image is converted into monocular depth
Degree figure, such as infrared speckle monocular depth figure.
Further, the step 1003 includes step：Flying spot is carried out to the monocular depth image to handle.It is described to go
Flying spot is handled：Removing depth, there are the pixels of larger difference with adjacent pixels.
The step 1003 includes step：Pixel to losing depth information in monocular depth image assigns initial depth
Value, that is, take the depth mean value of its surrounding pixel point as its depth value.
Include wherein step in the step 1003：
10031：Find out range image block PrK-1 nearest image block constitutes matrix M；
10032：Estimate the ideal matrix of matrix MOrder r；
10033：Find out sub- square A, B；With
10034：Matrix decomposition is run, depth is obtained.
In the wherein described step 10031, monocular depth image and RGB image are integrated to obtain an entire image, by entire image
It is decomposed into limited a image block (each tile size is m × m), according to formula
Find out range image block PrK-1 nearest image block.Wherein, IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,
It is mean depth, α and β are the weights for being color and depth respectively (default value is respectively 0.4 and 30).
In the wherein described step 10032, by image block PrAnd this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt isIdeal matrix (do not include noise and each pixel and do not lose depth information).
Object edge in coloured image and monocular depth image is detected to obtain the RGB-D structural relations of image block, uses throwing
Object line regression model estimates matrixOrder r.
In the step 10033, matrix can be decomposed according to the value of r：It is rightSubmatrix A and B are obtained using change algorithm.Wherein W is mask code matrix, loses information
Element position is 0, remaining element is 1.
In the step 10034, matrix A is multiplied to obtain with BIts each row is the corresponding depth of k image block
Degree.
Further, to each pixel p, depth of the depth mean value of all image blocks of covering p as the pixel is taken, finally
Obtain the depth map of entire image.
By the above method, by benefit value, the technologies such as flying spot, denoising, edge detection is gone to improve infrared speckle monocular depth
The resolution ratio of image greatly improves the clarity of the RGB-D images after fusion.
It should be understood by those skilled in the art that the embodiment of the present invention shown in foregoing description and attached drawing is only used as illustrating
And it is not intended to limit the present invention.The purpose of the present invention has been fully and effectively achieved.The function and structural principle of the present invention exists
It shows and illustrates in embodiment, under without departing from the principle, embodiments of the present invention can have any deformation or modification.
Claims (26)
1. a monocular depth image capturing system, which is characterized in that including：
One infrared imaging unit；
One depth converting unit；
One speckle projecting unit；
One color imaging member；With
One control unit, wherein the speckle projecting unit receives reflection for projecting infrared light, the infrared imaging unit
Infrared light, obtains Infrared Image Information, and the Infrared Image Information is converted to monocular depth image by the depth converting unit
Information, the color imaging member acquire RGB image, and described control unit is by the RGB image and the monocular depth image
Fusion obtains colored monocular depth image.
2. monocular depth image capturing system according to claim 1, wherein described control unit is to the monocular depth
Image goes flying spot to handle.
3. monocular depth image capturing system according to claim 1, wherein described control unit to the RGB image into
After row denoising, by the RGB image and the monocular depth image co-registration.
4. monocular depth image capturing system according to claim 1, wherein described control unit are equipped at least one
Output interface exports the color depth image in order to communicate to connect an electronic equipment to the electronic equipment.
5. monocular depth image capturing system according to claim 1, wherein described control unit are to the infrared imaging
Unit, the color imaging member, speckle projecting unit transmission control signal, to control the infrared imaging unit, institute
State the work of speckle projecting unit and the color imaging member.
6. monocular depth image capturing system according to claim 1, wherein described control unit are to the infrared imaging
Unit, the color imaging member, the speckle projecting unit transmit synchronizing signal, to synchronize the infrared imaging unit, institute
State color imaging member and the speckle projecting unit.
7. monocular depth image capturing system according to any one of claims 1 to 6, wherein described control unit melt in image
During conjunction, the pixel to losing depth information in monocular depth image assigns depth initial value.
8. monocular depth image capturing system according to claim 7, wherein to losing depth letter in monocular depth image
The mode that the pixel of breath assigns initial value is to take the depth mean value of its surrounding pixel point as its depth value.
9. monocular depth image capturing system according to any one of claims 1 to 6, wherein described control unit are schemed in fusion
As during, monocular depth image and RGB image are integrated to obtain an entire image, entire image is decomposed into limited figure
As block (each tile size is m × m), according to formulaIt finds out
Range image block PrK-1 nearest image block, wherein IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,It is flat
Equal depth, α and β are the weight of color and depth respectively.
10. monocular depth image capturing system according to claim 9, wherein mistake of the described control unit in blending image
Cheng Zhong, by image block PrAnd this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt is's
Ideal matrix, the RGB-D structures for being detected to obtain image block to the object edge in coloured image and monocular depth image are closed
System, matrix is estimated using parabolic regression modelOrder r.
11. monocular depth image capturing system according to claim 10, wherein described control unit is in blending image
In the process, matrix can be decomposed according to the value of r：It is rightMake
Submatrix A and B are obtained with change algorithm, wherein W is mask code matrix, and the element position for losing information is 0, remaining element is 1.
12. monocular depth image capturing system according to claim 11, wherein described control unit is in image co-registration
In the process, matrix A is multiplied to obtain with BIts each row is the corresponding depth of k image block.
13. monocular depth image capturing system according to claim 12, wherein described control unit is in image co-registration
In the process, to each pixel p, depth of the depth mean value of all image blocks of covering p as the pixel is taken, whole picture is thus obtained
The depth map of image.
14. a monocular depth image processing method, which is characterized in that including step：
(A) an at least RGB image is acquired；
(B) an at least monocular depth image is acquired；With
(C) it merges the RGB image and forms a color depth image with the monocular depth image.
15. monocular depth image processing method according to claim 14, wherein the step (B) includes step：Acquisition
The infrared image is converted to the monocular depth image by one infrared image.
16. monocular depth image processing method according to claim 14, wherein the step (C) includes step：To institute
It states RGB image and carries out denoising, to obtain the RGB image of high quality.
17. monocular depth image processing method according to claim 14, wherein the step (C) includes step：To institute
It states monocular depth image and carries out flying spot and handle.
18. monocular depth image processing method according to claim 14, wherein the step (C) includes step：To list
The pixel that depth information is lost in mesh depth image assigns initial value.
19. monocular depth image processing method according to claim 18, wherein the mode for assigning initial value is：It takes and loses
The depth mean value of the pixel surrounding pixel point of depth information is lost as its depth value.
20. according to any monocular depth image processing method of claim 14 to 19, wherein the step (C) includes step
Suddenly：
(C1) range image block P is found outrK-1 nearest image block constitutes matrix M；
(C2) ideal matrix of estimation matrix MOrder r；
(C3) sub- square A, B are found out；With
(C4) matrix decomposition is run, depth is obtained.
21. monocular depth image processing method according to claim 20, wherein in the step (C1), by monocular depth
Image and RGB image are integrated to obtain an entire image, and entire image is decomposed into limited a image block, and (each tile size is
M × m), according to formulaFind out range image block PrNearest k-1
A image block, wherein IrAnd DrIt is block of pixels P respectivelyrRGB and depth information,Mean depth, α and β be color respectively
Color and depth weight.
22. monocular depth image processing method according to claim 20, wherein in the step (C2), by image block Pr
And this k-1 image block constitutes image block matrix by row dischargeAssuming thatIt isIdeal matrix, to coloured silk
Object edge in color image and monocular depth image is detected to obtain the RGB-D structural relations of image block, uses parabola
Regression model estimates matrixOrder r.
23. monocular depth image processing method according to claim 20, wherein in the step (C3), according to the value of r
Matrix can be decomposed：It is rightSon is obtained using change algorithm
Matrix A and B, wherein W are mask code matrixes, and the element position for losing information is 0, remaining element is 1.
24. monocular depth image processing method according to claim 20, wherein in the step (C4), by matrix A and B
Multiplication obtainsIts each row is the corresponding depth of k image block.
25. monocular depth image processing method according to claim 20, wherein the step (C) includes step：To every
A pixel p takes depth of the depth mean value of all image blocks of covering p as the pixel, thus obtains the depth of entire image
Figure.
26. a monocular depth image collecting device, which is characterized in that the monocular depth image collecting device is according to claim
The mode of 1 to 13 any monocular depth image capturing system is run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710007466.8A CN108280807A (en) | 2017-01-05 | 2017-01-05 | Monocular depth image collecting device and system and its image processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710007466.8A CN108280807A (en) | 2017-01-05 | 2017-01-05 | Monocular depth image collecting device and system and its image processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108280807A true CN108280807A (en) | 2018-07-13 |
Family
ID=62800582
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710007466.8A Pending CN108280807A (en) | 2017-01-05 | 2017-01-05 | Monocular depth image collecting device and system and its image processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108280807A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110852312A (en) * | 2020-01-14 | 2020-02-28 | 深圳飞科机器人有限公司 | Cliff detection method, mobile robot control method, and mobile robot |
| CN111768450A (en) * | 2020-06-10 | 2020-10-13 | 北京的卢深视科技有限公司 | Automatic detection method and device for line deviation of structured light camera based on speckle pattern |
| CN113643337A (en) * | 2021-08-05 | 2021-11-12 | 北京的卢深视科技有限公司 | Camera line deviation calibration method and device, electronic equipment and storage medium |
| CN114143419A (en) * | 2020-09-04 | 2022-03-04 | 聚晶半导体股份有限公司 | Dual-sensor camera system and depth map calculation method thereof |
| CN116188510A (en) * | 2023-04-25 | 2023-05-30 | 安徽皖欣环境科技有限公司 | Enterprise emission data acquisition system based on multiple sensors |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102622591A (en) * | 2012-01-12 | 2012-08-01 | 北京理工大学 | 3D (three-dimensional) human posture capturing and simulating system |
| TW201310277A (en) * | 2011-08-30 | 2013-03-01 | Serafim Technologies Inc | Three-dimensional human-machine interface system and method thereof |
| CN102999892A (en) * | 2012-12-03 | 2013-03-27 | 东华大学 | Intelligent fusion method for depth images based on area shades and red green blue (RGB) images |
| CN103106604A (en) * | 2013-01-23 | 2013-05-15 | 东华大学 | Three dimensional (3D) virtual fitting method based on somatosensory technology |
| CN104019761A (en) * | 2014-04-15 | 2014-09-03 | 北京农业信息技术研究中心 | Three-dimensional configuration obtaining device and method of corn plant |
-
2017
- 2017-01-05 CN CN201710007466.8A patent/CN108280807A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201310277A (en) * | 2011-08-30 | 2013-03-01 | Serafim Technologies Inc | Three-dimensional human-machine interface system and method thereof |
| CN102622591A (en) * | 2012-01-12 | 2012-08-01 | 北京理工大学 | 3D (three-dimensional) human posture capturing and simulating system |
| CN102999892A (en) * | 2012-12-03 | 2013-03-27 | 东华大学 | Intelligent fusion method for depth images based on area shades and red green blue (RGB) images |
| CN103106604A (en) * | 2013-01-23 | 2013-05-15 | 东华大学 | Three dimensional (3D) virtual fitting method based on somatosensory technology |
| CN104019761A (en) * | 2014-04-15 | 2014-09-03 | 北京农业信息技术研究中心 | Three-dimensional configuration obtaining device and method of corn plant |
Non-Patent Citations (2)
| Title |
|---|
| SI LU等: "Depth Enhancement via Low-rank Matrix Completion", 《2014 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》 * |
| 王一娇: "结构光深度图像获取算法研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110852312A (en) * | 2020-01-14 | 2020-02-28 | 深圳飞科机器人有限公司 | Cliff detection method, mobile robot control method, and mobile robot |
| CN110852312B (en) * | 2020-01-14 | 2020-07-17 | 深圳飞科机器人有限公司 | Cliff detection method, mobile robot control method, and mobile robot |
| CN111768450A (en) * | 2020-06-10 | 2020-10-13 | 北京的卢深视科技有限公司 | Automatic detection method and device for line deviation of structured light camera based on speckle pattern |
| CN111768450B (en) * | 2020-06-10 | 2023-10-13 | 合肥的卢深视科技有限公司 | Automatic detection method and device for structured light camera row deviation based on speckle pattern |
| CN114143419A (en) * | 2020-09-04 | 2022-03-04 | 聚晶半导体股份有限公司 | Dual-sensor camera system and depth map calculation method thereof |
| CN114143419B (en) * | 2020-09-04 | 2023-12-26 | 聚晶半导体股份有限公司 | Dual-sensor camera system and depth map calculation method thereof |
| CN113643337A (en) * | 2021-08-05 | 2021-11-12 | 北京的卢深视科技有限公司 | Camera line deviation calibration method and device, electronic equipment and storage medium |
| CN113643337B (en) * | 2021-08-05 | 2022-05-31 | 合肥的卢深视科技有限公司 | Camera line deviation calibration method and device, electronic equipment and storage medium |
| CN116188510A (en) * | 2023-04-25 | 2023-05-30 | 安徽皖欣环境科技有限公司 | Enterprise emission data acquisition system based on multiple sensors |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220182598A1 (en) | Systems, methods and apparatuses for stereo vision and tracking | |
| CN108280807A (en) | Monocular depth image collecting device and system and its image processing method | |
| JP5158223B2 (en) | 3D modeling apparatus, 3D modeling method, and program | |
| US10311648B2 (en) | Systems and methods for scanning three-dimensional objects | |
| CN106251399B (en) | A kind of outdoor scene three-dimensional rebuilding method and implementing device based on lsd-slam | |
| CN108269238B (en) | Depth image acquisition device, depth image acquisition system and image processing method thereof | |
| CN109615652A (en) | A kind of depth information acquisition method and device | |
| KR101566543B1 (en) | Method and system for mutual interaction using space information argumentation | |
| CN109453517B (en) | Virtual character control method and device, storage medium and mobile terminal | |
| US10966599B2 (en) | Endoscopic stereo matching method and apparatus using direct attenuation model | |
| TW202025719A (en) | Method, apparatus and electronic device for image processing and storage medium thereof | |
| KR101510312B1 (en) | 3D face-modeling device, system and method using Multiple cameras | |
| JP2012248221A (en) | Three-dimensional modeling device, three-dimensional modeling method, and program | |
| CN107710276B (en) | Method, system, and non-transitory computer readable medium for image processing | |
| CN108632506A (en) | A kind of microlens array imaging system | |
| CN115526983A (en) | Three-dimensional reconstruction method and related equipment | |
| CN106023073A (en) | Image splicing system | |
| CN111583386B (en) | Multi-view human body posture reconstruction method based on label propagation algorithm | |
| CN113348489A (en) | Image processing method and device | |
| CN103500471A (en) | Method for realizing high-resolution augmented reality system | |
| WO2015069063A1 (en) | Method and system for creating a camera refocus effect | |
| WO2022257120A1 (en) | Pupil position determination method, device and system | |
| CN110933290A (en) | Virtual photographing integrated system and method based on human-computer interaction | |
| CN110705487B (en) | Palm print acquisition equipment and method and image acquisition device thereof | |
| CN111652110A (en) | Image processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180713 |
|
| RJ01 | Rejection of invention patent application after publication |