CN109360268A - Surface optimization method and device for reconstructing dynamic objects - Google Patents
Surface optimization method and device for reconstructing dynamic objects Download PDFInfo
- Publication number
- CN109360268A CN109360268A CN201811150607.2A CN201811150607A CN109360268A CN 109360268 A CN109360268 A CN 109360268A CN 201811150607 A CN201811150607 A CN 201811150607A CN 109360268 A CN109360268 A CN 109360268A
- Authority
- CN
- China
- Prior art keywords
- color
- rebuilding
- pixel
- movement
- optimization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000010586 diagram Methods 0.000 claims abstract description 29
- 230000004927 fusion Effects 0.000 claims abstract description 10
- 238000005286 illumination Methods 0.000 claims description 24
- 238000005381 potential energy Methods 0.000 claims description 23
- 239000007787 solid Substances 0.000 claims description 18
- 230000000007 visual effect Effects 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000005259 measurement Methods 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000002310 reflectometry Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 3
- 244000062793 Sorghum vulgare Species 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 235000019713 millet Nutrition 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/55—Radiosity
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of surface optimization method and devices for rebuilding dynamic object, wherein this method comprises: rebuilding the color image sequence and range image sequence of object by individually holding the acquisition of RGBD camera；Current camera position is solved to each frame according to color image sequence and range image sequence, the movement for rebuilding object and ambient lighting is obtained, object current geometry and surface information is updated with the mode of fusion；Multiple key frames are picked out from color image sequence according to clarity, and optimize the pixel of key frame, the color image after making optimization meets preset condition with the color diagram being originally inputted in block of pixels；The geological information for rebuilding object is encrypted, the color diagram of preset value is met using resolution ratio, object is returned into the color diagram back projection of each key frame after optimization, obtains the geometry and the surface that rebuild object.This method optimizes the surface of object on dynamic object, obtains clear, accurate surface information, and widely applicable.
Description
Technical field
The present invention relates to computer vision and graphics techniques field, in particular to a kind of surface for rebuilding dynamic object is excellent
Change method and device.
Background technique
Three-dimensional reconstruction extremely important problem in computer vision and iconology, it is in film, animation, virtual existing
Have a wide range of applications with fields such as augmented realities space in fact.Depth camera and color camera can respectively obtain each pixel pair
The depth value and color value answered can be used to rebuild the geometry of object and surface, but all there is some for these data
Error, cause the body surface effect finally reconstructed can be poor.
There are some relevant methods that can carry out the optimization of surface texture to static object, but these methods are not suitable for
Dynamic object.Some methods can rebuild dynamic object, but the surface effect rebuild is poor.Some methods can obtain
To the preferable surface information of effect, but dynamic object can not be rebuild.The reconstruction of dynamic object is applied again for various
It is highly important, such as the movement of people is rebuild.The reconstruction of dynamic object in the related technology then more concern pair
The reconstruction of object of which movement ignores the various errors of equipment bring, causes the surface rebuild poor.
Summary of the invention
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, this method can an object of the present invention is to provide a kind of surface optimization method for rebuilding dynamic object
To optimize on dynamic object to the surface of object, clear, accurate surface information is obtained, it is widely applicable.
It is another object of the present invention to propose a kind of surface optimization device for rebuilding dynamic object.
In order to achieve the above objectives, one aspect of the present invention embodiment proposes a kind of surface optimization side for rebuilding dynamic object
Method, comprising the following steps: the color image sequence and range image sequence of object are rebuild by individually holding the acquisition of RGBD camera；
Current camera position is solved for each frame according to the color image sequence and the range image sequence, and is obtained described heavy
Movement and the ambient lighting of object are built, and updates object current geological information and surface information using the mode of fusion；
Multiple key frames are picked out from the color image sequence according to clarity, and the pixel of the multiple key frame are carried out excellent
Change, so that the color image after optimization meets preset condition with the color diagram being originally inputted in block of pixels, while in different frame
The solid colour of the same surface point of middle object；The geological information for rebuilding object is encrypted, to make full use of point
Resolution meets the color diagram of preset value, and object is returned in the color diagram back projection after each crucial frame optimization, to obtain
The geometry for rebuilding object and surface.
The surface optimization method of the reconstruction dynamic object of the embodiment of the present invention, passes through the table using block of pixels to dynamic object
Face optimizes, and first has to the geometry for solving object and movement, it is only necessary to using individually hand-held RGBD camera, then optimize
The pixel value of color image frame so that the body surface seen under each visual angle is consistent, thus obtain object it is final it is clear,
Accurate surface information, and it is widely applicable.
In addition, it is according to the above embodiment of the present invention rebuild dynamic object surface optimization method can also have it is following attached
The technical characteristic added:
Further, in one embodiment of the invention, the geological information for rebuilding object passes through in voxel space
Truncation signed distance function method be indicated, the movement of voxel obtains by the movement weighted sum of sparse node around,
Movement is expressed using double quaternary mixed number, and the ambient lighting is indicated using spheric harmonic function, in known illumination by reflecting
Rate and surface normal acquire the color of body surface.
Further, in one embodiment of the invention, described according to the color image sequence and the depth map
Picture sequence solves Current camera position for each frame, and obtains the movement for rebuilding object and ambient lighting, and use
The mode of fusion updates object current geological information and surface information, comprising: by described in the estimation of iteration closest approach algorithm
Current camera position；Construction potential-energy function carries out joint solution to movement W and illumination L, potential-energy function include two data item and
Two regular terms.
Wherein, the potential-energy function are as follows:
E1(W, L)=ωdEd(W)+ωcEc(W,L)+ωwEw(W)+ωlEl(L),
Wherein, L is illumination, and W is movement, ωd、ωc、ωw、ω1Respectively corresponding four weight coefficients, data item EdFor
The quadratic sum of the depth value difference at respective pixel after each pixel inputs in depth image value and model sport, data
Item EcFor color value difference of the model under current light at respective pixel after the value and movement of each pixel in color image
Quadratic sum, regular terms Ew, regular terms E1。
Further, in one embodiment of the invention, the calculation formula of the color image after the optimization are as follows:
E2(T, M)=ωSES(T)+ωTET(T)+ωMEM(T, M),
Wherein, ωS、ωT、ωMRespectively three weight coefficients, ESItem and ETItem measurement SiAnd TiBetween gap, letter
Number D (s, t) is the diversity factor of the block of pixels s and t of identical size, is defined as the quadratic sum of each pixel color difference, Tj(yj) be
Pixel, EMFor solid colour item, it is desirable that the solid colour that the same surface point of object is seen under different perspectives, it is desirable that Mi(xi)
The respective pixel T of color and different perspectivesj(yj) color is close, MiExpression currently optimizes obtained surface in SiUnder visual angle
Image, yjIt is MiIn pixel xiIn SjRespective pixel under visual angle, corresponds to the identical surface point of object, and corresponding relationship passes through it
The motion calculation of each frame of preceding solution obtains, Mi(xi)-Tj(yj) it is xiWith yjThe difference of color value, Wj(yj) it is yjConfidence level,
It is its normal direction and SjSquare of sight included angle cosine.
In order to achieve the above objectives, another aspect of the present invention embodiment proposes a kind of surface optimization dress for rebuilding dynamic object
It sets, comprising: acquisition module, for rebuilding the color image sequence and depth image of object by individually holding the acquisition of RGBD camera
Sequence；Module is obtained, for solving current phase for each frame according to the color image sequence and the range image sequence
Seat in the plane is set, and obtain the movement for rebuilding object and ambient lighting, and it is current to update using the mode of fusion object
Geological information and surface information；Choosing module, for picking out multiple keys from the color image sequence according to clarity
Frame, and the pixel of the multiple key frame is optimized so that the color image after optimization in block of pixels be originally inputted
Color diagram meet preset condition, while in different frame the same surface point of object solid colour；Optimization module is used for
The geological information for rebuilding object is encrypted, to make full use of resolution ratio to meet the color diagram of preset value, and will be each
Object returns in the color diagram back projection after a key frame optimization, to obtain the geometry for rebuilding object and surface.
The surface optimization device of the reconstruction dynamic object of the embodiment of the present invention, passes through the table using block of pixels to dynamic object
Face optimizes, and first has to the geometry for solving object and movement, it is only necessary to using individually hand-held RGBD camera, then optimize
The pixel value of color image frame so that the body surface seen under each visual angle is consistent, thus obtain object it is final it is clear,
Accurate surface information, and it is widely applicable.
In addition, it is according to the above embodiment of the present invention rebuild dynamic object surface optimization device can also have it is following attached
The technical characteristic added:
Further, in one embodiment of the invention, the geological information for rebuilding object passes through in voxel space
Truncation signed distance function method be indicated, the movement of voxel obtains by the movement weighted sum of sparse node around,
Movement is expressed using double quaternary mixed number, and the ambient lighting is indicated using spheric harmonic function, in known illumination by reflecting
Rate and surface normal acquire the color of body surface.
Further, in one embodiment of the invention, the acquisition module is further used for through iteration closest approach
Algorithm estimates the Current camera position, and constructs potential-energy function and carry out joint solution, potential-energy function to movement W and illumination L
Include two data item and two regular terms.
Wherein, the potential-energy function are as follows:
E1(W, L)=ωdEd(W)+ωcEc(W,L)+ωwEw(W)+ωlEl(L),
Wherein, L is illumination, and W is movement, ωd、ωc、ωw、ωlRespectively corresponding four weight coefficients, data item EdFor
The quadratic sum of the depth value difference at respective pixel after each pixel inputs in depth image value and model sport, data
Item EcFor color value difference of the model under current light at respective pixel after the value and movement of each pixel in color image
Quadratic sum, regular terms Ew, regular terms El。
Further, in one embodiment of the invention, the calculation formula of the color image after the optimization are as follows:
E2(T, M)=ωSES(T)+ωTET(T)+ωMEM(T, M),
Wherein, ωS、ωT、ωMRespectively three weight coefficients, ESItem and ETItem measurement SiAnd TiBetween gap, letter
Number D (s, t) is the diversity factor of the block of pixels s and t of identical size, is defined as the quadratic sum of each pixel color difference, Tj(yj) be
Pixel, EMFor solid colour item, it is desirable that the solid colour that the same surface point of object is seen under different perspectives, it is desirable that Mi(xi)
The respective pixel T of color and different perspectivesj(yj) color is close, MiExpression currently optimizes obtained surface in SiUnder visual angle
Image, yjIt is MiIn pixel xiIn SjRespective pixel under visual angle, corresponds to the identical surface point of object, and corresponding relationship passes through it
The motion calculation of each frame of preceding solution obtains, Mi(xi)-Tj(yj) it is xiWith yjThe difference of color value, Wj(yj) it is yjConfidence level,
It is its normal direction and SjSquare of sight included angle cosine.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is the surface optimization method flow diagram according to the reconstruction dynamic object of one embodiment of the invention；
Fig. 2 is the flow chart that dynamic object reconstruction is carried out according to the input RGBD image sequence of one embodiment of the invention；
Fig. 3 is the surface optimization apparatus structure schematic diagram according to the reconstruction dynamic object of one embodiment of the invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
The surface optimization method and dress of reconstruction dynamic object proposed according to embodiments of the present invention is described with reference to the accompanying drawings
It sets, describes the surface optimization method of reconstruction dynamic object proposed according to embodiments of the present invention with reference to the accompanying drawings first.
Fig. 1 is the surface optimization method flow diagram of the reconstruction dynamic object of one embodiment of the invention.
As shown in Figure 1, the surface optimization method of the reconstruction dynamic object the following steps are included:
In step s101, the color image sequence and depth image of object are rebuild by individually holding the acquisition of RGBD camera
Sequence.
Further, in one embodiment of the invention, the geological information for rebuilding object passes through cutting in voxel space
Disconnected signed distance function method is indicated, and the movement of voxel is obtained by the movement weighted sum of sparse node around, movement
It is expressed using double quaternary mixed number, and ambient lighting is indicated using spheric harmonic function, in known illumination by reflectivity and surface
Normal direction acquires the color of body surface.
In step s 102, Current camera position is solved for each frame according to color image sequence and range image sequence
It sets, and obtains the movement for rebuilding object and ambient lighting, and update the current geological information of object using the mode of fusion
And surface information.
It further, in one embodiment of the invention, can also include: current by the estimation of iteration closest approach algorithm
Camera position；Construction potential-energy function carries out joint solution to movement W and illumination L, and potential-energy function includes two data item and two
Regular terms.
Wherein, potential-energy function are as follows:
E1(W, L)=ωdEd(W)+ωcEc(W,L)+ωwEw(W)+ωlEl(L),
Wherein, L is illumination, and W is movement, ωd、、ωc、ωw、ωlRespectively corresponding four weight coefficients, data item Ed
The quadratic sum of the depth value difference at respective pixel after the value and model sport that are inputted for each pixel in depth image, number
According to item EcFor color value difference of the model under current light at respective pixel after the value and movement of each pixel in color image
The quadratic sum of value, regular terms Ew, regular terms El。
Specifically, using the depth image of input and color image sequence, Current camera position is solved to each frame, is solved
The movement of object and ambient lighting, and object current geological information and surface information are updated using the mode of fusion.
In step s 103, multiple key frames are picked out from color image sequence according to clarity, and to multiple keys
The pixel of frame optimizes, so that the color image after optimization meets default item in block of pixels with the color diagram being originally inputted
Part, at the same in different frame the same surface point of object solid colour.
In other words, some key frames are picked out from colour sequential according to clarity, and to the pixel of these key frames
It optimizes.Wherein, it is desirable that the color image after optimization will approach in block of pixels with the color diagram being originally inputted, while also want
The color of the same surface point of object in different frame is asked to be consistent.
In step S104, the geological information for rebuilding object is encrypted, to make full use of resolution ratio to meet preset value
Color diagram, and object is returned into the color diagram back projection after each crucial frame optimization, to obtain rebuilding the geometry and table of object
Face.
That is, the geometry of object is encrypted, to make full use of the higher color diagram of resolution ratio, and by each
Object returns in color diagram back projection after crucial frame optimization, obtains the final surface of object.
Further, in one embodiment of the invention, the calculation formula of the color image after optimization are as follows:
E2(T, M)=ωSEs(T)+ωTET(T)+ωMEM(T, M),
Wherein, ωS、ωT、ωMRespectively three weight coefficients, EsItem and ETItem measurement SiAnd TiBetween gap, letter
Number D (s, t) is the diversity factor of the block of pixels s and t of identical size, is defined as the quadratic sum of each pixel color difference, Tj(yj) be
Pixel, EMFor solid colour item, it is desirable that the solid colour that the same surface point of object is seen under different perspectives, it is desirable that Mi(xi)
The respective pixel T of color and different perspectivesj(yj) color is close, MiExpression currently optimizes obtained surface in SiUnder visual angle
Image, yjIt is MiIn pixel xiIn SjRespective pixel under visual angle, corresponds to the identical surface point of object, and corresponding relationship passes through it
The motion calculation of each frame of preceding solution obtains, Mi(xi)-Tj(yj) it is then xiWith yjThe difference of color value, Wj(yj) it is yjIt is credible
Degree, is its normal direction and SjSquare of sight included angle cosine.
The embodiment of the present invention can offline optimize its surface, and surface is poor when solving dynamic object reconstruction asks
Topic, it is applied widely, the various objects such as people and object can be rebuild, and only need using single hand
Hold RGBD camera.Specifically, the input of the embodiment of the present invention is individually to hold the collected color of RGBD camera and depth map sequence
Column, final output by reconstruction object geometry and surface.
Advantage and implementation method the characteristics of embodiment in order to preferably explain the present invention, it is below in conjunction with attached drawing and specifically real
Example is described in detail.
As shown in Fig. 2, the embodiment of the present invention mainly includes three steps:
(1) it solves movement correlative and updates object model.In this step, according to each frame depth of input and
Color image solves camera position, movement and illumination, and updates the geometry and surface information of object.Wherein, the geometry of object
Using truncation signed distance function (Truncate Signed Distance Field, TSDF) method in voxel space come into
Row indicates that the movement of voxel is obtained by the movement weighted sum of sparse node around, and movement is expressed using double quaternary mixed number, light
According to using spheric harmonic function to indicate, the color of body surface can be acquired by reflectivity and surface normal in known illumination.Firstly, wanting
The position of camera is estimated by iteration closest approach algorithm (Iterative Closest Point, ICP), then constructs potential energy letter
Several couples of movement W and illumination L carry out joint solution, and potential-energy function includes two data item and two regular terms:
E1(W, L)=ωdEd(W)+ωcEc(W,L)+ωwEw(W)+ωlEl(L)
Wherein, ωd、ωc、ωw、ωlRespectively corresponding four weight coefficients, data item EdFor each in depth image
The quadratic sum of the depth value difference at respective pixel after the value and model sport of pixel input, so that after model sport more
Close to collected depth image, data item EcIt is model after the value and movement of each pixel in color image in current light
The quadratic sum of color value difference at lower respective pixel, so that model is under current light more close to color diagram after movement
Picture, regular terms EwIt is required that the movement of each node node adjacent thereto is close as far as possible, regular terms ElIt is required that the light of illumination and previous frame
According to approaching as far as possible, movement and illumination are alternately solved when minimizing potential-energy function, another is fixed when solving one, is finally used deep
Degree image and color image are updated the TSDF value of voxel each in model and the value of reflectivity.
(2) it selects and optimizes color key frame.It is chosen according to the clarity of image every some frames in color image sequence
Select a color image frame as key frame Si(i=1 ..., n) constructs potential-energy function to obtain the face after its corresponding optimization
Chromatic graph Ti(i=1 ..., n):
E2(T, M)=ωSES(T)+ωTET(T)+ωMEM(T,M)
Wherein, ωS、ωT、ωMRespectively three weight coefficients, ESItem and ETItem measurement SiAnd TiBetween gap, letter
Number D (s, t) is the diversity factor of the block of pixels s and t of identical size, is defined as the quadratic sum of each pixel color difference, to calculate
ES, firstly, will be to all SiIn block of pixels find TiIn with the smallest block of pixels of its diversity factor, then all differences degree carried out
Summation, ETCalculating it is similar to its, EMItem requires the same object table millet cake color under different perspectives consistent, MiIndicate current
Optimize obtained surface in SiImage under visual angle, yjIt is MiIn pixel xiIn SjRespective pixel under visual angle, correspondence are identical
Object table millet cake, Mi(xi)-Tj(yj) it is then xiWith yjThe difference of color value.Wj(yj) it is yjConfidence level, be its normal direction and SjSight
Square of included angle cosine solves T and M when minimizing potential-energy function, another is fixed when solving one, every time alternately will be again
Look for the smallest respective pixel block of a diversity factor.
(3) final surface is obtained with the color key frame after optimization.Wherein, before obtaining final surface, firstly,
The geometry of object is stored and encrypted with the grid configuration of tri patch, then for each encrypted net
Its key frame T after each optimization is found on lattice vertexiIn corresponding color, be weighted according still further to the weight in (2) flat
, the final surface of object is obtained.
The surface optimization method of the reconstruction dynamic object of the embodiment of the present invention, passes through the table using block of pixels to dynamic object
Face optimizes, and first has to the geometry for solving object and movement, it is only necessary to using individually hand-held RGBD camera, then optimize
The pixel value of color image frame so that the body surface seen under each visual angle is consistent, thus obtain object it is final it is clear,
Accurate surface information, and it is widely applicable.
Referring next to the surface optimization device for the reconstruction dynamic object that attached drawing description proposes according to embodiments of the present invention.
Fig. 3 is the surface optimization apparatus structure schematic diagram of the reconstruction dynamic object of one embodiment of the invention.
As shown in figure 3, the surface optimization device 10 of the reconstruction dynamic object include: acquisition module 100, obtain module 200,
Choosing module 300 and optimization module 400.
Wherein, acquisition module 100 be used for by individually hold RGBD camera acquisition rebuild object color image sequence and
Range image sequence.Module 200 is obtained to be used to solve currently each frame according to color image sequence and range image sequence
Camera position, and obtain the movement for rebuilding object and ambient lighting, and update using the mode of fusion current several of object
What information and surface information.Choosing module 300 is used to pick out multiple key frames from color image sequence according to clarity, and
The pixel of multiple key frames is optimized, so that the color image after optimization is expired in block of pixels with the color diagram being originally inputted
Sufficient preset condition, at the same in different frame the same surface point of object solid colour.Optimization module 400 will be for that will rebuild object
The geological information of body is encrypted, to make full use of resolution ratio to meet the color diagram of preset value, and by each crucial frame optimization
Object returns in color diagram back projection afterwards, to obtain rebuilding geometry and the surface of object.The optimization device 10 of the embodiment of the present invention can
To optimize on dynamic object to the surface of object, clear, accurate surface information is obtained, and widely applicable.
Further, in one embodiment of the invention, the geological information for rebuilding object passes through cutting in voxel space
Disconnected signed distance function method is indicated, and the movement of voxel is obtained by the movement weighted sum of sparse node around, movement
It is expressed using double quaternary mixed number, and ambient lighting is indicated using spheric harmonic function, in known illumination by reflectivity and surface
Normal direction acquires the color of body surface.
Further, in one embodiment of the invention, module 200 is obtained to be further used for counting recently by iteration
Method estimates Current camera position, and constructs potential-energy function and carry out joint solution to movement W and illumination L, and potential-energy function includes two
Data item and two regular terms.
Wherein, potential-energy function are as follows:
E1(W, L)=ωdEd(W)+ωcEc(W,L)+ωwEw(W)+ωlEl(L),
Wherein, L is illumination, and W is movement, ωd、ωc、ωw、ωlRespectively corresponding four weight coefficients, data item EdFor
The quadratic sum of the depth value difference at respective pixel after each pixel inputs in depth image value and model sport, data
Item EcFor color value difference of the model under current light at respective pixel after the value and movement of each pixel in color image
Quadratic sum, regular terms EwIt is required that the movement of each node node adjacent thereto is close as far as possible, regular terms ElIt is required that illumination and upper one
The illumination of frame is close as far as possible.
Further, in one embodiment of the invention, the calculation formula of the color image after optimization are as follows:
E2(T, M)=ωSES(T)+ωTET(T)+ωMEM(T, M),
Wherein, ωS、ωT、ωMRespectively three weight coefficients, ESItem and ETItem measurement SiAnd TiBetween gap, letter
Number D (s, t) is the diversity factor of the block of pixels s and t of identical size, is defined as the quadratic sum of each pixel color difference, Tj(yj) be
Pixel, EMFor solid colour item, it is desirable that the solid colour that the same surface point of object is seen under different perspectives, it is desirable that Mi(xi)
The respective pixel T of color and different perspectivesj(yj) color is close, MiExpression currently optimizes obtained surface in SiUnder visual angle
Image, yjIt is MiIn pixel xiIn SjRespective pixel under visual angle, corresponds to the identical surface point of object, and corresponding relationship passes through it
The motion calculation of each frame of preceding solution obtains, Mi(xi)-Tj(yj) it is then xiWith yjThe difference of color value, Wj(yj) it is yjIt is credible
Degree, is its normal direction and SjSquare of sight included angle cosine.
It should be noted that the aforementioned explanation to the surface optimization embodiment of the method for rebuilding dynamic object is also applied for
The device of the embodiment, details are not described herein again.
The surface optimization device of the reconstruction dynamic object of the embodiment of the present invention, passes through the table using block of pixels to dynamic object
Face optimizes, and first has to the geometry for solving object and movement, it is only necessary to using individually hand-held RGBD camera, then optimize
The pixel value of color image frame so that the body surface seen under each visual angle is consistent, thus obtain object it is final it is clear,
Accurate surface information, it is widely applicable.
In addition, term " first ", " second " are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance
Or implicitly indicate the quantity of indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, the meaning of " plurality " is at least two, such as two, three
It is a etc., unless otherwise specifically defined.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It can be combined in any suitable manner in one or more embodiment or examples.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the feature of different embodiments or examples described in this specification and different embodiments or examples
It closes and combines.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of surface optimization method for rebuilding dynamic object, which comprises the following steps:
The color image sequence and range image sequence of object are rebuild by individually holding the acquisition of RGBD camera；
Current camera position is solved for each frame according to the color image sequence and the range image sequence, and obtains institute
The movement for rebuilding object and ambient lighting are stated, and updates the current geological information of object and surface letter using the mode of fusion
Breath；
Pick out multiple key frames from the color image sequence according to clarity, and to the pixel of the multiple key frame into
Row optimization, so that the color image after optimization meets preset condition with the color diagram being originally inputted in block of pixels, while not
The solid colour of the same surface point of middle object at same frame；And
The geological information for rebuilding object is encrypted, to make full use of resolution ratio to meet the color diagram of preset value, and will
Object returns in the color diagram back projection after each crucial frame optimization, to obtain the geometry for rebuilding object and surface.
2. the surface optimization method according to claim 1 for rebuilding dynamic object, which is characterized in that the reconstruction object
Geological information is indicated by the truncation signed distance function method in voxel space, and the movement of voxel is by sparse section around
The movement weighted sum of point obtains, and movement is expressed using double quaternary mixed number, and the ambient lighting uses spheric harmonic function table
Show, acquires the color of body surface by reflectivity and surface normal in known illumination.
3. the surface optimization method according to claim 1 or 2 for rebuilding dynamic object, which is characterized in that described according to institute
It states color image sequence and the range image sequence and Current camera position is solved for each frame, and obtain the reconstruction object
Movement and ambient lighting, and object current geological information and surface information are updated using the mode of fusion, comprising:
The Current camera position is estimated by iteration closest approach algorithm；
Construction potential-energy function carries out joint solution to movement W and illumination L, and potential-energy function includes two data item and two canonicals
?.
4. the surface optimization method according to claim 3 for rebuilding dynamic object, which is characterized in that the potential-energy function
Are as follows:
E1(W, L)=ωdEd(W)+ωcEc(W, L)+ωwEw(W)+ωlEl(L),
Wherein, L is illumination, and W is movement, ωd、ωc、ωw、ωlRespectively corresponding four weight coefficients, data item EdFor depth
The quadratic sum of the depth value difference at respective pixel after each pixel inputs in image value and model sport, data item Ec
For after the value and movement of each pixel in color image color value difference of the model under current light at respective pixel it is flat
Fang He, regular terms Ew, regular terms El。
5. the surface optimization method according to claim 4 for rebuilding dynamic object, which is characterized in that the face after the optimization
The calculation formula of chromatic graph picture are as follows:
E2(T, M)=ωSES(T)+ωTET(T)+ωMEM(T, M),
Wherein, ωS、ωT、ωMRespectively three weight coefficients, ESItem and ETItem measurement SiAnd TiBetween gap, function D (s,
T) be identical size block of pixels s and t diversity factor, be defined as the quadratic sum of each pixel color difference, Tj(yj) it is pixel,
EMFor solid colour item, it is desirable that the solid colour that the same surface point of object is seen under different perspectives, it is desirable that Mi(xi) color with
The respective pixel T of different perspectivesj(yj) color is close, MiExpression currently optimizes obtained surface in SiImage under visual angle, yj
It is MiIn pixel xiIn SjRespective pixel under visual angle corresponds to the identical surface point of object, what corresponding relationship solved before
The motion calculation of each frame obtains, Mi(xi)-Tj(yj) it is xiWith yjThe difference of color value, Wj(yj) it is yjConfidence level, be its normal direction
With SjSquare of sight included angle cosine.
6. a kind of surface optimization device for rebuilding dynamic object, which is characterized in that comprise the following modules:
Acquisition module, for rebuilding the color image sequence and depth image sequence of object by individually holding the acquisition of RGBD camera
Column；
Module is obtained, for solving Current camera for each frame according to the color image sequence and the range image sequence
Position, and obtain the movement for rebuilding object and ambient lighting, and update using the mode of fusion current several of object
What information and surface information；
Choosing module, for picking out multiple key frames from the color image sequence according to clarity, and to the multiple
The pixel of key frame optimizes, and presets so that the color image after optimization meets in block of pixels with the color diagram being originally inputted
Condition, at the same in different frame the same surface point of object solid colour；And
Optimization module, for encrypting the geological information for rebuilding object, to make full use of resolution ratio to meet preset value
Color diagram, and object is returned into the color diagram back projection after each crucial frame optimization, to obtain the object of rebuilding
Geometry and surface.
7. the surface optimization device according to claim 6 for rebuilding dynamic object, which is characterized in that the reconstruction object
Geological information is indicated by the truncation signed distance function method in voxel space, and the movement of voxel is by sparse section around
The movement weighted sum of point obtains, and movement is expressed using double quaternary mixed number, and the ambient lighting uses spheric harmonic function table
Show, acquires the color of body surface by reflectivity and surface normal in known illumination.
8. the surface optimization device according to claim 6 or 7 for rebuilding dynamic object, which is characterized in that the acquisition mould
Block is further used for estimating the Current camera position by iteration closest approach algorithm, and constructs potential-energy function to movement W and light
Joint solution is carried out according to L, potential-energy function includes two data item and two regular terms.
9. the surface optimization device according to claim 8 for rebuilding dynamic object, which is characterized in that the potential-energy function
Are as follows:
E1(W, L)=ωdEd(W)+ωcEc(W, L)+ωwEw(W)+ωlEl(L),
Wherein, L is illumination, and W is movement, ωd、ωc、ωw、ωlRespectively corresponding four weight coefficients, data item EdFor depth
The quadratic sum of the depth value difference at respective pixel after each pixel inputs in image value and model sport, data item Ec
For after the value and movement of each pixel in color image color value difference of the model under current light at respective pixel it is flat
Fang He, regular terms Ew, regular terms El。
10. the surface optimization device according to claim 9 for rebuilding dynamic object, which is characterized in that after the optimization
The calculation formula of color image are as follows:
E2(T, M)=ωSES(T)+ωTET(T)+ωMEM(T, M),
Wherein, ωS、ωT、ωMRespectively three weight coefficients, ESItem and ETItem measurement SiAnd TiBetween gap, function D (s,
T) be identical size block of pixels s and t diversity factor, be defined as the quadratic sum of each pixel color difference, Tj(yj) it is pixel,
EMFor solid colour item, it is desirable that the solid colour that the same surface point of object is seen under different perspectives, it is desirable that Mi(xi) color with
The respective pixel T of different perspectivesj(yj) color is close, MiExpression currently optimizes obtained surface in SiImage under visual angle, yj
It is MiIn pixel xiIn SjRespective pixel under visual angle corresponds to the identical surface point of object, what corresponding relationship solved before
The motion calculation of each frame obtains, Mi(xi)-Tj(yj) it is xiWith yjThe difference of color value, Wj(yj) it is yjConfidence level, be its normal direction
With SjSquare of sight included angle cosine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811150607.2A CN109360268B (en) | 2018-09-29 | 2018-09-29 | Surface optimization method and device for reconstructing dynamic object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811150607.2A CN109360268B (en) | 2018-09-29 | 2018-09-29 | Surface optimization method and device for reconstructing dynamic object |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109360268A true CN109360268A (en) | 2019-02-19 |
| CN109360268B CN109360268B (en) | 2020-04-24 |
Family
ID=65348123
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811150607.2A Active CN109360268B (en) | 2018-09-29 | 2018-09-29 | Surface optimization method and device for reconstructing dynamic object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109360268B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110223351A (en) * | 2019-05-30 | 2019-09-10 | 杭州蓝芯科技有限公司 | A kind of depth camera localization method based on convolutional neural networks |
| CN111754615A (en) * | 2020-05-07 | 2020-10-09 | 清华大学 | Real-time reconstruction method and device for high-quality texture |
| CN114119839A (en) * | 2022-01-24 | 2022-03-01 | 阿里巴巴（中国）有限公司 | Three-dimensional model reconstruction and image generation method, equipment and storage medium |
| US20230290057A1 (en) * | 2022-03-10 | 2023-09-14 | Nvidia Corporation | Action-conditional implicit dynamics of deformable objects |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101976455A (en) * | 2010-10-08 | 2011-02-16 | 东南大学 | Color image three-dimensional reconstruction method based on three-dimensional matching |
| CN102609977A (en) * | 2012-01-12 | 2012-07-25 | 浙江大学 | Depth integration and curved-surface evolution based multi-viewpoint three-dimensional reconstruction method |
| CN104318569A (en) * | 2014-10-27 | 2015-01-28 | 北京工业大学 | Space salient region extraction method based on depth variation model |
| US20160040985A1 (en) * | 2014-08-05 | 2016-02-11 | Sho NAGAI | Specimen measuring device and computer program product |
| CN105654492A (en) * | 2015-12-30 | 2016-06-08 | 哈尔滨工业大学 | Robust real-time three-dimensional (3D) reconstruction method based on consumer camera |
| CN106875482A (en) * | 2017-01-13 | 2017-06-20 | 浙江大学 | A kind of positioning simultaneously and dense three-dimensional rebuilding method |
| CN107680073A (en) * | 2016-08-02 | 2018-02-09 | 富士通株式会社 | The method and apparatus of geometrical reconstruction object |
| CN107845134A (en) * | 2017-11-10 | 2018-03-27 | 浙江大学 | A kind of three-dimensional rebuilding method of the single body based on color depth camera |
-
2018
- 2018-09-29 CN CN201811150607.2A patent/CN109360268B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101976455A (en) * | 2010-10-08 | 2011-02-16 | 东南大学 | Color image three-dimensional reconstruction method based on three-dimensional matching |
| CN102609977A (en) * | 2012-01-12 | 2012-07-25 | 浙江大学 | Depth integration and curved-surface evolution based multi-viewpoint three-dimensional reconstruction method |
| US20160040985A1 (en) * | 2014-08-05 | 2016-02-11 | Sho NAGAI | Specimen measuring device and computer program product |
| CN104318569A (en) * | 2014-10-27 | 2015-01-28 | 北京工业大学 | Space salient region extraction method based on depth variation model |
| CN105654492A (en) * | 2015-12-30 | 2016-06-08 | 哈尔滨工业大学 | Robust real-time three-dimensional (3D) reconstruction method based on consumer camera |
| CN107680073A (en) * | 2016-08-02 | 2018-02-09 | 富士通株式会社 | The method and apparatus of geometrical reconstruction object |
| CN106875482A (en) * | 2017-01-13 | 2017-06-20 | 浙江大学 | A kind of positioning simultaneously and dense three-dimensional rebuilding method |
| CN107845134A (en) * | 2017-11-10 | 2018-03-27 | 浙江大学 | A kind of three-dimensional rebuilding method of the single body based on color depth camera |
Non-Patent Citations (2)
| Title |
|---|
| 吴宣沛等: "《基于迭代反投影的彩色图像超分辨率重建》", 《计算机与数学工程》 * |
| 彭敏放等: "《一种基于深度和颜色的3D表面重建方法》", 《计算机工程》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110223351A (en) * | 2019-05-30 | 2019-09-10 | 杭州蓝芯科技有限公司 | A kind of depth camera localization method based on convolutional neural networks |
| CN110223351B (en) * | 2019-05-30 | 2021-02-19 | 杭州蓝芯科技有限公司 | Depth camera positioning method based on convolutional neural network |
| CN111754615A (en) * | 2020-05-07 | 2020-10-09 | 清华大学 | Real-time reconstruction method and device for high-quality texture |
| CN114119839A (en) * | 2022-01-24 | 2022-03-01 | 阿里巴巴（中国）有限公司 | Three-dimensional model reconstruction and image generation method, equipment and storage medium |
| US20230290057A1 (en) * | 2022-03-10 | 2023-09-14 | Nvidia Corporation | Action-conditional implicit dynamics of deformable objects |
| US12165258B2 (en) * | 2022-03-10 | 2024-12-10 | Nvidia Corporation | Action-conditional implicit dynamics of deformable objects |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109360268B (en) | 2020-04-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113160375B (en) | Three-dimensional reconstruction and camera pose estimation method based on multi-task learning algorithm | |
| CN109360268A (en) | Surface optimization method and device for reconstructing dynamic objects | |
| US20060232584A1 (en) | Stereoscopic picture generating apparatus | |
| EP3057066B1 (en) | Generation of three-dimensional imagery from a two-dimensional image using a depth map | |
| CN104376552B (en) | A kind of virtual combat method of 3D models and two dimensional image | |
| CN102521586B (en) | High-resolution three-dimensional face scanning method for camera phone | |
| KR20030002995A (en) | Method for image-based representation and rendering of three-dimensional object and animated three-dimensional object | |
| CN101520902A (en) | System and method for low cost motion capture and demonstration | |
| CN114049464B (en) | Reconstruction method and device of three-dimensional model | |
| Weiskopf et al. | Explanatory and illustrative visualization of special and general relativity | |
| US20060152579A1 (en) | Stereoscopic imaging system | |
| CN109218706B (en) | A method for generating stereoscopic vision images from a single image | |
| US12080012B2 (en) | Systems and methods for low compute high-resolution depth map generation using low-resolution cameras | |
| CN108932725A (en) | Scene flows estimation method based on convolutional neural networks | |
| CN103077552A (en) | Three-dimensional displaying method based on multi-view video | |
| CN102361497B (en) | A display method of spatial stereoscopic video | |
| Zhu et al. | Eye contact in video conference via fusion of time-of-flight depth sensor and stereo | |
| CN115937482A (en) | Holographic scene dynamic construction method and system capable of adapting to screen size | |
| Stoker et al. | Analyzing pathfinder data using virtual reality and superresolved imaging | |
| CN108898550A (en) | Image split-joint method based on the fitting of space triangular dough sheet | |
| US7432878B1 (en) | Methods and systems for displaying three-dimensional images | |
| CN108769644B (en) | Binocular animation stylized rendering method based on deep learning | |
| CN106991637A (en) | The method that multiresolution light field is decomposed is realized in a kind of utilization GPU parallel computations | |
| JP2001222723A (en) | Method and device for generating stereoscopic image | |
| CN116029919A (en) | An intelligent optimization method for compressed 3D light field display |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |