CN110084845A - Deformation Prediction method, apparatus and computer readable storage medium - Google Patents
Deformation Prediction method, apparatus and computer readable storage medium Download PDFInfo
- Publication number
- CN110084845A CN110084845A CN201910365833.0A CN201910365833A CN110084845A CN 110084845 A CN110084845 A CN 110084845A CN 201910365833 A CN201910365833 A CN 201910365833A CN 110084845 A CN110084845 A CN 110084845A
- Authority
- CN
- China
- Prior art keywords
- network
- prediction
- deformation
- deformation prediction
- target object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The invention discloses a kind of Deformation Prediction methods, by obtaining the depth image of target object and inputting three-dimensional reconstruction network, to obtain the threedimensional model that the three-dimensional reconstruction network is generated based on the depth image；Then the threedimensional model and the conditional vector generated based on target object characteristic are inputted into Deformation Prediction network, obtains the prediction model of deformation of Deformation Prediction network output；Task is executed based on the prediction model of deformation later.The invention also discloses a kind of Deformation Prediction device and computer readable storage mediums.This method inherits the extensive attribute for generating confrontation network, this means that network can in the case where the single depth views of given object reconstructed object entire three-dimensional appearance, and it is opposite with traditional finite element method, this method does not need largely to calculate cost, realize sufficiently fast Deformation Prediction speed, can be used for including robot grab article in real time including numerous real-time applications.
Description
Technical field
The present invention relates to field of artificial intelligence more particularly to a kind of Deformation Prediction method, apparatus and computer-readable
Storage medium.
Background technique
Key areas of the deep learning as current machine learning suffers from weight in directions such as image recognition, speech analysis
It applies.Deep learning is from neural network evolution, and deep learning is substantially the artificial neural network of deep layer, it is not one
The isolated technology of item, but the synthesis of the multiple fields such as mathematics, statistical machine learning, computer science and artificial neural network.
Up to the present, oneself warp of people is directed to all kinds of different problems such as classification, detection, identification and devises a variety of effective networks
Structure.
Ambient enviroment is understood and is interacted, is one of the significant capability that robot needs to possess, and including machine
The precondition that many application technologies including people's technology and augmented reality are achieved, especially in prediction target object
In the real-time deformation situation under by external force.It is blocked, interferes however, being frequently run onto target object in practice
The situations such as object and localized loss, difficulty increases when above situation all can cause to predict the deformation of target object in real time,
Such as robot can only be observed when manipulating an object target object part and non-integral.In addition, current technology exists
It is all often that all target objects are regarded as to static during handling machine people grabs object problem, it is non-deformable
Hard objects.However in reality most of objects be all it is deformable, different journeys can be generated in the effect for encountering external force
The deformation of degree.If can be inferred that object issuable deformation when being crawled, facilitate robot to being crawled object
Preparatory control judgement is generated, to generate the control strategy of adaptation.
The depth image that traditional Deformation Prediction method gets robot directly inputs generation network, is generating network
Two steps of prediction of reconstruction and three-dimensional stereo model deflection of the middle completion from partial-depth image to three-dimensional stereo model,
Due to generating network more suitable for high-dimensional model treatment, there are also to be improved for this process.
Above content is only used to facilitate the understanding of the technical scheme, and is not represented and is recognized that above content is existing skill
Art.
Summary of the invention
The main purpose of the present invention is to provide a kind of Deformation Prediction method, apparatus and computer readable storage medium, purports
Depth image is directly handled solving tradition generation network, the technical problem for causing resulting prediction model of deformation effect bad.
To achieve the above object, the present invention provides a kind of Deformation Prediction method, is applied to Deformation Prediction system, the deformation
Forecasting system includes three-dimensional reconstruction network and Deformation Prediction network, the Deformation Prediction method the following steps are included:
It obtains the depth image of target object and inputs three-dimensional reconstruction network, be based on institute to obtain the three-dimensional reconstruction network
State the threedimensional model of depth image generation；
The threedimensional model and the conditional vector generated based on target object characteristic are inputted into Deformation Prediction network, become
Shape predicts the prediction model of deformation of network output；
Task is executed based on the prediction model of deformation.
Preferably, it is described obtain target object depth image and input three-dimensional reconstruction network, to obtain the Three-dimensional Gravity
The step of threedimensional model that establishing network is generated based on the depth image includes:
The 2.5D depth image is inputted three-dimensional reconstruction network by the 2.5D depth image for obtaining target object；
Three-dimensional reconstruction processing is carried out to the 2.5D depth image in three-dimensional reconstruction network, generates corresponding three-dimensional mould
Type.
Preferably, the Deformation Prediction network includes generating network and differentiating network, described by the threedimensional model and base
Deformation Prediction network is inputted in the conditional vector that target object characteristic generates, obtains the Deformation Prediction mould of Deformation Prediction network output
The step of type includes:
First condition vector is generated based on target object characteristic, the threedimensional model and the first condition vector are inputted
Network is generated, the prediction of distortion amount of target object is obtained；
The true strain amount for obtaining target object, using the prediction of distortion amount of the generation network output as differentiation network
Input value determines whether the prediction of distortion amount for generating network output meets expectation in conjunction with the true strain amount；
When the prediction of distortion amount of the generation network output meets desired, determine that the prediction of distortion amount is Deformation Prediction
Model.
Preferably, the differentiation network successively includes five warp laminations, wherein each warp lamination includes one and swashs
Function living；
The generation network includes variation autocoder and decoder；
The variation autocoder successively includes five convolutional layers, a full articulamentum and two μ and σ layers,
In, each warp lamination includes an activation primitive；
The decoder successively includes five warp laminations, wherein each warp lamination includes an activation primitive.
Preferably, described the step of generating first condition vector based on target object characteristic, includes:
Determine the material property of target object and the stress condition of target object；
First condition vector is generated based on the material property and stress condition.
Preferably, the true strain amount for obtaining target object makees the prediction of distortion amount of the generation network output
Determine whether the prediction of distortion amount for generating network output meets the phase in conjunction with the true strain amount for the input value for differentiating network
The step of prestige includes:
Obtain the true strain amount that physical modeler generates；
The prediction of distortion amount input of the generation network output is differentiated into network, in conjunction with the true strain amount and is based on mesh
The second condition vector that object properties generate is marked, determines whether the prediction of distortion amount meets expectation.
Preferably, it is described obtain target object depth image and input three-dimensional reconstruction network, to obtain the Three-dimensional Gravity
Before the step of threedimensional model that establishing network is generated based on the depth image, the method also includes:
Several real-world object deformation patterns are collected, corresponding image data base is established；
Deformation Prediction network is trained based on described image database, so that the prediction energy of the Deformation Prediction network
Power gradually increases.
Preferably, described that Deformation Prediction network is trained based on described image database, so that the Deformation Prediction
The step of predictive ability of network gradually increases include:
Several voxel grids and conditional vector are obtained based on image data base, if generating by physical modeler by described
Several ground truths pair of dry voxel grid and conditional vector composition；
With several described ground truths to training Deformation Prediction network, the related ginseng of Deformation Prediction network is continued to optimize
Number, so that the predictive ability of the Deformation Prediction network gradually increases.
In addition, to achieve the above object, the present invention also provides a kind of Deformation Prediction device, the Deformation Prediction device packet
It includes: memory, processor and being stored in the Deformation Prediction program that can be run on the memory and on the processor, it is described
Deformation Prediction program realizes the step of Deformation Prediction method as described in any one of the above embodiments when being executed by the processor.
In addition, to achieve the above object, it is described computer-readable the present invention also provides a kind of computer readable storage medium
It is stored with Deformation Prediction program on storage medium, realizes when the Deformation Prediction program is executed by processor such as any of the above-described institute
The step of Deformation Prediction method stated.
The present invention program, by obtaining the depth image of target object and inputting three-dimensional reconstruction network, to obtain described three
Dimension rebuilds the threedimensional model that network is generated based on the depth image；Then by the threedimensional model and based on target object characteristic
The conditional vector of generation inputs Deformation Prediction network, obtains the prediction model of deformation of Deformation Prediction network output；It is based on institute later
It states prediction model of deformation and executes task；This method inherits the extensive attribute for generating confrontation network, it means that network can be
The entire three-dimensional appearance of reconstructed object in the case where the single depth views of given object, and with traditional finite element method phase
Instead, this method does not need largely to calculate cost, realizes sufficiently fast Deformation Prediction speed, can be used for including that robot is real-time
Grab numerous real-time applications including article.
Detailed description of the invention
Fig. 1 is the structure of the affiliated terminal of Deformation Prediction device in the hardware running environment that the embodiment of the present invention is related to
Schematic diagram；
Fig. 2 is the flow diagram of Deformation Prediction method first embodiment of the present invention；
Fig. 3 is that the connection of three-dimensional reconstruction network and Deformation Prediction network in Deformation Prediction method first embodiment of the present invention is closed
It is schematic diagram；
Fig. 4 is to obtain the depth image of target object in Deformation Prediction method second embodiment of the present invention and input Three-dimensional Gravity
Establishing network, refinement process the step of to obtain threedimensional model that the three-dimensional reconstruction network is generated based on the depth image are shown
It is intended to；
Fig. 5 is in Deformation Prediction method 3rd embodiment of the present invention by the threedimensional model and based on the life of target object characteristic
At conditional vector input Deformation Prediction network, obtain Deformation Prediction network output prediction model of deformation the step of refinement stream
Journey schematic diagram；
Fig. 6 is generation network and differentiation schematic network structure in Deformation Prediction method 3rd embodiment of the present invention；
Fig. 7 is the flow diagram in Deformation Prediction method fourth embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
As shown in Figure 1, the affiliated terminal structure of device that Fig. 1 is the hardware running environment that the embodiment of the present invention is related to shows
It is intended to.
The terminal of that embodiment of the invention can be PC, be also possible to smart phone, tablet computer, E-book reader, MP3
(Moving Picture Experts Group Audio Layer III, dynamic image expert's compression standard audio level 3)
Player, MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert's compression standard sound
Frequency level 3) the packaged type terminal device having a display function such as player, portable computer.
As shown in Figure 1, the terminal may include: processor 1001, such as CPU, network interface 1004, user interface
1003, memory 1005, communication bus 1002.Wherein, communication bus 1002 is for realizing the connection communication between these components.
User interface 1003 may include display screen (Display), input unit such as keyboard (Keyboard), optional user interface
1003 can also include standard wireline interface and wireless interface.Network interface 1004 optionally may include that the wired of standard connects
Mouth, wireless interface (such as WI-FI interface).Memory 1005 can be high speed RAM memory, be also possible to stable memory
(non-volatile memory), such as magnetic disk storage.Memory 1005 optionally can also be independently of aforementioned processor
1001 storage device.
Optionally, terminal can also include camera, RF (Radio Frequency, radio frequency) circuit, sensor, audio
Circuit, WiFi module etc..Wherein, sensor such as optical sensor, motion sensor and other sensors.Specifically, light
Sensor may include ambient light sensor and proximity sensor, wherein ambient light sensor can according to the light and shade of ambient light come
The brightness of display screen is adjusted, proximity sensor can close display screen and/or backlight when mobile terminal is moved in one's ear.As
One kind of motion sensor, gravity accelerometer can detect the size of (generally three axis) acceleration in all directions, quiet
Size and the direction that can detect that gravity when only, the application that can be used to identify mobile terminal posture are (such as horizontal/vertical screen switching, related
Game, magnetometer pose calibrating), Vibration identification correlation function (such as pedometer, tap) etc.；Certainly, mobile terminal can also match
The other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor are set, details are not described herein.
It will be understood by those skilled in the art that the restriction of the not structure paired terminal of terminal structure shown in Fig. 1, can wrap
It includes than illustrating more or fewer components, perhaps combines certain components or different component layouts.
As shown in Figure 1, as may include that operating system, network are logical in a kind of memory 1005 of computer storage medium
Believe module, Subscriber Interface Module SIM and Deformation Prediction program.
In terminal shown in Fig. 1, network interface 1004 is mainly used for connecting background server, carries out with background server
Data communication；User interface 1003 is mainly used for connecting client (user terminal), carries out data communication with client；And processor
1001 can be used for calling the Deformation Prediction program stored in memory 1005.
In the present embodiment, Deformation Prediction device includes: memory 1005, processor 1001 and is stored in the memory
On 1005 and the Deformation Prediction program that can be run on the processor 1001, wherein processor 1001 calls memory 1005
When the Deformation Prediction program of middle storage, and execute following operation:
It obtains the depth image of target object and inputs three-dimensional reconstruction network, be based on institute to obtain the three-dimensional reconstruction network
State the threedimensional model of depth image generation；
The threedimensional model and the conditional vector generated based on target object characteristic are inputted into Deformation Prediction network, become
Shape predicts the prediction model of deformation of network output；
Task is executed based on the prediction model of deformation.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
The 2.5D depth image is inputted three-dimensional reconstruction network by the 2.5D depth image for obtaining target object；
Three-dimensional reconstruction processing is carried out to the 2.5D depth image in three-dimensional reconstruction network, generates corresponding three-dimensional mould
Type.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
First condition vector is generated based on target object characteristic, the threedimensional model and the first condition vector are inputted
Network is generated, the prediction of distortion amount of target object is obtained；
The true strain amount for obtaining target object, using the prediction of distortion amount of the generation network output as differentiation network
Input value determines whether the prediction of distortion amount for generating network output meets expectation in conjunction with the true strain amount；
When the prediction of distortion amount of the generation network output meets desired, determine that the prediction of distortion amount is Deformation Prediction
Model.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
The differentiation network successively includes five warp laminations, wherein each warp lamination includes an activation primitive；
The generation network includes variation autocoder and decoder；
The variation autocoder successively includes five convolutional layers, a full articulamentum and two μ and σ layers,
In, each warp lamination includes an activation primitive；
The decoder successively includes five warp laminations, wherein each warp lamination includes an activation primitive.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
Determine the material property of target object and the stress condition of target object；
First condition vector is generated based on the material property and stress condition.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
Obtain the true strain amount that physical modeler generates；
The prediction of distortion amount input of the generation network output is differentiated into network, in conjunction with the true strain amount and is based on mesh
The second condition vector that object properties generate is marked, determines whether the prediction of distortion amount meets expectation.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
Several real-world object deformation patterns are collected, corresponding image data base is established；
Deformation Prediction network is trained based on described image database, so that the prediction energy of the Deformation Prediction network
Power gradually increases.
Further, processor 1001 can call the Deformation Prediction program stored in memory 1005, also execute following
Operation:
Several voxel grids and conditional vector are obtained based on image data base, if generating by physical modeler by described
Several ground truths pair of dry voxel grid and conditional vector composition；
With several described ground truths to training Deformation Prediction network, the related ginseng of Deformation Prediction network is continued to optimize
Number, so that the predictive ability of the Deformation Prediction network gradually increases.
First embodiment of the invention provides a kind of Deformation Prediction method, is applied to Deformation Prediction system, the Deformation Prediction
System includes three-dimensional reconstruction network and Deformation Prediction network, is Deformation Prediction method first embodiment of the present invention referring to Fig. 2, Fig. 2
Flow diagram, the Deformation Prediction method includes:
Step S10 obtains the depth image of target object and inputs three-dimensional reconstruction network, to obtain the Three-dimensional Gravity networking
The threedimensional model that network is generated based on the depth image；
Depth image is also range image, refer to using distance (depth) value of each point in from image acquisition device to scene as
The image of pixel value.Acquisition methods have: laser radar Depth Imaging method, computer stereo vision imaging, coordinate measuring machine method, not
That the Schlieren method, Structure light method.The material property of the target object includes Young's modulus and Poisson's ratio, the target object by
Power situation includes amount of force and position of action point.Conditional vector is to carry out discretization to multiple physical conditions, is obtained multiple
Corresponding numerical value set, is indicated with the form of vector.For example, when robot needs to judge the deflection of target obstacle,
Obtain the size of the power applied to it, it, can be by the way that these three features be compiled when the material of the active position of power and barrier
Code is only hot vector (f, a, m), including each feature including f, a, m indicates the condition of discretization with binary form；
Fixed number of bits each feature can also be arranged.In the present embodiment, it is shooting function by robot itself
The depth image of target object is got, this image not only includes target object, further includes surrounding environment.Depth image
It is converted by certain processing, such as coordinate, may be calculated point cloud data, while the point cloud data met certain condition can also
Using inverse as depth image.
As shown in figure 3, the structure of three-dimensional reconstruction network and Deformation Prediction network is cascade form.The work of three-dimensional reconstruction network
Be by 2.5D depth image by three-dimensional reconstruction generate have stereochemical structure threedimensional model, it is later that the threedimensional model is defeated
Enter into Deformation Prediction network, so that Deformation Prediction network handles the threedimensional model, and then obtains prediction model of deformation.
In this embodiment, robot realization is helped to grab in such a way that three-dimensional reconstruction network and Deformation Prediction network are connected in series
Prediction when taking object to object deflection.When predicting the deflection of target object, which is shot, is obtained
One depth image, usually 2.5D depth image, obtains the 2.5D depth image of target object, by the 2.5D depth image
Input three-dimensional reconstruction network；Three-dimensional reconstruction processing is carried out to the 2.5D depth image in three-dimensional reconstruction network, generates and corresponds to
Threedimensional model.
The threedimensional model and the conditional vector generated based on target object characteristic are inputted Deformation Prediction net by step S20
Network obtains the prediction model of deformation of Deformation Prediction network output；
It interacts what robot and circumstances not known generated without constraint, can possess as the common-sense mankind's intuition pushes away
Reason ability be it is necessary, can especially know target object generated reaction when by external force.To everyday objects
Physical characteristic carry out modeling be autonomous robot basic premise.We have proposed a new generations to fight network, can
Deformation of the target object under by external force is predicted from the threedimensional model of a target object.The network is based on generating confrontation net
Network, and be trained on the different object sets generated by physics finite element model simulator.Network is generated mainly by certainly
Dynamic encoder and decoder composition, for the ease of the internal structure of reconstructed voxel grid, the autocoder in the present invention is being passed
There is great-jump-forward connection between the encoder and decoder of system.
Using certain way by target object characteristic encoding at conditional vector, above-mentioned target object characteristic includes target object
Material property and target object stress condition.The material property of target object or stress condition are encoded into continuously
Using conditional vector as the input of form, rather than according to the discontinuous classification of rank.
Step S30 executes task based on the prediction model of deformation.
When getting the prediction of distortion amount with generation network output, can executing for task includes: manipulation task, crawl
Task, ground deformation assessment and other in the end-to-end academic environment of intuitive physical model predictive power to non-rigid object
Influence task.
The Deformation Prediction method proposed in the present embodiment, by obtaining the depth image of target object and inputting three-dimensional reconstruction
Network, to obtain the threedimensional model that the three-dimensional reconstruction network is generated based on the depth image；Then by the threedimensional model
Deformation Prediction network is inputted with the conditional vector generated based on target object characteristic, the deformation for obtaining the output of Deformation Prediction network is pre-
Survey model；Task is executed based on the prediction model of deformation later；This method inherits the extensive attribute for generating confrontation network, this
Mean network can in the case where the single depth views of given object reconstructed object entire three-dimensional appearance, and with biography
The finite element method of system realizes sufficiently fast Deformation Prediction speed, can use on the contrary, this method does not need largely to calculate cost
In including robot grab article in real time including numerous real-time applications.
Based on first embodiment, the second embodiment of Deformation Prediction method of the present invention is proposed, referring to Fig. 4, step S10 packet
It includes:
Step S11 obtains the 2.5D depth image of target object, and the 2.5D depth image is inputted Three-dimensional Gravity networking
Network；
Step S12 carries out three-dimensional reconstruction processing to the 2.5D depth image in three-dimensional reconstruction network, generates corresponding
Threedimensional model.
Obtain the 2.5D depth image of target object；Voxelization processing is carried out to the 2.5D depth image, is corresponded to
The first voxel grid.Voxelization processing is the process for converting depth image to voxel grid, voxel grid namely one
3D point cloud.The definition of point cloud is from when beam of laser is irradiated to body surface, and the laser reflected can carry orientation, distance
Etc. information.If laser beam is scanned according to certain track, the laser point information of reflection will be recorded in scanning, due to
Scanning is extremely fine, then can obtain a large amount of laser point, thus can form laser point cloud.Point cloud format has * .las；*
.pcd；* .txt etc..Depth image may be calculated point cloud data by coordinate conversion；Regular and necessary information point cloud number
According to can be using inverse as depth image.
The Deformation Prediction method proposed in the present embodiment will be described by obtaining the 2.5D depth image of target object
2.5D depth image inputs three-dimensional reconstruction network；Then Three-dimensional Gravity is carried out to the 2.5D depth image in three-dimensional reconstruction network
Processing is built, corresponding threedimensional model is generated；The present invention predicts what it was generated by obtaining the 2.5D depth map of target object
Deformation, realizes the reconstruction of accurate threedimensional model.
Based on first embodiment, the 3rd embodiment of Deformation Prediction method of the present invention is proposed, referring to Fig. 5, Deformation Prediction net
Network includes generating network and differentiating that network, step S20 include:
Step S21 generates first condition vector based on target object characteristic, by the threedimensional model and the first condition
Vector input generates network, obtains the prediction of distortion amount of target object；
Step S22 obtains the true strain amount of target object, using the prediction of distortion amount of the generation network output as sentencing
The input value of other network determines whether the prediction of distortion amount for generating network output meets expectation in conjunction with the true strain amount；
Step S23 determines that the prediction of distortion amount is when the prediction of distortion amount of the generation network output meets desired
Prediction model of deformation.
Determine the material property of target object and the stress condition of target object；Based on the material property and stress
Situation generates first condition vector, and the first condition vector is inputted and generates network.The material property of target object includes mesh
The Young's modulus and Poisson's ratio of object materials are marked, the stress condition of target object includes the size and work of external force suffered by target object
With point.
As shown in fig. 6, the generation network includes variation autocoder and decoder；The variation autocoder according to
Secondary includes five convolutional layers, a full articulamentum and two μ and σ layers, wherein each warp lamination includes an activation
Function；The decoder successively includes five warp laminations, wherein each warp lamination includes an activation primitive.
Voxel grid is inputted after generating network, initially enters and divide autocoder part partially, available one one-dimensional
Vector is merged with based on first condition vector made of target object characteristic encoding, one after being merged it is one-dimensional to
Amount, input decoder obtain the prediction of distortion amount generated by decoder processes.
As shown in fig. 6, differentiating that network successively includes five warp laminations, wherein each warp lamination includes one and swashs
Function living.
In conjunction with the true strain amount and the second condition vector generated based on target object characteristic, determine that the prediction becomes
Whether shape amount meets expectation.Second condition vector be modified so as to prediction of distortion amount with voxel grid form and true
Real deflection is connected.Since the deflection of prediction target object is a high-dimensional problem, differentiate that network can export one
A intensive vector differentiates the similarity as a result, namely between prediction of distortion amount and true strain amount to represent.According to this phase
Judge to generate whether the prediction of distortion amount that network exports meets expectation like degree result.
The Deformation Prediction method proposed in the present embodiment will by generating first condition vector based on target object characteristic
The threedimensional model and first condition vector input generate network, obtain the prediction of distortion amount of target object；Then it obtains
The true strain amount of target object, using it is described generation network output prediction of distortion amount as differentiation network input value, in conjunction with
The true strain amount, determines whether the prediction of distortion amount for generating network output meets expectation；It is defeated in the generation network later
When prediction of distortion amount out meets desired, determine that the prediction of distortion amount is prediction model of deformation；The method of the present invention utilizes intuition
Physical model predicts real-time deformation situation of three-dimension object when by external force, by the material property and stress condition of the object
As conditional vector, it is added to depth variation autocoder and antagonistic training network, from the material property of target object
The property of learning objective body enables the network in the method for the present invention to promote on a large scale, mitigates to a large amount of training datas
Demand.
Based on first embodiment, propose the fourth embodiment of Deformation Prediction method of the present invention, reference Fig. 7, step S10 it
Before, the method also includes:
Step S40 collects several real-world object deformation patterns, establishes corresponding image data base；
Step S50 is trained based on described image database to network is generated, so that the prediction energy for generating network
Power gradually increases.
Several voxel grids and conditional vector are obtained based on image data base, if generating by physical modeler by described
Several ground truths pair of dry voxel grid and conditional vector composition；With several described ground truths pair and differentiate network
Common training generates network, continues to optimize the relevant parameter for generating network, so that the predictive ability for generating network gradually increases
By force.
Physical modeler can simulate the rule of various object of which movement in real world, can be by preset picture database
It is stored in the memory of itself, when being trained to generation network, several opposite true value is generated based on picture database
Right, which forms to by three-dimensional point cloud and conditional vector, converts corresponding body by software for three-dimensional point cloud later
Plain grid.By the ground truth of physical engine generation to generation network is input to, in the predictive information for obtaining generating network output
When, by the predictive information and above-mentioned ground truth to as the input for differentiating network, predictive information be exactly by generation network over the ground
Face true value is to the predicted value estimated according to certain rule.Later by between differentiation network query function predictive information and ground truth pair
Difference, updated according to the difference differentiate network loss function in corresponding parameter.Meanwhile more according to above-mentioned loss function
The relevant parameter of newly-generated network, so that the predictive ability for generating network gradually increases, that is, the predictive information exported is more next
Closer to the ground truth pair of input, reach the degree for differentiating that network can not be differentiated.
The machine learning method of the intuition physics proposed in the present embodiment, by collecting several real-world object deformation patterns,
Establish corresponding image data base；It is then based on described image database and is trained to network is generated, so that the generation net
The predictive ability of network gradually increases；The present invention uses the structure system of cascade form, and conditional vector can be made as generation network
A part with network inputs value is differentiated, realizes effective network training mode.
In addition, the embodiment of the present invention also proposes a kind of computer readable storage medium, the computer readable storage medium
On be stored with Deformation Prediction program, following operation is realized when the Deformation Prediction program is executed by processor:
It obtains the depth image of target object and inputs three-dimensional reconstruction network, be based on institute to obtain the three-dimensional reconstruction network
State the threedimensional model of depth image generation；
The threedimensional model and the conditional vector generated based on target object characteristic are inputted into Deformation Prediction network, become
Shape predicts the prediction model of deformation of network output；
Task is executed based on the prediction model of deformation.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
The 2.5D depth image is inputted three-dimensional reconstruction network by the 2.5D depth image for obtaining target object；
Three-dimensional reconstruction processing is carried out to the 2.5D depth image in three-dimensional reconstruction network, generates corresponding three-dimensional mould
Type.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
First condition vector is generated based on target object characteristic, the threedimensional model and the first condition vector are inputted
Network is generated, the prediction of distortion amount of target object is obtained；
The true strain amount for obtaining target object, using the prediction of distortion amount of the generation network output as differentiation network
Input value determines whether the prediction of distortion amount for generating network output meets expectation in conjunction with the true strain amount；
When the prediction of distortion amount of the generation network output meets desired, determine that the prediction of distortion amount is Deformation Prediction
Model.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
The generation network includes variation autocoder and decoder；
The variation autocoder successively includes five convolutional layers, a full articulamentum and two μ and σ layers,
In, each warp lamination includes an activation primitive；
The decoder successively includes five warp laminations, wherein each warp lamination includes an activation primitive.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
Determine the material property of target object and the stress condition of target object；
First condition vector is generated based on the material property and stress condition.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
Obtain the true strain amount that physical modeler generates；
The prediction of distortion amount input of the generation network output is differentiated into network, in conjunction with the true strain amount and is based on mesh
The second condition vector that object properties generate is marked, determines whether the prediction of distortion amount meets expectation.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
Several real-world object deformation patterns are collected, corresponding image data base is established；
Deformation Prediction network is trained based on described image database, so that the prediction energy of the Deformation Prediction network
Power gradually increases.
Further, following operation is also realized when the Deformation Prediction program is executed by processor:
Several voxel grids and conditional vector are obtained based on image data base, if generating by physical modeler by described
Several ground truths pair of dry voxel grid and conditional vector composition；
With several described ground truths to training Deformation Prediction network, the related ginseng of Deformation Prediction network is continued to optimize
Number, so that the predictive ability of the Deformation Prediction network gradually increases.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the system that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or system institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or system.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that terminal device (it can be mobile phone,
Computer, server, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of Deformation Prediction method is applied to Deformation Prediction system, which is characterized in that the Deformation Prediction system includes three-dimensional
Rebuild network and Deformation Prediction network, the Deformation Prediction method the following steps are included:
It obtains the depth image of target object and inputs three-dimensional reconstruction network, be based on the depth to obtain the three-dimensional reconstruction network
Spend the threedimensional model that image generates；
The threedimensional model and the conditional vector generated based on target object characteristic are inputted into Deformation Prediction network, it is pre- to obtain deformation
The prediction model of deformation of survey grid network output；
Task is executed based on the prediction model of deformation.
2. Deformation Prediction method as described in claim 1, which is characterized in that the depth image for obtaining target object is simultaneously defeated
Enter three-dimensional reconstruction network, packet the step of to obtain threedimensional model that the three-dimensional reconstruction network is generated based on the depth image
It includes:
The 2.5D depth image is inputted three-dimensional reconstruction network by the 2.5D depth image for obtaining target object；
Three-dimensional reconstruction processing is carried out to the 2.5D depth image in three-dimensional reconstruction network, generates corresponding threedimensional model.
3. Deformation Prediction method as described in claim 1, which is characterized in that the Deformation Prediction network include generate network and
Differentiate network, it is described that the threedimensional model and the conditional vector generated based on target object characteristic are inputted into Deformation Prediction network,
Obtain Deformation Prediction network output prediction model of deformation the step of include:
First condition vector is generated based on target object characteristic, the threedimensional model and first condition vector input are generated
Network obtains the prediction of distortion amount of target object；
The true strain amount for obtaining target object, using the prediction of distortion amount of the generation network output as the input for differentiating network
Value determines whether the prediction of distortion amount for generating network output meets expectation in conjunction with the true strain amount；
When the prediction of distortion amount of the generation network output meets desired, determine that the prediction of distortion amount is Deformation Prediction mould
Type.
4. Deformation Prediction method as claimed in claim 3, which is characterized in that the differentiation network successively includes five deconvolution
Layer, wherein each warp lamination includes an activation primitive；
The generation network includes variation autocoder and decoder；
The variation autocoder successively includes five convolutional layers, a full articulamentum and two μ and σ layers, wherein every
A warp lamination all includes an activation primitive；
The decoder successively includes five warp laminations, wherein each warp lamination includes an activation primitive.
5. Deformation Prediction method as claimed in claim 3, which is characterized in that described to generate first based on target object characteristic
The step of part vector includes:
Determine the material property of target object and the stress condition of target object；
First condition vector is generated based on the material property and stress condition.
6. Deformation Prediction method as claimed in claim 3, which is characterized in that the true strain amount for obtaining target object,
Life is determined in conjunction with the true strain amount using the prediction of distortion amount of the generation network output as the input value for differentiating network
Include: at whether the prediction of distortion amount that network exports meets the step of desired
Obtain the true strain amount that physical modeler generates；
The prediction of distortion amount input of the generation network output is differentiated into network, in conjunction with the true strain amount and is based on object
The second condition vector that bulk properties generates, determines whether the prediction of distortion amount meets expectation.
7. Deformation Prediction method as described in claim 1, which is characterized in that the depth image for obtaining target object is simultaneously defeated
Enter three-dimensional reconstruction network, the step of to obtain threedimensional model that the three-dimensional reconstruction network is generated based on the depth image it
Before, the method also includes:
Several real-world object deformation patterns are collected, corresponding image data base is established；
Deformation Prediction network is trained based on described image database so that the predictive ability of the Deformation Prediction network by
It is cumulative strong.
8. Deformation Prediction method as claimed in claim 7, which is characterized in that described pre- to deforming based on described image database
Survey grid network is trained, so that the predictive ability of the Deformation Prediction network includes: the step of gradually increasing
Obtain several voxel grids and conditional vector based on image data base, by physical modeler generate by it is described several
Several ground truths pair of voxel grid and conditional vector composition；
With several described ground truths to training Deformation Prediction network, the relevant parameter of Deformation Prediction network is continued to optimize, with
Gradually increase the predictive ability of the Deformation Prediction network.
9. a kind of Deformation Prediction device, which is characterized in that the Deformation Prediction device includes: memory, processor and is stored in
On the memory and the Deformation Prediction program that can run on the processor, the Deformation Prediction program is by the processor
It realizes when execution such as the step of Deformation Prediction method described in any item of the claim 1 to 8.
10. a kind of computer readable storage medium, which is characterized in that it is pre- to be stored with deformation on the computer readable storage medium
Ranging sequence realizes such as Deformation Prediction described in any item of the claim 1 to 8 when the Deformation Prediction program is executed by processor
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910365833.0A CN110084845A (en) | 2019-04-30 | 2019-04-30 | Deformation Prediction method, apparatus and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910365833.0A CN110084845A (en) | 2019-04-30 | 2019-04-30 | Deformation Prediction method, apparatus and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110084845A true CN110084845A (en) | 2019-08-02 |
Family
ID=67418455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910365833.0A Pending CN110084845A (en) | 2019-04-30 | 2019-04-30 | Deformation Prediction method, apparatus and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110084845A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110517352A (en) * | 2019-08-27 | 2019-11-29 | 电子科技大学 | A kind of three-dimensional rebuilding method of object, storage medium, terminal and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017120897A1 (en) * | 2016-01-15 | 2017-07-20 | 武汉武大卓越科技有限责任公司 | Object surface deformation feature extraction method based on line scanning three-dimensional point cloud |
| CN107220600A (en) * | 2017-05-17 | 2017-09-29 | 清华大学深圳研究生院 | A kind of Picture Generation Method and generation confrontation network based on deep learning |
| CN108021131A (en) * | 2017-11-28 | 2018-05-11 | 王智华 | A kind of robot vision recognition methods, device and computer-readable recording medium |
| CN108510573A (en) * | 2018-04-03 | 2018-09-07 | 南京大学 | A method of the multiple views human face three-dimensional model based on deep learning is rebuild |
| CN108564119A (en) * | 2018-04-04 | 2018-09-21 | 华中科技大学 | A kind of any attitude pedestrian Picture Generation Method |
-
2019
- 2019-04-30 CN CN201910365833.0A patent/CN110084845A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017120897A1 (en) * | 2016-01-15 | 2017-07-20 | 武汉武大卓越科技有限责任公司 | Object surface deformation feature extraction method based on line scanning three-dimensional point cloud |
| CN107220600A (en) * | 2017-05-17 | 2017-09-29 | 清华大学深圳研究生院 | A kind of Picture Generation Method and generation confrontation network based on deep learning |
| CN108021131A (en) * | 2017-11-28 | 2018-05-11 | 王智华 | A kind of robot vision recognition methods, device and computer-readable recording medium |
| CN108510573A (en) * | 2018-04-03 | 2018-09-07 | 南京大学 | A method of the multiple views human face three-dimensional model based on deep learning is rebuild |
| CN108564119A (en) * | 2018-04-04 | 2018-09-21 | 华中科技大学 | A kind of any attitude pedestrian Picture Generation Method |
Non-Patent Citations (1)
| Title |
|---|
| ZHIHUA WANG 等: "3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations", 《ARXIV:1805.00328V2》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110517352A (en) * | 2019-08-27 | 2019-11-29 | 电子科技大学 | A kind of three-dimensional rebuilding method of object, storage medium, terminal and system |
| CN110517352B (en) * | 2019-08-27 | 2022-06-03 | 电子科技大学 | Three-dimensional reconstruction method, storage medium, terminal and system of object |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11941719B2 (en) | Learning robotic tasks using one or more neural networks | |
| CN108021131B (en) | Robot vision identification method and device and computer readable storage medium | |
| CN112614213B (en) | Facial expression determining method, expression parameter determining model, medium and equipment | |
| IL297846B1 (en) | Deep learning system for cuboid detection | |
| Shi et al. | RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks | |
| CN105051755A (en) | Part and state detection for gesture recognition | |
| CN104508709A (en) | Animating objects using the human body | |
| CN110503686A (en) | Object pose estimation method and electronic equipment based on deep learning | |
| CN112991503B (en) | Model training method, device, equipment and medium based on skin weight | |
| CN110135497A (en) | Method, the method and device of Facial action unit intensity estimation of model training | |
| US11138812B1 (en) | Image processing for updating a model of an environment | |
| CN112348285B (en) | Crowd evacuation simulation method in dynamic environment based on deep reinforcement learning | |
| CN108460152A (en) | A kind of the space-filling curve method, apparatus and computer readable storage medium of data | |
| CN115131849A (en) | Image generation method and related device | |
| CN114140841A (en) | Point cloud data processing method, neural network training method and related equipment | |
| CN108845833A (en) | Intelligent shutdown method, apparatus and computer readable storage medium | |
| CN110084845A (en) | Deformation Prediction method, apparatus and computer readable storage medium | |
| CN113066125A (en) | Augmented reality method and related equipment thereof | |
| CN116968024A (en) | Method, computing device and medium for obtaining control strategy for generating shape closure grabbing pose | |
| Kiatos et al. | Grasping unknown objects by exploiting complementarity with robot hand geometry | |
| CN110070542A (en) | Machine learning method, device and the computer readable storage medium of intuition physics | |
| CN113869189A (en) | Human behavior recognition method, system, device and medium | |
| CN112037315A (en) | Method and device for generating local descriptor and method and device for generating model | |
| CN115601513A (en) | Model hyper-parameter selection method and related device | |
| Kim et al. | Automatic labeled LiDAR data generation and distance-based ensemble learning for human segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190802 |
|
| RJ01 | Rejection of invention patent application after publication |