CN109213884A - A kind of cross-module state search method based on Sketch Searching threedimensional model - Google Patents
A kind of cross-module state search method based on Sketch Searching threedimensional model Download PDFInfo
- Publication number
- CN109213884A CN109213884A CN201811414596.4A CN201811414596A CN109213884A CN 109213884 A CN109213884 A CN 109213884A CN 201811414596 A CN201811414596 A CN 201811414596A CN 109213884 A CN109213884 A CN 109213884A
- Authority
- CN
- China
- Prior art keywords
- sketch
- view
- network
- threedimensional model
- anchor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
Abstract
The cross-module state search method based on Sketch Searching threedimensional model that the invention discloses a kind of, comprising steps of 1) data set is chosen；2) three-dimensional modeling data concentrated to data renders, and a threedimensional model obtains multiple two dimension views, these two dimension views are used to indicate threedimensional model, and sketch data uniform sizes are at 256 × 256；3) training sketch classifier and view classification device；4) construct depth measure studying space: using the sketch of initial data concentration and view as the input of network, the parameter that sketch and view classification device are obtained reaches searched targets as the parameter of network, by training.The present invention is using view as intermediary, the semantic gap of threedimensional model and sketch is reduced as far as possible, construct the network structure of a novel Sketch Searching threedimensional model, extraordinary classification accuracy is achieved on SHREC2013 and SHREC2014, experimental result sufficiently shows that our network frame completely can be by Sketch Searching to corresponding threedimensional model.
Description
Technical field
The present invention relates to the technical fields of cross-module state retrieval, refer in particular to a kind of cross-module based on Sketch Searching threedimensional model
State search method.
Background technique
In recent years, since threedimensional model has stronger visual stimulus and wider array of application scenarios, three-dimensional modeling data
In explosive growth.With the development of the technologies such as 3-D scanning, 3 D-printing, people can more efficiently obtain three-dimensional data.
In order to help user easily to obtain required threedimensional model, and threedimensional model can be multiplexed and be shared, threedimensional model
Retrieval technique has become the hot topic of computer graphics.
Widely available and expression packet the prevalence of touch-screen equipment and handwriting equipment, so that people begin trying with Freehandhand-drawing grass
Figure replaces text or the language to carry out abstract expression.By sketch, user can more fast and accurately express the idea of oneself,
The randomness and creativeness of input have also obtained greatly putting to good use.Cartographical sketching can be with as a kind of conveniently interactive mode
The preferably inquiry intention of expression user.Compared with based on keyword and based on sample mode, sketch is that one kind is more suitable for three-dimensional
The interactive mode of model index.Although the method based on Sketch Searching has the advantages that convenient and is easy to obtain, it is still deposited
In many stubborn problems.Firstly, sketch and threedimensional model belong to two different mode, and the feature of different modalities follows not
The same regularity of distribution is so there is very big semantic gap in the two.Secondly, sketch lines are fairly simple and the difference of same object
Expression is also not quite similar.
Summary of the invention
The purpose of the present invention is to overcome the shortcomings of the existing technology and deficiency, proposes a kind of based on Sketch Searching three-dimensional mould
The cross-module state search method of type is reduced the semantic gap of threedimensional model and sketch as far as possible, is utilized three-dimensional using view as intermediary
Multiple views of model characterize the information of threedimensional model, construct the network knot of a novel Sketch Searching threedimensional model
Structure achieves extraordinary classification accuracy on SHREC2013 and SHREC2014, and currently the best way is suitable, experiment
As a result sufficiently show that our network frame completely can be by Sketch Searching to corresponding threedimensional model.
To achieve the above object, technical solution provided by the present invention are as follows: it is a kind of based on Sketch Searching threedimensional model across
Mode search method, this method are the cross-domain classifiers of training using three metanetworks as basic structure, constantly reduce cross-domain mode it
Between difference, improve retrieval precision comprising following steps:
1) data set is chosen
SHREC2013, SHREC2014 data set of data set selection standard, each data set include three-dimensional modeling data
Subset and sketch data subset；
2) original data set pre-processes
The three-dimensional modeling data concentrated to data renders, and a threedimensional model obtains multiple two dimension views, these two
Dimensional view is for indicating threedimensional model, and sketch data uniform sizes are at 256 × 256；
3) training sketch classifier and view classification device
For sketch classifier, two dimension view is converted into sketch by edge extracting, is anchor with primal sketch, turned
The sketch got in return is divided into positive example and negative example, training in three metanetworks is sent to, using the parameter of network as sketch classifier
Output；
For view classification device, primal sketch is converted into view by GAN network, is anchor with this view, three
The view of dimension module is divided into positive example and negative example, is sent to ternary network training, using the parameter of network as the defeated of view classification device
Out；
4) depth measure studying space is constructed
Input of the sketch and view concentrated using initial data as network, the parameter that sketch and view classification device are obtained
As the parameter of network, searched targets are reached by training.
In step 1), selection data set is SHREC2013 and SHREC2014, and SHREC2013 data set includes 1258
Threedimensional model and 7200 sketches amount to 90 classes, and every class sketch includes 50 training samples and 30 test samples, amount to 80
Sample；SHREC2014 data set includes 8987 threedimensional models and 13680 sketches, and totally 171 class, every class sketch include 50
Training sample and 30 test samples amount to 80 samples.
In step 2), all threedimensional models progress regularizations are put in data set, by the position that three-dimensional space is set
Placement location virtual camera renders two dimension view, while in order to significantly more protrude the depth information on threedimensional model surface, wash with watercolours
It also added illumination when dye, the depth information on threedimensional model surface characterized with the light and shade information of the two dimension view of rendering；It will be careless
Figure size adjusting be with size needed for network be adapted size, its object is to data image regulation to reduce parameter
Amount, so that network more efficient operation.
In step 3), using three metanetworks as basic structure, Alexnet network is used for each branch, specifically such as
Under:
In sketch classifier, need to obtain the connection of sketch and view in sketch domain, therefore two dimension view is first
It is converted into sketch, is a selection well generally for cross-domain conversion GAN network, but threedimensional model in the data set used
The amount of views of rendering is significantly larger than sketch quantity, is found through experiments that view, which is switched to sketch effect, using GAN network pays no attention to
Think, therefore need to make alternatively to be handled: firstly, using the marginal information of canny operator extraction view, then into
Row expansion and gaussian filtering process obtain sketch, when constructing three metanetworks, using the sketch that initial data is concentrated as anchor,
It is selected in the sketch data set converted using edge extraction operation and anchor is generic different classes of as positive example and anchor
The negative example of conduct, shared parameter between different branches constructs the loss of three metanetworks, to allow cross-domain feature distribution to have correlation,
Loss function indicates are as follows:
In formula, LsketchIndicate the loss of sketch classifier, in order to make the distance of anchor and positive example increasingly
Closely, the distance of anchor and negative example is more and more remoter,It represents the sketch as anchor and is converted to
Positive example and negative example in view, fs() is the feature obtained by network training, and α indicates positive sample between negative sample pair
Constraint makes loss reach minimum by continuing to optimize, and sketch classifier just complete by training；
In view classification device, need to obtain the connection of sketch and view in view field, the texture information of sketch is than view
Figure is few, and view has texture information abundant, is difficult that sketch is converted into view from scratch using existing mathematical tool；And
A type of image can be converted into another type of image by GAN network, and cyclegan is used to turn view as sketch
Tool, due to threedimensional model rendering viewdata collection be far longer than primal sketch data set, construct three metanetworks
When, using the view that sketch is transformed into as anchor, is selected in viewdata concentration and anchor is generic as positive example, and
The negative example of anchor different classes of conduct, shared parameter between different branches, building ternary loss, to learn cross-domain feature distribution
Correlation, loss function indicate are as follows:
In formula, LviewIndicate the loss of view classification device,It respectively indicates and is converted to by GAN
The positive example and negative example in view that view and threedimensional model as anchor render, fv() expression is arrived by network training
Feature, by keeping the distance of anchor and positive example more and more closer to the continuous iteration optimization of network, the distance of anchor and negative example
It is increasingly remoter, correlation and distinction between parameter, that is, representative sample of network.
In step 4), multi-modal data constructs metric space, which must be similar approximate to the greatest extent in semantic feature, foreign peoples
Xiang Yuan, sketch and view have common level on semantic level；Entrance is sketch and view, but exporting all is feature, i.e.,
Semantically there is a logical phasic property, whether network in a certain layer can reach shared, therefore consider weight sharing problem, with initial data
The view that the sketch and threedimensional model of concentration render, using sketch as anchor, selects just as input in view-set
Example and negative example, the parameter for using sketch classifier and view classification device to obtain is as the parameter of network, the ginseng of the two classifiers
Number represents relationships in sketch domain and view field between different modalities as classifier network training process, positive example and negative
Example network branches share weight, continue to optimize to obtain required retrieval network, and the three-dimensional model search based on sketch is dependent on grass
The distance between figure feature and multiple view feature measurement, the feature of the sketch and view that extract for last retrieval network are retouched
Symbol is stated, the distance between they are measured by Wasserstein distance, the distance the close, indicates more similar, measuring similarity
It is as follows:
In formula,For the feature of m-th of model and n-th of view in threedimensional model,For the feature of i-th of sketch, D
(Mm,Si) smaller, show sketch SiWith threedimensional model view MmIt is more similar.
Compared with prior art, the present invention have the following advantages that with the utility model has the advantages that
1, the present invention in order to make up the semantic gap between sketch and threedimensional model, using view established as intermediary sketch with
Correlation between threedimensional model.
2, conventional method can all select the optimal viewing angle of threedimensional model, and the selection of optimal viewing angle often has subjectivity, this
Invention does not need the selection of optimal viewing angle, directly carries out cross-domain study using three metanetworks.
3, present invention uses GAN networks to carry out sketch to the cross-domain conversion of view, and GAN has non-in terms of image interpretation
Often good effect, both makes the data for respectively obtaining cross-module state in this way, and further expansion visual information reduces sketch and view
Semantic difference.
4, the present invention constructs two domain classifiers, to learn the semantic dependency between sketch and view, reduces sketch
With the semantic difference of view.
5, the present invention is lost using ternary for the first time, can effectively reduce the distance between similar sample, is increased different
The distance between sample improves retrieval precision.
6, the present invention has used Wasserstein distance for the first time to carry out measuring similarity, uses compared to tradition
Euclidean distance is measured, and Wasserstein distance more can accurately measure similarity.
7, the present invention has better accuracy in terms of Sketch Searching threedimensional model, easy to operate, practical, has
Good Utilization prospects.
Detailed description of the invention
Fig. 1 is multilevel iteration metric learning schematic diagram.
Fig. 2 is that two different views obtain comparison diagram.
Fig. 3 is triplet network structure.
Specific embodiment
The present invention is further explained in the light of specific embodiments.
Cross-module state search method based on Sketch Searching threedimensional model provided by the present embodiment, is using three metanetworks as base
This structure, the cross-domain classifier of training, constantly reduces the difference between cross-domain mode, improves retrieval precision, as shown in figure 3, exhibition
Our overall network structure is shown, part weight is shared between anchor anchor point and the negative example of positive example, grasps by convolution, pond etc.
Make extraction feature, completes retrieval tasks by three metanetworks.Itself the following steps are included:
1) data set is chosen
SHREC2013, SHREC2014 data set of data set selection standard, each data set include three-dimensional modeling data
Subset and sketch data subset；SHREC2013 data set includes 1258 threedimensional models and 7200 sketches, amounts to 90 classes, often
Class sketch includes 50 training samples and 30 test samples, amounts to 80 samples；SHREC2014 data set includes 8987 three
Dimension module and 13680 sketches, totally 171 class, every class sketch include 50 training samples and 30 test samples, amount to 80 samples
This.
2) original data set pre-processes
The three-dimensional modeling data concentrated to data renders, as shown in Fig. 2, in the first way, it will be assumed that mould
The consistent placement in type direction uniformly places 12 virtual cameras in the plane (such as perpendicular to horizontal line) around model,
12 different views can be obtained；In the second way, model can be placed arbitrarily, we are three-dimensional with a regular dodecahedron package
Model, and model mass center is aligned with the center of regular dodecahedron, then put in 20 fixed points of 12 face bodies towards model center
Set 20 virtual cameras, each video camera using the line at the center of oneself and the center of model as axis rotate 0 °, 90 °, 180 °,
270 ° obtain the view of four different rotary angles, and the two dimension view of 80 different angles is obtained in 20 virtual cameras one.
Compared with the second way, people are more prone to selection first way when observing object, so there is employed herein the first
View rendering mode.One threedimensional model obtains multiple two dimension views, these two dimension views are used to indicate threedimensional model, sketch number
According to uniform sizes at 256 × 256；All threedimensional models progress regularizations are put in data set, pass through what is set in three-dimensional space
Virtual camera is placed to render two dimension view in position, while in order to significantly more protrude the depth information on threedimensional model surface,
It also added illumination when rendering, the depth information on threedimensional model surface characterized with the light and shade information of the two dimension view of rendering；It will
Sketch size adjusting be with size needed for network be adapted size, its object is to data image regulation to reduce parameter
Amount, so that network more efficient operation.
Giving primal sketch size herein is 1111 × 1111, is 256 × 256 by its scaled, gives threedimensional model,
We have continued to use the view acquisition methods of MVCNN, render two by placing virtual camera in the position that three-dimensional space is set
Dimensional view, while in order to significantly more protrude the depth information on threedimensional model surface, when rendering, also added illumination, with rendering
The light and shade information of two dimension view characterizes the depth information on threedimensional model surface, these two dimension views reflect three under different perspectives
Geological information (such as curved surface, plane, corner angle), structural information and the depth information characterized by light and shade on dimension module surface, most
The latter threedimensional model renders to obtain 12 views expressions.
3) training sketch classifier and view classification device
Using three metanetworks as basic structure, Alexnet network is used for each branch, experiment shows that Alexnet exists
Extracting two dimensional image characteristic aspect has good effect.Concrete condition is as follows:
In sketch classifier, it would be desirable to obtain the connection of sketch and view in sketch domain, therefore we are by two
Dimensional view is first converted into sketch, is a selection well generally for cross-domain conversion GAN network, but our data for using
It concentrates the amount of views of threedimensional model rendering to be significantly larger than sketch quantity, is found through experiments that and is switched to view using GAN network
Sketch effect is undesirable, so it is contemplated that making alternatively to be handled: firstly, being regarded using canny operator extraction
Then the marginal information of figure carries out the processing such as expansion and gaussian filtering and obtains sketch, when constructing three metanetworks, with initial data
The sketch of concentration is as anchor, with selection in the sketch data set of the operations such as edge extracting conversion and anchor generic work
For positive example and the different classes of negative example of conduct of anchor, shared parameter between different branches constructs the loss of three metanetworks, comes
Cross-domain feature distribution is allowed to have correlation.
For sketchThe sketch translated into viewThe sketch character representation extracted in a network is fs(S)
And fs(YV).In sketch metric learning, Wo MencongIn
Random selectionAs anchor (Indicate k-th of anchor sample in sketch metric learning), fromIt is middle selection withSimilarAs positive (Indicate sketch metric learning in k-th of positive sample), selection withIt is inhomogeneous to be used as negative(
Indicate k-th of negative sample in sketch metric learning).Loss function indicates are as follows:
In formula, LsketchIndicate the loss of sketch classifier, in order to make the distance of anchor and positive example increasingly
Closely, the distance of anchor and negative example is more and more remoter,It represents the sketch as anchor and is converted to
Positive example and negative example in view, fs() is the feature obtained by network training, and α indicates positive sample between negative sample pair
Constraint makes loss reach minimum by continuing to optimize, and sketch classifier just complete by training.
In view classification device, it would be desirable to obtain the connection of sketch and view in view field, sketch has fewer
Texture information, view have texture information very abundant, be difficult from scratch to convert sketch using existing mathematical tool
At view；A type of image can be converted into another type of image by GAN network；We use cyclegan as
Sketch turns the tool of view, since the viewdata collection of threedimensional model rendering is far longer than primal sketch data set, we
When constructing three metanetworks, using the view that sketch is transformed into as anchor, concentrate selection similar with anchor in viewdata
It is other as positive example and the different classes of negative example of conduct of anchor, shared parameter between different branches, building ternary loss comes
Learn the correlation of cross-domain feature distribution.
For sketch, we are defined asNsketchFor training set medium-height grass
The number of figure.For the view that threedimensional model renders, we are defined as
NviewFor the number (N of view in training setview=NM×NV), NMFor the number of threedimensional model in database, NVFor each three-dimensional
The view number that model rendering obtains.For two fields view V of sketch S and threedimensional model, { S } { V } is used to indicate two
The image collection in field.The purpose of CycleGAN translation is exactly the mapping learnt from sketch S to view V.We assume that sketch arrives
View is mapped as X:S → V；View is mapped as Y:V → S to sketch.Sketch S generates view X (S) by generator G, X (S)
By obtaining image D after arbiterV(X(s)).Similarly view generates sketch Y (V) by generator G, and Y (V) passes through arbiter
Image Ds (Y (v)) is obtained later.
The arbiter of X:S → V loses are as follows:
The arbiter of Y:V → S loses are as follows:
The loss that X:S → V and Y:V → S are generated twice are as follows:
The final loss function of cross-module state translation are as follows:
L(X,Y,DS,DV)=LGAN-X(X,DV,S,V)+LGAN-Y(Y,DS,V,S)+Lcyc(X,Y)
In cross-module state translation process, by sketch and the continuous study of view, mutually translation, sketch translation finally can be obtained
At viewThe sketch that view is translated into is obtained simultaneouslyFor viewThe view translated into sketchThe view feature extracted in a network is expressed as fv
(XS) and fv(V).In view metric learning, Wo MencongIn
Random selectionAs anchor (Indicate k-th of anchor sample in view metric learning), fromIt is middle selection withSimilarAs positive (Indicate view
K-th of positive sample in metric learning), selection withIt is inhomogeneousAs negative (Indicate view measurement
K-th of negative sample in study).Loss function indicates are as follows:
In formula, LviewIndicate the loss of view classification device,It respectively indicates and is converted to by GAN
The positive example and negative example in view that view and threedimensional model as anchor render, fv() expression is arrived by network training
Feature, by keeping the distance of anchor and positive example more and more closer to the continuous iteration optimization of network, the distance of anchor and negative example
It is increasingly remoter, correlation and distinction between parameter, that is, representative sample of network.
4) depth measure studying space is constructed
Input of the sketch and view concentrated using initial data as network, the parameter that sketch and view classification device are obtained
As the parameter of network, searched targets are reached by training, as shown in Figure 1, illustrating our retrieval network frame, three-dimensional mould
Type passes through MVCNN first and obtains multiple two dimension views, randomly select a part therein as positive example and negative example (positive example be with
The similar view of anchor anchor point sketch, negative example are the views with anchor anchor point sketch foreign peoples), it is mentioned by convolutional neural networks
The feature for taking view and sketch completes retrieval tasks finally by similarity measurement.It is specific as follows:
Multi-modal data constructs metric space, which must be similar approximate to the greatest extent in semantic feature, and foreign peoples is mutually remote, in semanteme
Sketch and view have common level in level.Entrance is sketch and view, but exporting all is that feature (is semantically having logical phase
Property), whether network in a certain layer can reach shared, therefore consider weight sharing problem, the sketch concentrated with initial data and
The view that threedimensional model renders is as input, likewise, we select in view-set still using sketch as anchor
Positive example and negative example, the parameter for using sketch classifier and view classification device to obtain is as the parameter of the network, the two classifiers
Parameter represent the relationship in sketch domain and view field between different modalities as classifier network training process, positive example
Weight is shared with negative example network branches, continues to optimize the retrieval network for obtaining us, the three-dimensional model search based on sketch relies on
It is measured in the distance between sketch feature and multiple view feature, the spy of the sketch and view that are extracted for last retrieval network
Descriptor is levied, we measure the distance between they by Wasserstein distance, and the distance the close, indicate more similar.
Herein using the similitude between euclidean distance metric cartographical sketching and threedimensional model render view.
(NMFor the number of threedimensional model in database, NVIt is rendered for each threedimensional model
The view number arrived) be threedimensional model view set,For sketch
Set,Indicate sketch SiWith threedimensional model viewBetween Euclidean distance.Measuring similarity is as follows:
In formula,For the feature of m-th of model and n-th of view in threedimensional model,For the feature of i-th of sketch, D
(Mm,Si) smaller, show sketch SiWith threedimensional model view MmIt is more similar.
Experimental configuration: the hardware environment tested herein is 1070 8G+8G RAM of Intel Core i7 2600k+GTX,
Software environment is that windows 7x64+CUDA 8.0+cuDNN 5.1+Caffe+Python. uses AlexNet extraction to scheme herein
As feature, data are handled using python, retrieval metrics evaluation and partial visual effect is completed using MATLAB, uses
Python stores characteristics of image and completes retrieval experiment.
Data set: 2013,2014 standard data sets that data set used is SHREC Sketch Searching special topic, letter are tested herein
Claim SHREC2013, SHREC2014.SHREC annual data can have a greater change, not only the quantity and sketch of threedimensional model
Quantity increased, and the complicated classification degree of threedimensional model also increases accordingly.In addition, the type of threedimensional model is more, the number of every class
It is especially few to measure inconsistent and some categorical measures.This paper algorithm overcomes influence of the different data collection to search result, in difference
Good performance is demonstrated by under data set test.SHREC2013 data set includes 1258 threedimensional models and 7200 sketches,
Totally 90 class.Every class sketch includes 50 training samples and 30 test samples, totally 80 samples.Number of the threedimensional model in every class
Amount is different from, average 14 models of every class.SHREC2014 data set includes 8987 threedimensional models and 13680 sketches, altogether
171 classes.Every class sketch includes 50 training samples and 30 test samples, totally 80 samples.Table 1 gives selects data herein
The essential information of collection.
Table 1 selects the essential information of data set herein
| Data set | Model number | The number of class | All kinds of sketch numbers |
| SHREC2013 | 1258 | 90 | 80 |
| SHREC2013 | 8987 | 171 | 80 |
Embodiment described above is only the preferred embodiments of the invention, and but not intended to limit the scope of the present invention, therefore
All shapes according to the present invention change made by principle, should all be included within the scope of protection of the present invention.
Claims (5)
1. a kind of cross-module state search method based on Sketch Searching threedimensional model, which is characterized in that this method is with three metanetworks
For basic structure, the cross-domain classifier of training constantly reduces the difference between cross-domain mode, improves retrieval precision comprising with
Lower step:
1) data set is chosen
SHREC2013, SHREC2014 data set of data set selection standard, each data set include three-dimensional modeling data subset
With sketch data subset；
2) original data set pre-processes
The three-dimensional modeling data concentrated to data renders, and a threedimensional model obtains multiple two dimension views, these two dimension views
Figure is for indicating threedimensional model, and sketch data uniform sizes are at 256 × 256；
3) training sketch classifier and view classification device
For sketch classifier, two dimension view is converted into sketch by edge extracting, is anchor with primal sketch, converts
To sketch be divided into positive example and negative example, training is sent in three metanetworks, using the parameter of network as the output of sketch classifier；
For view classification device, primal sketch is converted into view by GAN network, is anchor with this view, three-dimensional mould
The view of type is divided into positive example and negative example, is sent to ternary network training, using the parameter of network as the output of view classification device；
4) depth measure studying space is constructed
Using initial data concentrate sketch and view as network input, the parameter that sketch and view classification device are obtained as
The parameter of network reaches searched targets by training.
2. a kind of cross-module state search method based on Sketch Searching threedimensional model according to claim 1, it is characterised in that:
In step 1), selection data set is SHREC2013 and SHREC2014, and SHREC2013 data set includes 1258 threedimensional models
With 7200 sketches, amount to 90 classes, every class sketch includes 50 training samples and 30 test samples, amounts to 80 samples；
SHREC2014 data set includes 8987 threedimensional models and 13680 sketches, and totally 171 class, every class sketch include 50 trained samples
Originally with 30 test samples, amount to 80 samples.
3. a kind of cross-module state search method based on Sketch Searching threedimensional model according to claim 1, it is characterised in that:
In step 2), all threedimensional models progress regularizations are put in data set, empty by placing in the position that three-dimensional space is set
Quasi- video camera renders two dimension view, while in order to significantly more protrude the depth information on threedimensional model surface, and when rendering also adds
Illumination is added, the depth information on threedimensional model surface is characterized with the light and shade information of the two dimension view of rendering；By sketch size tune
The whole size to be adapted with size needed for network, its object is to reduce parameter amount to data image regulation, so that net
Network more efficient operation.
4. a kind of cross-module state search method based on Sketch Searching threedimensional model according to claim 1, it is characterised in that:
In step 3), using three metanetworks as basic structure, Alexnet network is used for each branch, specific as follows:
In sketch classifier, need to obtain the connection of sketch and view in sketch domain, therefore two dimension view is first converted
It is a selection well generally for cross-domain conversion GAN network, but threedimensional model renders in the data set used at sketch
Amount of views be significantly larger than sketch quantity, be found through experiments that that view is switched to sketch effect using GAN network is undesirable, therefore
It needs to make alternatively to be handled: firstly, then carrying out swollen using the marginal information of canny operator extraction view
Swollen and gaussian filtering process obtains sketch, and when constructing three metanetworks, the sketch concentrated using initial data is as anchor, with side
It is selected and the generic work different classes of as positive example and anchor of anchor in the sketch data set of edge extraction operation conversion
Be negative example, shared parameter between different branches, constructs the loss of three metanetworks, to allow cross-domain feature distribution to have correlation, loss
Function representation are as follows:
In formula, LsketchIndicate the loss of sketch classifier, in order to keep the distance of anchor and positive example more and more closer,
The distance of anchor and negative example is more and more remoter,The view for representing the sketch as anchor and being converted to
In positive example and negative example, fs() is the feature obtained by network training, and α indicates positive sample to the pact between negative sample pair
Beam makes loss reach minimum by continuing to optimize, and sketch classifier just complete by training；
In view classification device, needing to obtain the connection of sketch and view in view field, the texture information of sketch is fewer than view,
View has texture information abundant, is difficult that sketch is converted into view from scratch using existing mathematical tool；And GAN net
A type of image can be converted into another type of image by network, and cyclegan is used to turn the work of view as sketch
Tool, since the viewdata collection of threedimensional model rendering is far longer than primal sketch data set, when constructing three metanetworks, with
The view that sketch is transformed into selects in viewdata concentration as anchor and anchor is generic as positive example, and
The negative example of anchor different classes of conduct, shared parameter between different branches, building ternary loss, to learn cross-domain feature distribution
Correlation, loss function indicate are as follows:
In formula, LviewIndicate the loss of view classification device,It respectively indicates and conduct is converted to by GAN
The positive example and negative example in view that the view and threedimensional model of anchor renders, fv() indicates the spy arrived by network training
Sign, by keeping the distance of anchor and positive example more and more closer to the continuous iteration optimization of network, the distance of anchor and negative example is more next
Remoter, between parameter, that is, representative sample of network correlation and distinction.
5. a kind of cross-module state search method based on Sketch Searching threedimensional model according to claim 1, it is characterised in that:
In step 4), multi-modal data constructs metric space, which must be similar approximate to the greatest extent in semantic feature, and foreign peoples is mutually remote,
Sketch and view have common level on semantic level；Entrance is sketch and view, but exporting all is feature, i.e., semantically
There is a logical phasic property, whether network in a certain layer can reach shared, therefore consider weight sharing problem, the grass concentrated with initial data
The view that figure and threedimensional model render is as input, and using sketch as anchor, positive example and negative example are selected in view-set,
The parameter for using sketch classifier and view classification device to obtain represents grass as the parameter of network, the parameter of the two classifiers
Relationship in figure domain and view field between different modalities is as classifier network training process, positive example and negative example network branches
Shared weight, continues to optimize to obtain required retrieval network, and the three-dimensional model search based on sketch is dependent on sketch feature and more
The distance between view feature measurement, the feature descriptor of the sketch and view that extract for last retrieval network pass through
Wasserstein distance measures the distance between they, and the distance the close, indicates more similar, measuring similarity is as follows:
In formula,For the feature of m-th of model and n-th of view in threedimensional model,For the feature of i-th of sketch, D (Mm,
Si) smaller, show sketch SiWith threedimensional model view MmIt is more similar.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811414596.4A CN109213884B (en) | 2018-11-26 | 2018-11-26 | Cross-modal retrieval method based on sketch retrieval three-dimensional model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811414596.4A CN109213884B (en) | 2018-11-26 | 2018-11-26 | Cross-modal retrieval method based on sketch retrieval three-dimensional model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109213884A true CN109213884A (en) | 2019-01-15 |
| CN109213884B CN109213884B (en) | 2021-10-19 |
Family
ID=64994295
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811414596.4A Active CN109213884B (en) | 2018-11-26 | 2018-11-26 | Cross-modal retrieval method based on sketch retrieval three-dimensional model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109213884B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109948197A (en) * | 2019-02-28 | 2019-06-28 | 北方民族大学 | System optimization analysis method based on AADL V1 |
| CN110059205A (en) * | 2019-03-20 | 2019-07-26 | 杭州电子科技大学 | A kind of threedimensional model classification retrieving method based on multiple view |
| CN110162655A (en) * | 2019-04-30 | 2019-08-23 | 厦门大学 | A kind of threedimensional model partial retrieval, device, terminal device based on sketch |
| CN110188228A (en) * | 2019-05-28 | 2019-08-30 | 北方民族大学 | Cross-module state search method based on Sketch Searching threedimensional model |
| CN110457515A (en) * | 2019-07-19 | 2019-11-15 | 天津理工大学 | The method for searching three-dimension model of the multi-angle of view neural network of polymerization is captured based on global characteristics |
| CN110580302A (en) * | 2019-08-13 | 2019-12-17 | 天津大学 | Sketch image retrieval method based on semi-heterogeneous joint embedded network |
| CN111368123A (en) * | 2020-02-17 | 2020-07-03 | 同济大学 | Three-dimensional model sketch retrieval method based on cross-mode guide network |
| CN111382301A (en) * | 2020-03-19 | 2020-07-07 | 辽宁石油化工大学 | Three-dimensional model generation method and system based on generation countermeasure network |
| CN112085072A (en) * | 2020-08-24 | 2020-12-15 | 北方民族大学 | Cross-modal retrieval method of sketch retrieval three-dimensional model based on space-time characteristic information |
| CN112989063A (en) * | 2021-03-10 | 2021-06-18 | 青岛弯弓信息技术有限公司 | 3D modeling method and system based on knowledge graph |
| CN113392244A (en) * | 2021-06-10 | 2021-09-14 | 北京印刷学院 | Three-dimensional model retrieval method and system based on depth measurement learning |
| CN113554115A (en) * | 2021-08-12 | 2021-10-26 | 同济大学 | Three-dimensional model sketch retrieval method based on uncertain learning |
| CN113779287A (en) * | 2021-09-02 | 2021-12-10 | 天津大学 | Cross-domain multi-view target retrieval method and device based on multi-stage classifier network |
| CN114372538A (en) * | 2022-03-22 | 2022-04-19 | 中国海洋大学 | Method for convolution classification of scale vortex time series in towed sensor array |
| CN116108375A (en) * | 2022-12-19 | 2023-05-12 | 南京理工大学 | Graph classification method based on structure sensitive graph dictionary embedding |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101004748A (en) * | 2006-10-27 | 2007-07-25 | 北京航空航天大学 | Method for searching 3D model based on 2D sketch |
| CN104751463A (en) * | 2015-03-31 | 2015-07-01 | 梁爽 | Three-dimensional model optimal visual angle selection method based on sketch outline features |
| CN105243137A (en) * | 2015-09-30 | 2016-01-13 | 华南理工大学 | Draft-based three-dimensional model retrieval viewpoint selection method |
| CN107122396A (en) * | 2017-03-13 | 2017-09-01 | 西北大学 | Three-dimensional model searching algorithm based on depth convolutional neural networks |
| CN108595636A (en) * | 2018-04-25 | 2018-09-28 | 复旦大学 | The image search method of cartographical sketching based on depth cross-module state correlation study |
-
2018
- 2018-11-26 CN CN201811414596.4A patent/CN109213884B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101004748A (en) * | 2006-10-27 | 2007-07-25 | 北京航空航天大学 | Method for searching 3D model based on 2D sketch |
| CN104751463A (en) * | 2015-03-31 | 2015-07-01 | 梁爽 | Three-dimensional model optimal visual angle selection method based on sketch outline features |
| CN105243137A (en) * | 2015-09-30 | 2016-01-13 | 华南理工大学 | Draft-based three-dimensional model retrieval viewpoint selection method |
| CN107122396A (en) * | 2017-03-13 | 2017-09-01 | 西北大学 | Three-dimensional model searching algorithm based on depth convolutional neural networks |
| CN108595636A (en) * | 2018-04-25 | 2018-09-28 | 复旦大学 | The image search method of cartographical sketching based on depth cross-module state correlation study |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109948197A (en) * | 2019-02-28 | 2019-06-28 | 北方民族大学 | System optimization analysis method based on AADL V1 |
| CN110059205A (en) * | 2019-03-20 | 2019-07-26 | 杭州电子科技大学 | A kind of threedimensional model classification retrieving method based on multiple view |
| CN110162655A (en) * | 2019-04-30 | 2019-08-23 | 厦门大学 | A kind of threedimensional model partial retrieval, device, terminal device based on sketch |
| CN110162655B (en) * | 2019-04-30 | 2021-07-27 | 厦门大学 | Sketch-based three-dimensional model local retrieval method and device and terminal equipment |
| CN110188228B (en) * | 2019-05-28 | 2021-07-02 | 北方民族大学 | Cross-modal retrieval method based on sketch retrieval three-dimensional model |
| CN110188228A (en) * | 2019-05-28 | 2019-08-30 | 北方民族大学 | Cross-module state search method based on Sketch Searching threedimensional model |
| CN110457515A (en) * | 2019-07-19 | 2019-11-15 | 天津理工大学 | The method for searching three-dimension model of the multi-angle of view neural network of polymerization is captured based on global characteristics |
| CN110457515B (en) * | 2019-07-19 | 2021-08-24 | 天津理工大学 | Three-dimensional model retrieval method of multi-view neural network based on global feature capture aggregation |
| CN110580302A (en) * | 2019-08-13 | 2019-12-17 | 天津大学 | Sketch image retrieval method based on semi-heterogeneous joint embedded network |
| CN110580302B (en) * | 2019-08-13 | 2022-12-02 | 天津大学 | Sketch image retrieval method based on semi-heterogeneous joint embedded network |
| CN111368123A (en) * | 2020-02-17 | 2020-07-03 | 同济大学 | Three-dimensional model sketch retrieval method based on cross-mode guide network |
| CN111368123B (en) * | 2020-02-17 | 2022-06-28 | 同济大学 | Three-dimensional model sketch retrieval method based on cross-modal guide network |
| CN111382301B (en) * | 2020-03-19 | 2024-02-02 | 辽宁石油化工大学 | Three-dimensional model generation method and system based on generation countermeasure network |
| CN111382301A (en) * | 2020-03-19 | 2020-07-07 | 辽宁石油化工大学 | Three-dimensional model generation method and system based on generation countermeasure network |
| CN112085072A (en) * | 2020-08-24 | 2020-12-15 | 北方民族大学 | Cross-modal retrieval method of sketch retrieval three-dimensional model based on space-time characteristic information |
| CN112989063B (en) * | 2021-03-10 | 2023-02-10 | 青岛弯弓信息技术有限公司 | 3D modeling method and system based on knowledge graph |
| CN112989063A (en) * | 2021-03-10 | 2021-06-18 | 青岛弯弓信息技术有限公司 | 3D modeling method and system based on knowledge graph |
| CN113392244A (en) * | 2021-06-10 | 2021-09-14 | 北京印刷学院 | Three-dimensional model retrieval method and system based on depth measurement learning |
| CN113554115B (en) * | 2021-08-12 | 2022-09-13 | 同济大学 | Three-dimensional model sketch retrieval method based on uncertain learning |
| CN113554115A (en) * | 2021-08-12 | 2021-10-26 | 同济大学 | Three-dimensional model sketch retrieval method based on uncertain learning |
| CN113779287A (en) * | 2021-09-02 | 2021-12-10 | 天津大学 | Cross-domain multi-view target retrieval method and device based on multi-stage classifier network |
| CN113779287B (en) * | 2021-09-02 | 2023-09-15 | 天津大学 | Cross-domain multi-view target retrieval method and device based on multi-stage classifier network |
| CN114372538A (en) * | 2022-03-22 | 2022-04-19 | 中国海洋大学 | Method for convolution classification of scale vortex time series in towed sensor array |
| CN116108375A (en) * | 2022-12-19 | 2023-05-12 | 南京理工大学 | Graph classification method based on structure sensitive graph dictionary embedding |
| CN116108375B (en) * | 2022-12-19 | 2023-08-01 | 南京理工大学 | Graph classification method based on structure sensitive graph dictionary embedding |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109213884B (en) | 2021-10-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109213884A (en) | A kind of cross-module state search method based on Sketch Searching threedimensional model | |
| CN107122396B (en) | Method for searching three-dimension model based on depth convolutional neural networks | |
| Hua et al. | Pointwise convolutional neural networks | |
| Miller et al. | Quantum loewner evolution | |
| CN106469299B (en) | A kind of vehicle search method and device | |
| CN106383912B (en) | A kind of picture retrieval method and device | |
| CN101271469B (en) | Two-dimension image recognition based on three-dimensional model warehouse and object reconstruction method | |
| CN107220277A (en) | Image retrieval algorithm based on cartographical sketching | |
| CN108596329A (en) | Threedimensional model sorting technique based on end-to-end Deep integrating learning network | |
| CN108009286A (en) | A kind of Sketch Searching method based on deep learning | |
| CN107123027A (en) | A kind of cosmetics based on deep learning recommend method and system | |
| CN111967533B (en) | Sketch image translation method based on scene recognition | |
| CN104751463B (en) | A kind of threedimensional model optimal viewing angle choosing method based on sketch outline feature | |
| CN105469117B (en) | A kind of image-recognizing method and device extracted based on robust features | |
| CN108764138A (en) | A kind of highlands cloud snow sorting technique cascading forest based on the more granularities of multidimensional | |
| Demir et al. | Detecting visual design principles in art and architecture through deep convolutional neural networks | |
| CN108009557A (en) | A kind of threedimensional model method for describing local characteristic based on shared weight convolutional network | |
| CN102567465B (en) | Method for searching geometrical similarity of three-dimensional models on basis of compound eye vision | |
| CN104750762A (en) | Information retrieval method and device | |
| Deng et al. | Exploring the representativity of art paintings | |
| CN113989291A (en) | Building roof plane segmentation method based on PointNet and RANSAC algorithm | |
| CN107301425A (en) | A kind of allowing child daubing methods of marking based on deep learning | |
| Esposito et al. | Kiloneus: Implicit neural representations with real-time global illumination | |
| CN101894267B (en) | Three-dimensional object characteristic view selection method | |
| CN105447901B (en) | Image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |