CN107330852A - A kind of image processing method based on real-time zero point image manipulation network - Google Patents
A kind of image processing method based on real-time zero point image manipulation network Download PDFInfo
- Publication number
- CN107330852A CN107330852A CN201710531090.0A CN201710531090A CN107330852A CN 107330852 A CN107330852 A CN 107330852A CN 201710531090 A CN201710531090 A CN 201710531090A CN 107330852 A CN107330852 A CN 107330852A
- Authority
- CN
- China
- Prior art keywords
- mrow
- msub
- pnet
- network
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 27
- 238000012360 testing method Methods 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 6
- 230000008569 process Effects 0.000 claims abstract description 4
- 238000010606 normalization Methods 0.000 claims description 14
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 5
- 230000002708 enhancing effect Effects 0.000 claims description 4
- 238000003780 insertion Methods 0.000 claims description 4
- 230000037431 insertion Effects 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 3
- 210000002569 neuron Anatomy 0.000 claims description 3
- 239000004576 sand Substances 0.000 claims description 3
- 230000007704 transition Effects 0.000 claims description 3
- 230000005945 translocation Effects 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 230000035772 mutation Effects 0.000 abstract description 3
- 230000009466 transformation Effects 0.000 abstract description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003475 lamination Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/18—Image warping, e.g. rearranging pixels individually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
A kind of image processing method based on real-time zero point image manipulation network proposed in the present invention, its main contents include：Image procossing and convolutional neural networks (CNN), zero point operation network, training and test, its process is, zero point operation network is made up of parameter network (PNet) and image switching network (TNet), PNet is the universal model for producing key transformation parameter hierarchical structure, and these parameters generated are combined by TNet with the signal Mutation parameter of its own, content images are converted into stylized image.The guiding signal that zero point operation network can be in different forms in the present invention carries out high-quality image procossing in real time, with receiving any ability for instructing signal in many ways；Meanwhile, it supports the various mobile devices of connection, and so that user can obtain required output immediately, the stand-by period greatly shortens；And the stylistic category for being available for user to select is more, can more meet the demand of user.
Description
Technical field
Image processing field of the present invention, more particularly, to a kind of image procossing based on real-time zero point image manipulation network
Method.
Background technology
Graphic Arts stylization is the main research of non-photorealistic rendering in computer graphics in recent years, it with
Computer is instrument, and the form of expression of visual information in the drafting style of different art forms, enhancing image is gone out with algorithm simulation,
This technology for effectively being combined computer technology and aesthetics is increasingly liked by user.Non-photorealistic rendering can
There is provided for artist and create help, entertained for science and medical illustration field, engineering field and film, game, animation etc.
Industry etc., can be also used for the picture editting of mobile phone etc., and any common photo of user is converted into van gogh's body, Mo Nai by moment
The fine piece of writing of a variety of different artistic styles such as wind, impressionist.It can also predict the correlation of environment in advance by simulating varying environment
Situation, this provides convenience to operations on the sea such as marine traffic control, fishings；Meanwhile, by realizing image from daytime to night
Conversion, conversion from fine day to rainy day etc., people is predicted environment in advance, conveniently go on a journey or implement other work.However, existing
Some methods can not handle multiple signals simultaneously or can not handle the signal from other mode, and efficiency is very low.
The present invention proposes a kind of image processing method based on real-time zero point image manipulation network, zero point operate network by
Parameter network (PNet) and image switching network (TNet) composition, PNet is the Universal Die for producing key transformation parameter hierarchical structure
Type, and these parameters generated are combined by TNet with the signal Mutation parameter of its own, and content images are converted into stylization
Image.The guiding signal that zero point operation network can be in different forms in the present invention carries out high-quality image procossing in real time, has
Have and receive any ability for instructing signal in many ways；Meanwhile, it supports the various mobile devices of connection, so that user can stand
Required output is obtained, the stand-by period greatly shortens；And the stylistic category for being available for user to select is more, can more meet user
Demand.
The content of the invention
It is the problem of for that can not handle multiple signals simultaneously or the signal from other mode can not be handled, of the invention
Purpose is a kind of image processing method based on real-time zero point image manipulation network of offer, and zero point operates network by parameter network
(PNet) constituted with image switching network (TNet), PNet is the universal model for producing key transformation parameter hierarchical structure, and
These parameters generated are combined by TNet with the signal Mutation parameter of its own, and content images are converted into stylized image.
To solve the above problems, the present invention provides a kind of image processing method based on real-time zero point image manipulation network,
Its main contents includes：
(1) image procossing and convolutional neural networks (CNN)；
(2) zero point operation network；
(3) train and test.
Wherein, described image procossing and convolutional neural networks (CNN), give content imagesAnd guiding
Signal (for example, pattern diagram picture)Output transform imageSo that the Y and X in contentcIt is similar,
While and XsIt is similar；Therefore, effective expression no less important of study content and pattern, could so be carried out at rational image
Reason.
Further, described convolutional neural networks (CNN), using constant depth CNN φ (), the feature in l layers is reflected
PenetrateImage X content, and φ can be representedl(X) form matrix is expressed as
It is calculated as：
Above formula can represent the desired style images of image X；Only when each correspondence represents (that is, φlOr G (φ (X)l
(X) difference between) is this black norm of not Luo Beini one small, could assess including two images similar in perhaps style；
It therefore, it can training feedforward image converting network Y=T (Xc), typically depth CNN, with loss function：
Wherein,It is the style loss for generating image Y,It is contents lost, λsAnd λcIt is hyper parameter；S is " sample
The set of formula layer ", C is the set of " content layer ", ZlIt is the sum of neuron in l layers；Train after converting network T (), give
Fixed new content images X 'c, stylization image Y=T (X ' are generated in the case of without using network is lostc)。
Wherein, described zero point operation network, zero point operates network integration image switching network (TNet) and parameter net
Network (PNet), except carrying out transition diagram picture using TNet, goes back training parameter network (PNet), signal is condition to produce to guide
The TNet of (such as style image) key parameter；Because guiding signal is embedded into the communal space by PNet study, so zero
Point operation network can carry out zero point image procossing to unknown guiding signal.
Further, described TNet is normalized with dynamic instance, in order to realize zero point image procossing, in test process
TNet network parameter is dynamically specified, so as to handle invisible signal；Based on guiding signal XsOn PNet regulation
Characteristic pattern, initialization mode is the wave filter for directly generating TNet；However, in practice, TNet each layer generally has
More than 100000 parameters (for example, 128 × 128 × 3 × 3), and the characteristic pattern in PNet each layer generally has about 1000000
Individual entry (for example, 128 × 80 × 80)；Therefore, it is difficult to effectively by higher-dimension vector median filters into another.
Further, described dynamic instance normalization, using dynamic enhancing example normalization (each volume in TNet
Performed after lamination), pass through the PNet scalings produced and shift parameters γ (Xs) and β (Xs)；Here, scaling and translocation factor γ
(Xs) and β (Xs) it is considered as the key parameter in TNet each layer；In form, before example normalization, orderFor tensor, xijkI-th jk element is represented, wherein, i indicative character figures, j, k crosses over Spatial Dimension；Therefore,
The output of dynamic instance normalization (DIN)For：
Wherein, μiIt is the average value in characteristic pattern i,It is corresponding variance；γi(Xs) it is by being similar to that PNet is generated
βi(Xs) CiDimension vector γ (Xs) i-th of element；If γi(Xs)=1, βi(Xs)=0, then DIN be changed into example normalizing
Change；If γi(Xs)=γi,βi(Xs)=βi, then they are as the direct learning parameter unrelated with PNet；DIN will be changed into condition
Example is normalized；In both cases, the model all loses the ability of zero point study, therefore can not be generalized to and see not
The signal seen.
Further, described parameter network (PNet), in order to have with dynamic instance normalization driving TNet, PNet
There is serial or parallel framework.
Further, described serial PNet, in serial PNet, can use the depth with the structure similar to TNet
CNN is spent, γ is generated in l layers(l)(Xs) and β(l)(Xs)；In serial PNet, the γ of formula (3) on adjustment characteristic pattern(l)(Xs)
And β(l)(Xs), it is expressed as l layers in PNet of ψl(Xs)；
Wherein, if input XsImage, then ψl(Xs) be convolutional layer in TNet output；If inputting XsIt is word insertion
(vector), then ψl(Xs) be the layer being fully connected output；It is the parameter to be learnt.
Further, described parallel PNet, uses individually shallow network (the be fully connected or convolution) generation in l layer
ψl(Xs), then used it for calculating γ according to formula (4) and (5)(l)(Xs) and β(l)(Xs)；With the ψ from higher levell(Xs) raw
Into higher level γ(l)(Xs) and β(l)(Xs) serial PNet it is different, be from X heresTo γ(l)(Xs) and β(l)(Xs) follow it is shallow and
Parallel structure, but this structure will limit PNet validity, and the TNet of generation quality is somewhat reduced, and therefore produce
Raw image Y；Therefore, mostly using serial PNet.
Wherein, described training and test, zero point operation network can carry out terminal instruction with the monitor mode for losing network
Practice；In test phase, content images XcWith guiding signal XsIt is separately input in TNet and PNet, the image Y after generation conversion.
Brief description of the drawings
Fig. 1 is a kind of system flow chart of the image processing method based on real-time zero point image manipulation network of the present invention.
Fig. 2 is a kind of depth convolutional Neural net of the image processing method based on real-time zero point image manipulation network of the present invention
The loss function of network.
Fig. 3 is a kind of serial PNet of the image processing method based on real-time zero point image manipulation network of the present invention.
Fig. 4 is a kind of parallel PNet of the image processing method based on real-time zero point image manipulation network of the present invention.
Embodiment
It should be noted that in the case where not conflicting, the feature in embodiment and embodiment in the application can phase
Mutually combine, the present invention is described in further detail with specific embodiment below in conjunction with the accompanying drawings.
Fig. 1 is a kind of system flow chart of the image processing method based on real-time zero point image manipulation network of the present invention.It is main
To include image procossing and convolutional neural networks (CNN), zero point operation network, training and test.
Image procossing and convolutional neural networks (CNN), give content imagesWith guiding signal (for example, sample
Formula image)Output transform imageSo that the Y and X in contentcIt is similar, while and XsIt is similar；
Therefore, effective expression no less important of study content and pattern, could so carry out rational image procossing.
Zero point operates network, combines image switching network (TNet) and parameter network (PNet), except being come using TNet
Transition diagram picture, goes back training parameter network (PNet), and to produce to guide, signal is the TNet of condition (such as style image) pass
Bond parameter；Because guiding signal is embedded into the communal space by PNet study, so zero point operation network can draw to unknown
Lead signal and carry out zero point image procossing.
Wherein, TNet is normalized with dynamic instance, in order to realize zero point image procossing, is dynamically specified in test process
TNet network parameter, so as to handle invisible signal；Based on guiding signal XsOn PNet regulation characteristic pattern, just
Beginning mode is the wave filter for directly generating TNet；However, in practice, TNet each layer generally has more than 100000
Parameter (for example, 128 × 128 × 3 × 3), and characteristic pattern in PNet each layer generally have about 1000000 entries (for example,
128×80×80)；Therefore, it is difficult to effectively by higher-dimension vector median filters into another.
Using dynamic enhancing example normalization (being performed after each convolutional layer in TNet), the contracting produced by PNet
Put and shift parameters γ (Xs) and β (Xs)；Here, scaling and translocation factor γ (Xs) and β (Xs) be considered as in TNet each layer
Key parameter；In form, before example normalization, orderFor tensor, xijkI-th jk element is represented, its
In, i indicative character figures, j, k crosses over Spatial Dimension；Therefore, the output of dynamic instance normalization (DIN)For：
Wherein, μiIt is the average value in characteristic pattern i,It is corresponding variance；γi(Xs) it is by being similar to that PNet is generated
βi(Xs) CiDimension vector γ (Xs) i-th of element；If γi(Xs)=1, βi(Xs)=0, then DIN be changed into example normalizing
Change；If γi(Xs)=γi,βi(Xs)=βi, then they are as the direct learning parameter unrelated with PNet；DIN will be changed into condition
Example is normalized；In both cases, the model all loses the ability of zero point study, therefore can not be generalized to and see not
The signal seen.
Parameter network (PNet), in order to have serial or parallel frame with dynamic instance normalization driving TNet, PNet
Structure.
Training and test, zero point operation network can carry out terminal training with the monitor mode for losing network；In test rank
Section, content images XcWith guiding signal XsIt is separately input in TNet and PNet, the image Y after generation conversion.
Fig. 2 is a kind of depth convolutional Neural net of the image processing method based on real-time zero point image manipulation network of the present invention
The loss function of network.Using constant depth CNN φ (), the Feature Mapping in l layersImage can be represented
X content, and φl(X) form matrix is expressed asIt is calculated as：
Above formula can represent the desired style images of image X；Only when each correspondence represents (that is, φlOr G (φ (X)l
(X) difference between) is this black norm of not Luo Beini one small, could assess including two images similar in perhaps style；
It therefore, it can training feedforward image converting network Y=T (Xc), typically depth CNN, with loss function：
Wherein,It is the style loss for generating image Y,It is contents lost, λsAnd λcIt is hyper parameter；S is " sample
The set of formula layer ", C is the set of " content layer ", ZlIt is the sum of neuron in l layers；Train after converting network T (), give
Fixed new content images X 'c, stylization image Y=T (X ' are generated in the case of without using network is lostc)。
Fig. 3 is a kind of serial PNet of the image processing method based on real-time zero point image manipulation network of the present invention.In string
In row PNet, the depth CNN with the structure similar to TNet can be used, γ is generated in l layers(l)(Xs) and β(l)(Xs)；
In serial PNet, the γ of formula (3) on adjustment characteristic pattern(l)(Xs) and β(l)(Xs), it is expressed as l layers in PNet of ψl(Xs)；
Wherein, if input XsImage, then ψl(Xs) be convolutional layer in TNet output；If inputting XsIt is word insertion
(vector), then ψl(Xs) be the layer being fully connected output；It is the parameter to be learnt.
Fig. 4 is a kind of parallel PNet of the image processing method based on real-time zero point image manipulation network of the present invention.Use
Individually shallow network (be fully connected or convolution) generates ψ in l layersl(Xs), meter is then used it for according to formula (4) and (5)
Calculate r(l)(Xs) and β(l)(Xs)；With the ψ from higher levell(Xs) generation higher level γ(l)(Xs) and β(l)(Xs) serial PNet
Difference, is from X heresTo γ(l)(Xs) and β(l)(Xs) shallow and parallel structure is followed, but this structure will limit having for PNet
Effect property, and the TNet of generation quality is somewhat reduced, and therefore produce image Y；Therefore, mostly using serial PNet.
For those skilled in the art, the present invention is not restricted to the details of above-described embodiment, in the essence without departing substantially from the present invention
In the case of refreshing and scope, the present invention can be realized with other concrete forms.In addition, those skilled in the art can be to this hair
Bright to carry out various changes and modification without departing from the spirit and scope of the present invention, these improvement and modification also should be regarded as the present invention's
Protection domain.Therefore, appended claims are intended to be construed to include preferred embodiment and fall into all changes of the scope of the invention
More and modification.
Claims (10)
1. a kind of image processing method based on real-time zero point image manipulation network, it is characterised in that mainly including image procossing
With convolutional neural networks (CNN) (one)；Zero point operation network (two)；Training and test (three).
2. based on the image procossing described in claims 1 and convolutional neural networks (CNN), it is characterised in that given content graph
PictureWith guiding signal (for example, pattern diagram picture)Output transform imageMake
Obtain Y and the X in contentcIt is similar, while and XsIt is similar；Therefore, effective expression no less important of study content and pattern, so
Rational image procossing can be carried out.
3. based on the convolutional neural networks (CNN) described in claims 1, it is characterised in that use constant depth CNN φ
(), the Feature Mapping in l layersImage X content, and φ can be representedl(X) form matrix table
It is shown asIt is calculated as：
Above formula can represent the desired style images of image X；Only when each correspondence represents (that is, φlOr G (φ (X)l(X)) it
Between difference be this black norm of one small not Luo Beini, could assess including two images similar in perhaps style；Therefore, may be used
To train feedforward image converting network Y=T (Xc), typically depth CNN, with loss function：
Wherein,It is the style loss for generating image Y,It is contents lost, λsAnd λcIt is hyper parameter；S is " pattern layer "
Set, C is the set of " content layer ", ZlIt is the sum of neuron in l layers；Train after converting network T (), give newly
Content images X 'c, stylization image Y=T (X ' are generated in the case of without using network is lostc)。
4. based on the zero point operation network (two) described in claims 1, it is characterised in that zero point operates network integration image
Switching network (TNet) and parameter network (PNet), except carrying out transition diagram picture using TNet, go back training parameter network (PNet), with
Produce to guide signal as the TNet of condition (such as style image) key parameter；Because PNet study will guiding signal insertion
Into the communal space, so zero point operation network can carry out zero point image procossing to unknown guiding signal.
5. normalized based on the TNet described in claims 4 and dynamic instance, it is characterised in that in order to realize at zero point image
Reason, dynamically specifies TNet network parameter, so as to handle invisible signal in test process；Based on guiding signal
XsOn PNet regulation characteristic pattern, initialization mode is the wave filter for directly generating TNet；However, in practice, TNet's
Each layer generally has more than 100000 parameters (for example, 128 × 128 × 3 × 3), and the characteristic pattern in PNet each layer leads to
Often there are about 1000000 entries (for example, 128 × 80 × 80)；Therefore, it is difficult to effectively by higher-dimension vector median filters into another.
6. based on the dynamic instance normalization described in claims 5, it is characterised in that using dynamic enhancing example normalization
(being performed after each convolutional layer in TNet), passes through the PNet scalings produced and shift parameters γ (Xs) and β (Xs)；Here,
Scaling and translocation factor γ (Xs) and β (Xs) it is considered as the key parameter in TNet each layer；In form, in example normalization
Before, makeFor tensor, xijkI-th jk element is represented, wherein, i indicative character figures, j, k crosses over space dimension
Degree；Therefore, the output of dynamic instance normalization (DIN)For：
<mrow>
<mtable>
<mtr>
<mtd>
<mrow>
<msub>
<mi>y</mi>
<mrow>
<mi>i</mi>
<mi>j</mi>
<mi>k</mi>
</mrow>
</msub>
<mo>=</mo>
<mfrac>
<mrow>
<msub>
<mi>x</mi>
<mrow>
<mi>i</mi>
<mi>j</mi>
<mi>k</mi>
</mrow>
</msub>
<mo>-</mo>
<msub>
<mi>&mu;</mi>
<mi>i</mi>
</msub>
</mrow>
<msqrt>
<mrow>
<msubsup>
<mi>&sigma;</mi>
<mi>i</mi>
<mn>2</mn>
</msubsup>
<mo>+</mo>
<mo>&Element;</mo>
</mrow>
</msqrt>
</mfrac>
<msub>
<mi>&gamma;</mi>
<mi>i</mi>
</msub>
<mrow>
<mo>(</mo>
<msub>
<mi>X</mi>
<mi>s</mi>
</msub>
<mo>)</mo>
</mrow>
<mo>+</mo>
<msub>
<mi>&beta;</mi>
<mi>i</mi>
</msub>
<mrow>
<mo>(</mo>
<msub>
<mi>X</mi>
<mi>s</mi>
</msub>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<msub>
<mi>&mu;</mi>
<mi>i</mi>
</msub>
<mo>=</mo>
<mfrac>
<mn>1</mn>
<mrow>
<mi>H</mi>
<mi>W</mi>
</mrow>
</mfrac>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>j</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mi>H</mi>
</munderover>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>k</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mi>W</mi>
</munderover>
<msub>
<mi>x</mi>
<mrow>
<mi>i</mi>
<mi>j</mi>
<mi>k</mi>
</mrow>
</msub>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<msubsup>
<mi>&sigma;</mi>
<mi>i</mi>
<mn>1</mn>
</msubsup>
<mo>=</mo>
<mfrac>
<mn>1</mn>
<mrow>
<mi>H</mi>
<mi>W</mi>
</mrow>
</mfrac>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>j</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mi>H</mi>
</munderover>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>k</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mi>W</mi>
</munderover>
<mrow>
<mo>(</mo>
<msub>
<mi>x</mi>
<mrow>
<mi>i</mi>
<mi>j</mi>
<mi>k</mi>
</mrow>
</msub>
<mo>-</mo>
<msub>
<mi>&mu;</mi>
<mi>i</mi>
</msub>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
</mtr>
</mtable>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>3</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein, μiIt is the average value in characteristic pattern i, σi 2It is corresponding variance；γi(Xs) it is to be similar to β by what PNet was generatedi(Xs)
CiDimension vector γ (Xs) i-th of element；If γi(Xs)=1, βi(Xs)=0, then DIN be changed into example normalization；If
γi(Xs)=γi,βi(Xs)=βi, then they are as the direct learning parameter unrelated with PNet；DIN will be changed into condition example and return
One changes；In both cases, the model all loses the ability of zero point study, therefore can not be generalized to invisible letter
Number.
7. based on the parameter network (PNet) described in claims 4, it is characterised in that driven to be normalized with dynamic instance
TNet, PNet can have serial or parallel framework.
8. based on the serial PNet described in claims 7, it is characterised in that in serial PNet, it can use with similar
In the depth CNN of TNet structure, γ is generated in l layers(l)(Xs) and β(l)(Xs)；It is public on adjustment characteristic pattern in serial PNet
The γ of formula (3)(l)(Xs) and β(l)(Xs), it is expressed as l layers in PNet of ψl(Xs)；
<mrow>
<msup>
<mi>&gamma;</mi>
<mrow>
<mo>(</mo>
<mi>l</mi>
<mo>)</mo>
</mrow>
</msup>
<mrow>
<mo>(</mo>
<msub>
<mi>X</mi>
<mi>s</mi>
</msub>
<mo>)</mo>
</mrow>
<mo>=</mo>
<msub>
<mi>&psi;</mi>
<mi>l</mi>
</msub>
<mrow>
<mo>(</mo>
<msub>
<mi>X</mi>
<mi>S</mi>
</msub>
<mo>)</mo>
</mrow>
<msubsup>
<mi>W</mi>
<mi>&gamma;</mi>
<mrow>
<mo>(</mo>
<mi>l</mi>
<mo>)</mo>
</mrow>
</msubsup>
<mo>+</mo>
<msubsup>
<mi>b</mi>
<mi>&gamma;</mi>
<mrow>
<mo>(</mo>
<mi>l</mi>
<mo>)</mo>
</mrow>
</msubsup>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>4</mn>
<mo>)</mo>
</mrow>
</mrow>
<mrow>
<msup>
<mi>&beta;</mi>
<mrow>
<mo>(</mo>
<mi>l</mi>
<mo>)</mo>
</mrow>
</msup>
<mrow>
<mo>(</mo>
<msub>
<mi>X</mi>
<mi>s</mi>
</msub>
<mo>)</mo>
</mrow>
<mo>=</mo>
<msub>
<mi>&psi;</mi>
<mi>l</mi>
</msub>
<mrow>
<mo>(</mo>
<msub>
<mi>X</mi>
<mi>s</mi>
</msub>
<mo>)</mo>
</mrow>
<msubsup>
<mi>W</mi>
<mi>&beta;</mi>
<mrow>
<mo>(</mo>
<mi>l</mi>
<mo>)</mo>
</mrow>
</msubsup>
<mo>+</mo>
<msubsup>
<mi>b</mi>
<mi>&beta;</mi>
<mrow>
<mo>(</mo>
<mi>l</mi>
<mo>)</mo>
</mrow>
</msubsup>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>5</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein, if input XsImage, then ψl(Xs) be convolutional layer in TNet output；If inputting XsBe word insertion (to
Amount), then ψl(Xs) be the layer being fully connected output；It is the parameter to be learnt.
9. based on the parallel PNet described in claims 7, it is characterised in that (be fully connected or roll up using single shallow network
Long-pending) generate ψ in l layersl(Xs), then used it for calculating γ according to formula (4) and (5)(l)(Xs) and β(l)(Xs)；With from
The ψ of higher levell(Xs) generation higher level γ(l)(Xs) and β(l)(Xs) serial PNet it is different, be from X heresTo γ(l)
(Xs) and β(l)(Xs) shallow and parallel structure is followed, but this structure will limit PNet validity, and somewhat reduction generation
TNet quality, and therefore produce image Y；Therefore, mostly using serial PNet.
10. based on the training described in claims 1 and test (three), it is characterised in that zero point operation network can be with loss
The monitor mode of network carries out terminal training；In test phase, content images XcWith guiding signal XsBe separately input to TNet and
In PNet, the image Y after generation conversion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710531090.0A CN107330852A (en) | 2017-07-03 | 2017-07-03 | A kind of image processing method based on real-time zero point image manipulation network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710531090.0A CN107330852A (en) | 2017-07-03 | 2017-07-03 | A kind of image processing method based on real-time zero point image manipulation network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107330852A true CN107330852A (en) | 2017-11-07 |
Family
ID=60197679
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710531090.0A Withdrawn CN107330852A (en) | 2017-07-03 | 2017-07-03 | A kind of image processing method based on real-time zero point image manipulation network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107330852A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107730474A (en) * | 2017-11-09 | 2018-02-23 | 京东方科技集团股份有限公司 | Image processing method, processing unit and processing equipment |
| WO2019144608A1 (en) * | 2018-01-26 | 2019-08-01 | 京东方科技集团股份有限公司 | Image processing method, processing apparatus and processing device |
| WO2020073758A1 (en) * | 2018-10-10 | 2020-04-16 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method and apparatus for training machine learning modle, apparatus for video style transfer |
| CN111476341A (en) * | 2019-01-23 | 2020-07-31 | 斯特拉德视觉公司 | Method and device for converting CNN convolution layer |
| CN113114541A (en) * | 2021-06-15 | 2021-07-13 | 上海兴容信息技术有限公司 | Method and system for judging whether network connection can be established between network nodes |
-
2017
- 2017-07-03 CN CN201710531090.0A patent/CN107330852A/en not_active Withdrawn
Non-Patent Citations (1)
| Title |
|---|
| HAO WANG 等: "ZM-net ：real-time Zero-shot image manipulation network", 《网页在线公开:HTTPS://ARXIV.ORG/ABS/1703.07255V1》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107730474A (en) * | 2017-11-09 | 2018-02-23 | 京东方科技集团股份有限公司 | Image processing method, processing unit and processing equipment |
| US10706504B2 (en) | 2017-11-09 | 2020-07-07 | Boe Technology Group Co., Ltd. | Image processing methods and image processing devices |
| WO2019144608A1 (en) * | 2018-01-26 | 2019-08-01 | 京东方科技集团股份有限公司 | Image processing method, processing apparatus and processing device |
| US11281938B2 (en) | 2018-01-26 | 2022-03-22 | Boe Technology Group Co., Ltd. | Image processing method, processing apparatus and processing device |
| WO2020073758A1 (en) * | 2018-10-10 | 2020-04-16 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method and apparatus for training machine learning modle, apparatus for video style transfer |
| CN112823379A (en) * | 2018-10-10 | 2021-05-18 | Oppo广东移动通信有限公司 | Method and device for training machine learning model and device for video style transfer |
| CN111476341A (en) * | 2019-01-23 | 2020-07-31 | 斯特拉德视觉公司 | Method and device for converting CNN convolution layer |
| CN111476341B (en) * | 2019-01-23 | 2024-04-12 | 斯特拉德视觉公司 | Method and device for converting convolutional layer of CNN |
| CN113114541A (en) * | 2021-06-15 | 2021-07-13 | 上海兴容信息技术有限公司 | Method and system for judging whether network connection can be established between network nodes |
| CN113114541B (en) * | 2021-06-15 | 2021-09-14 | 上海兴容信息技术有限公司 | Method and system for judging whether network connection can be established between network nodes |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107330852A (en) | A kind of image processing method based on real-time zero point image manipulation network | |
| CN108734749A (en) | The visual style of image converts | |
| CN105374007A (en) | Generation method and generation device of pencil drawing fusing skeleton strokes and textural features | |
| CN105488472A (en) | Digital make-up method based on sample template | |
| CN106875361A (en) | A kind of method that poisson noise is removed based on depth convolutional neural networks | |
| CN106408595A (en) | Neural network painting style learning-based image rendering method | |
| CN101556699A (en) | Face-based facial aging image synthesis method | |
| US20200327316A1 (en) | Interactive method for generating strokes with chinese ink painting style and device thereof | |
| CN105631807A (en) | Single-frame image super resolution reconstruction method based on sparse domain selection | |
| CN107203969A (en) | A kind of high magnification image super-resolution rebuilding method of medium scale constraint | |
| CN106023276A (en) | Pencil drawing making method and pencil drawing making device based on image processing | |
| CN106971189A (en) | A kind of noisy method for recognising star map of low resolution | |
| CN104091364B (en) | Single-image super-resolution reconstruction method | |
| CN102831621B (en) | Video significance processing method based on spectral analysis | |
| Zhang et al. | CSAST: Content self-supervised and style contrastive learning for arbitrary style transfer | |
| Zheng et al. | CFA-GAN: Cross fusion attention and frequency loss for image style transfer | |
| CN106683129A (en) | Method for decomposing relatively reduced texture based on bilateral kernel regression and device thereof | |
| CN107169948A (en) | A kind of visual characteristic moving method based on depth image analogy | |
| CN104077798B (en) | High-reality-sense animation synthesis method for deformable object | |
| CN106354449A (en) | Document online demonstration method and client | |
| Wang | Evolution of StyleGAN3 | |
| CN114037644A (en) | Artistic digital image synthesis system and method based on generation countermeasure network | |
| CN107256534A (en) | The diminution of GQIR quantum images based on bilinear interpolation and amplification method | |
| CN103729835B (en) | A kind of vegetation parameter approximating method based on middle high-resolution remote sensing | |
| CN113111906A (en) | Method for generating confrontation network model based on condition of single pair image training |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20171107 |
|
| WW01 | Invention patent application withdrawn after publication |