CN109978804A - Human eye sight antidote and system based on deep learning - Google Patents
Human eye sight antidote and system based on deep learning Download PDFInfo
- Publication number
- CN109978804A CN109978804A CN201910175164.0A CN201910175164A CN109978804A CN 109978804 A CN109978804 A CN 109978804A CN 201910175164 A CN201910175164 A CN 201910175164A CN 109978804 A CN109978804 A CN 109978804A
- Authority
- CN
- China
- Prior art keywords
- human eye
- image
- network
- deep learning
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004438 eyesight Effects 0.000 title claims abstract description 52
- 238000013135 deep learning Methods 0.000 title claims abstract description 35
- 239000000729 antidote Substances 0.000 title claims abstract description 19
- 230000007547 defect Effects 0.000 claims abstract description 56
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 39
- 238000012937 correction Methods 0.000 claims description 40
- 238000012549 training Methods 0.000 claims description 21
- 238000001514 detection method Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 11
- 238000006467 substitution reaction Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 8
- 238000013527 convolutional neural network Methods 0.000 claims description 7
- 238000001228 spectrum Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 33
- 230000002708 enhancing effect Effects 0.000 abstract description 7
- 101100117236 Drosophila melanogaster speck gene Proteins 0.000 abstract description 4
- 230000006399 behavior Effects 0.000 description 6
- 238000005286 illumination Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000005457 optimization Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 210000003128 head Anatomy 0.000 description 3
- 238000002715 modification method Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of human eye sight antidote and system based on deep learning, wherein this method comprises: obtaining human eye picture；The human eye that network is handled to obtain the thick stage to human eye picture, which is distorted, by coarse adjustment generates image；Human eye is detected by fine corrective networks and generates the defects of image region, and defect area is modified.Human eye picture of this method for input, the generation image in thick stage is obtained using the method based on distortion, it reuses the cyclic policy network based on depth enhancing study and detects the defects of the image of thick stage output region, it is effectively reduced the error generated between image and true picture, the defects of vision and feeling of unreality in image are eliminated, while the image details such as reflective speck can be restored.
Description
Technical field
The present invention relates to digital image processing techniques field, in particular to a kind of human eye sight correction based on deep learning
Method and system.
Background technique
It is exactly to handle the picture of the eyes of people that sight, which corrects (Gaze Correction), changes human eye in picture
Direction of visual lines.Sight amendment has actual value and extensive prospect in the exchange scene such as video calling.But due to
The image or video of human eye can be in sizes in acquisition, resolution ratio, visual angle, illumination, texture, and block etc. has larger change
Change, vision amendment problem in the real world is still a challenging problem.
Currently existing sight modification method is broadly divided into two classes: the sight based on figure is corrected and based on pixel distortion
Sight amendment.For the first kind, the sight amendment based on figure is mainly using the 3D human-eye model with artificial texture come mould
The continuous movement of anthropomorphic eye and head, by using the extensive color applying drawing of geometry of dynamic and controllable human eye area model
Eye image.However, the eye image and true eye image gap after such methods synthesis are larger.It is needed simultaneously in application
The 3D model of human eye, but construct 3D model cost it is very high, cause such method in practical applications have significant limitation.It is right
In the second class, the sight modification method based on distortion is the flow field that distortion is predicted by study warp function, thus directly
The revised image of sight is generated by original eye image.Such as Gain et al. proposes a kind of depth feed-forward system,
It combines the processing of thickness two-stage, scalloping, the operating principles such as intensity correction.Kononenko et al. proposes a kind of logical
Cross random forest time span of forecast and realize can on CPU (central processing unit, Central Processing Unit) real time execution
Human eye distort field method, due to warp function be it is specific to posture, can use with different sight direction and head
The eye image of portion's posture synthesizes image more true to nature, has solved head pose and sight angle variation in practical application.
However, eye image usually has complicated texture, illumination, situations such as blocking, the influence of these specific factors is difficult by whole
Body amendment operation is to complete.As shown in Figure 1, only can have obvious shortcoming and feeling of unreality is asked using the image that warping method generates
Topic.
In recent years, in-depth intensity study (Deep Reinforcement Learning) all takes in the application of various visions
Obtained significant success, such as object detection, object tracking, object search and action recognition.Current depth enhances learning method
Two classes: depth Q study and Policy-Gradient can be divided into.For the first kind, Q value is fitted to capture and take spy in a particular state
Determine the adaptive expectations of behavior.For example, a kind of cooperation depth that Kong et al. is proposed enhances learning method, combine in iteration several times
Position object.For the second class, tactful distribution is explicitly represented in the vmcs, and optimizes plan by updating the parameter on gradient direction
Slightly.Liu et al. people fights network respectively using a kind of Policy-Gradient method to optimize title measurement and production.Recently, depth increases
Strong study has played important function in recognition of face and synthesis.
Summary of the invention
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, an object of the present invention is to provide a kind of human eye sight antidote based on deep learning, the party
Method is effectively reduced the error generated between image and true picture, eliminates the defects of vision and feeling of unreality in image, simultaneously
It can restore the image details such as reflective speck.
It is another object of the present invention to propose a kind of human eye sight correction system based on deep learning.
In order to achieve the above objectives, one aspect of the present invention embodiment proposes a kind of human eye sight correction based on deep learning
Method, comprising: obtain human eye picture；It distorts network by coarse adjustment the human eye picture is handled to obtain the human eye in thick stage
Generate image；Detect the human eye by fine corrective networks and generate the defects of image region, and to the defect area into
Row amendment.
The human eye sight antidote based on deep learning of the embodiment of the present invention utilizes the human eye picture of input
Method based on distortion obtains the generation image in thick stage, and it is thick to reuse the cyclic policy network detection based on depth enhancing study
The defects of the image of stage output region.By considering the local correction network of overall Vision feature to the defect area detected
Domain carries out refine, greatly eliminates due to illumination, texture, blocks and waits the specific factors bring defects of vision and non-genuine property.
In addition, the human eye sight antidote according to the above embodiment of the present invention based on deep learning can also have with
Under additional technical characteristic:
Further, in one embodiment of the invention, network is distorted to the human eye picture by coarse adjustment described
Handled to obtain the thick stage human eye generate image before, further includes: the training one thick convolutional neural networks for arriving fine texture
To generate a two-dimensional compensation map；The substitution operation that the compensation map carries out Pixel-level to original image generates training figure
Picture；The coarse adjustment is trained to turn round as loss function using the mean square error between the generation training image and the original image
Bent network.
Further, in one embodiment of the invention, the fine corrective networks, comprising: cyclic policy network and
The corrective networks of part；
It is described to detect the defects of human eye generation image region by fine corrective networks, and to the defect area
It is modified, comprising: the cyclic policy network detects the human eye and generates the defects of image region；The local amendment
Network is modified the defect area by convolutional layer.
Further, in one embodiment of the invention, the cyclic policy network detects the human eye and generates image
The defects of region, comprising:
The image I of given t stept-1, the cyclic policy network by selection one part block coordinate position ltCome
Select a block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of the cyclic policy network, by input picture It-1After coding
History hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In give
L is set in positioningtBlock cut down as a result.
Further, in one embodiment of the invention, the local corrective networks are by convolutional layer come to described
Defect area is modified, comprising: to be modified piece of each step selectionWith the local correction network feIt is modified,
It obtains one and passes through modified piece, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept,
After T step, final image I has been obtainedT。
In order to achieve the above objectives, it is strong to propose a kind of human eye sight based on deep learning for another aspect of the present invention embodiment
Positive system, comprising: module is obtained, for obtaining human eye picture；Processing module, for distorting network to the human eye by coarse adjustment
The human eye that picture is handled to obtain the thick stage generates image；Correction module, for detecting the people by fine corrective networks
The defects of image region is looked unfamiliar into, and the defect area is modified.
The human eye sight correction system based on deep learning of the embodiment of the present invention utilizes the human eye picture of input
Method based on distortion obtains the generation image in thick stage, and it is thick to reuse the cyclic policy network detection based on depth enhancing study
The defects of the image of stage output region.By considering the local correction network of overall Vision feature to the defect area detected
Domain carries out refine, greatly eliminates due to illumination, texture, blocks and waits the specific factors bring defects of vision and non-genuine property.
In addition, the human eye sight correction system according to the above embodiment of the present invention based on deep learning can also have with
Under additional technical characteristic:
Further, in one embodiment of the invention, further includes: generation module, the generation module is for training
One thick to generate a two-dimensional compensation map to the convolutional neural networks of fine texture, the compensation map to original image into
The substitution operation of row Pixel-level generates training image, utilizes the mean square error between the generation training image and the original image
The coarse adjustment distortion network is trained as loss function.
Further, in one embodiment of the invention, the fine corrective networks, comprising: cyclic policy network and
The corrective networks of part；
The correction module, comprising: detection unit and amending unit；
The detection unit detects the human eye for the cyclic policy network and generates the defects of image region；Institute
Amending unit is stated, the defect area is modified by convolutional layer for the local corrective networks.
Further, in one embodiment of the invention, the detection unit, is specifically used for:
The image I of given t stept-1, the cyclic policy network by selection one part block coordinate position ltCome
Select a block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of the cyclic policy network, by input picture It-1After coding
History hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In give
L is set in positioningtBlock cut down as a result.
Further, in one embodiment of the invention, the amending unit, is specifically used for:
To be modified piece of each step selectionWith the local correction network feIt is modified, obtains one by repairing
Positive block, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept, after T step, obtain
Final image IT。
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is the image schematic diagram generated using warping method according to one embodiment of the invention；
Fig. 2 is the human eye sight antidote flow chart based on deep learning according to one embodiment of the invention；
Fig. 3 is the human eye sight antidote flow chart based on deep learning according to a specific embodiment of the invention；
Fig. 4 is the structural schematic diagram that network is distorted according to the coarse adjustment of one embodiment of the invention；
Fig. 5 is the structure chart according to the cyclic policy network of one embodiment of the invention；
Fig. 6 is the local correction flow through a network figure according to one embodiment of the invention；
Fig. 7 is the human eye sight correction system structural schematic diagram based on deep learning according to one embodiment of the invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
The human eye sight antidote based on deep learning proposed according to embodiments of the present invention is described with reference to the accompanying drawings
And system.
The human eye sight correction side based on deep learning proposed according to embodiments of the present invention is described with reference to the accompanying drawings first
Method.
Fig. 2 is the human eye sight antidote flow chart based on deep learning according to one embodiment of the invention.
As shown in Fig. 2, should human eye sight antidote based on deep learning the following steps are included:
In step s101, human eye picture is obtained.
In step s 102, network is distorted by coarse adjustment human eye picture is handled to obtain the human eye generation figure in thick stage
Picture.
Further, in one embodiment of the invention, before step S102 further include: training one thick to thin knot
The convolutional neural networks of structure generate a two-dimensional compensation map；The substitution that map carries out Pixel-level to original image is compensated to grasp
Make generation training image；Coarse adjustment is trained to turn round as loss function using the mean square error generated between training image and original image
Bent network.
Specifically, generate through the above steps coarse adjustment distortion network come to human eye picture acquired in step s101 into
Row preliminary treatment, the human eye for obtaining the thick stage generate image.
In step s 103, human eye is detected by fine corrective networks and generate the defects of image region, and to defect area
Domain is modified.
Further, in one embodiment of the invention, fine corrective networks, comprising: cyclic policy network and part
Corrective networks.
Human eye is detected by fine corrective networks and generates the defects of image region, and defect area is modified, and is wrapped
Include: cyclic policy network detects human eye and generates the defects of image region；The corrective networks of part are by convolutional layer come to defect
Region is modified.
Wherein, cyclic policy network detection human eye generates the defects of image region, comprising:
The image I of given t stept-1, cyclic policy network by selection one part block coordinate position ltTo select
One block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of cyclic policy network, by input picture It-1History after coding
Hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In to positioning
Set ltBlock cut down as a result.
Further, in one embodiment of the invention, local corrective networks are by convolutional layer come to defect area
It is modified, comprising: to be modified piece of each step selectionWith local correction network feIt is modified, obtains a process
Modified piece, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept, after T step, obtain
Final image IT。
The method of the embodiment of the present invention is different from the method based on distortion and the whole modification method to generation image, whole
It is obviously improved in body effect.New concern area is dynamically gradually specified using based on depth enhancing study circulation strategy network
Domain, can detecte thick stage generation image contains defects of vision block (patch).Utilize the part for considering overall Vision feature
Corrective networks are modified the block detected, are effectively reduced the error generated between image and true picture, eliminate in image
The defects of vision and feeling of unreality, while restoring the image details such as reflective speck.
The embodiment of the present invention carries out sight amendment by dual-stage method from thick to thin, as shown in figure 3, for one
The change angle [alpha] of given human eye picture I and sight are opened, the method for the embodiment of the present invention is divided into two parts: coarse adjustment distorts net
Network (CWN, Coarse Warping Network) and fine corrective networks (FCN, Fine Corrected Network).Its
In, CWN is used for the first step, substitutes operation by pixel integrally to correct image.And FCN is used for second step, refine CWN output
Image to increase generate image authenticity.
The human eye sight antidote based on deep learning of the embodiment of the present invention is introduced below by specific embodiment.
1. coarse adjustment distorts network (CWN)
The task that coarse adjustment distorts network is the distortion flow field generated for distorting original image.To reach this purpose, instruct
Practice one and thick generates a two-dimensional compensation map to the convolutional neural networks of fine texture.The map to each pixel (x,
Y) there is a compensation vector (u (x, y), v (x, y)).This compensation map is used to carry out original image the substitution of Pixel-level
Operation.The calculation method of warp image is as follows:
O (x, y)=I (x+u (x, y), y+v (x, y))
The pixel for generating each point of image as a result, be all with a pixel " substitution " in original image,
The position of the point of substitution is determined by compensation vector.
Using original image, sight change angle and the human eye feature point position detected as the defeated of coarse adjustment distortion network
Enter, network will generate the map D in two channelsC.Using lower section distortion original image I to generate coarse adjustment warp image OC:
OC(x, y)=I { (x, y)+DC(x, y) }
Wherein, braces represents bilinearity difference operation.
Using the mean square error (Mean Squared Error, MSE) generated between image and real image as loss letter
Number is to train CWN.The specific network structure of CWN is as shown in Fig. 4.
2. fine corrective networks (FCN)
The result that coarse adjustment distortion network generates usually contains local defect, seriously affects the authenticity of picture.To solve
This problem, the image generated with fine corrective networks come refine coarse adjustment network lock.
Fine corrective networks are broadly divided into two parts: (1) a cyclic policy network each step all select one it is to be repaired
The corrective networks of positive (2) parts of block, by convolutional layer come defective piece of correction tape.The loop-body process of FCN is such as
Under:
The image I of given t stept-1, cyclic policy network by selection one part block coordinate position ltTo select
One block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of cyclic policy network, by input picture It-1, history after coding
Hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In to positioning
Set ltBlock cut down as a result.
Later to be modified piece of the selection of each stepWith local correction network feIt is modified, obtains a process
Modified piece, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept.After T step, we
Final image I is obtainedT。
The specific implementation of above-mentioned cyclic policy network is as follows: regarding this process as under discrete time interval
Markovian decision problem.In each step, which of eye image be the current state feature of decision networks team encoded and determined
A part needs to be corrected.Before reaching maximum step number, the block of eye image is corrected step by step, and state feature is also always by more
Newly.
In Orders Corrected finally, the global reward an of delay is taken to carry out the training of guiding strategies network.Tactful network
A best searching route is iteratively explored, to make each independent eye image that can reach maximum global prize
It encourages, the CONSTRUCTED SPECIFICATION of network is as shown in Figure 5.
Wherein, the state of tactful network, behavior and reward are specifically provided that
State: state stIt is extracted from the input picture and behavior over history currently walked, includes three parts:
(1) from the eye image I currently walkedtThe characteristic spectrum of middle extraction, by with convolutional network knot identical in local corrective networks
Structure extracts this feature, and specific structure can be in subsequent explanation.(2) the position I of the block of previous step selectiont-1.(3) LSTM layers
Hidden unit ht, wherein LSTM uses GRU network structure.
Behavior: in tactful network, behavior is exactly the position for selecting this step to want modified piece from all possible position
It sets.This network is first encoded the block position that the characteristic spectrum of present image and previous step select by full articulamentum,
Vector h is hidden in combination with the vector sum history obtained after codingt-1To generate new hidden unit ht, last tactful network πθ
From htIn obtain the position l of this stept。
Reward: reward is used to refer to lead how e-learning selects a series of behavior to be optimal final defeated
Out.With the mean square error loss between final output image and true picture as the reward of network.In addition, only in final step
Final delay reward is generated, the error of each step is all not counted in training among network.Therefore in the following institute of reward r of t step
Show:
Wherein, IgtRepresent true picture.In the method for the present embodiment, discount factor γ is set as 1, i.e., each step
Correct the evaluation no less important for final result.
Local correction network is specific as follows:
The position l obtained according to cyclic policy networktCome to image It-1It is cut out to obtain this step block to be modifiedBy position ltIt is encoded, by itself and blockMerge, the input as network.By containing a series of convolutional layer
Depth convolutional network obtains a residual error map δ, by its value be applied directly to modification before block, and using result as modify after
Block substitute original block.Detailed process is as shown in Figure 6.
Optimal method:
Come joint training cyclic policy network and corrective networks are refused with the framework of enhancing study.The entirety of optimization problem
Formula is as follows:
Firstly, utilizing following formula optimization cyclic policy network { μ, ∑ }=πθ(st):
Here using the probability distribution being just distributed very much as action selection
Secondly, optimization local correction network:
Local convolutional network can all carry out parameter update and optimization in each step.Still mean square error is used to miss as passback
The loss function of difference.It will not influence the parameter of cyclic policy network for the optimization process of local convolutional network.
The human eye sight antidote flow chart based on deep learning proposed according to embodiments of the present invention, for input
Human eye picture obtains the generation image in thick stage using the method based on distortion, reuses the circulation based on depth enhancing study
Tactful network detects the defects of the image of thick stage output region.By the local correction network pair for considering overall Vision feature
The defect area detected carries out refine, greatly eliminates due to illumination, texture, blocks and waits the specific factors bring defects of vision
And non-genuine property.
The human eye sight correction system based on deep learning proposed according to embodiments of the present invention is described referring next to attached drawing.
Fig. 7 is the human eye sight correction system structural schematic diagram based on deep learning according to one embodiment of the invention.
As shown in fig. 7, the human eye sight correction system includes: to obtain module 100, processing module 200 and correction module
300。
Wherein, module 100 is obtained for obtaining human eye picture.Processing module 200 is used to distort network to people by coarse adjustment
The human eye that eye picture is handled to obtain the thick stage generates image.Correction module 300 is used to detect people by fine corrective networks
The defects of image region is looked unfamiliar into, and defect area is modified.The system, which is effectively reduced, generates image and true figure
Error as between eliminates the defects of vision and feeling of unreality in image, while can restore the image details such as reflective speck.
Further, in one embodiment of the invention, further includes: generation module, generation module is for training one
A two-dimensional compensation map is slightly generated to the convolutional neural networks of fine texture, compensation map carries out Pixel-level to original image
Substitution operation generate training image, instructed using the mean square error generated between training image and original image as loss function
Practice coarse adjustment and distorts network.
Further, in one embodiment of the invention, fine corrective networks, comprising: cyclic policy network and part
Corrective networks；
Correction module, comprising: detection unit and amending unit；
Detection unit generates the defects of image region for cyclic policy network detection human eye；
Amending unit, the corrective networks for part are modified defect area by convolutional layer.
Further, in one embodiment of the invention, detection unit is specifically used for:
The image I of given t stept-1, cyclic policy network by selection one part block coordinate position ltTo select
One block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of cyclic policy network, by input picture It-1History after coding
Hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In to positioning
Set ltBlock cut down as a result.
Further, in one embodiment of the invention, amending unit is specifically used for:
To be modified piece of each step selectionWith local correction network feIt is modified, obtains one by modified
Block, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept, after T step, obtained final
Image IT。
It should be noted that the aforementioned explanation to the human eye sight antidote embodiment based on deep learning is also fitted
For the system of the embodiment, details are not described herein again.
The human eye sight correction system based on deep learning proposed according to embodiments of the present invention, for the human eye figure of input
Piece obtains the generation image in thick stage using the method based on distortion, reuses the cyclic policy net based on depth enhancing study
Network detects the defects of the image of thick stage output region.By considering the local correction network of overall Vision feature to detecting
Defect area carry out refine, greatly eliminate due to illumination, texture, block the equal specific factors bring defects of vision and non-real
Reality.
In addition, term " first ", " second " are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance
Or implicitly indicate the quantity of indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, the meaning of " plurality " is at least two, such as two, three
It is a etc., unless otherwise specifically defined.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It can be combined in any suitable manner in one or more embodiment or examples.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the feature of different embodiments or examples described in this specification and different embodiments or examples
It closes and combines.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of human eye sight antidote based on deep learning, which comprises the following steps:
Obtain human eye picture；
The human eye that network is handled to obtain the thick stage to the human eye picture, which is distorted, by coarse adjustment generates image；
The human eye is detected by fine corrective networks and generates the defects of image region, and the defect area is repaired
Just.
2. the human eye sight antidote according to claim 1 based on deep learning, which is characterized in that pass through described
Coarse adjustment distorts before the human eye that network is handled to obtain the thick stage to the human eye picture generates image, further includes:
Training one is thick to generate a two-dimensional compensation map to the convolutional neural networks of fine texture；
The substitution operation that the compensation map carries out Pixel-level to original image generates training image；
The coarse adjustment is trained as loss function using the mean square error between the generation training image and the original image
Distort network.
3. the human eye sight antidote according to claim 1 based on deep learning, which is characterized in that described finely to repair
Positive network, comprising: the corrective networks of cyclic policy network and part；
It is described to detect the defects of human eye generation image region by fine corrective networks, and the defect area is carried out
Amendment, comprising:
The cyclic policy network detects the human eye and generates the defects of image region；
The local corrective networks are modified the defect area by convolutional layer.
4. the human eye sight antidote according to claim 3 based on deep learning, which is characterized in that the circulation plan
Slightly network detects the human eye and generates the defects of image region, comprising:
The image I of given t stept-1, the cyclic policy network by selection one part block coordinate position ltTo select
One block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of the cyclic policy network, by input picture It-1History after coding
Hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In to positioning
Set ltBlock cut down as a result.
5. the human eye sight antidote according to claim 4 based on deep learning, which is characterized in that
The local corrective networks are modified the defect area by convolutional layer, comprising:
To be modified piece of each step selectionWith the local correction network feIt is modified, obtains one by modified
Block, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept, after T step, obtained final
Image IT。
6. a kind of human eye sight correction system based on deep learning characterized by comprising
Module is obtained, for obtaining human eye picture；
Processing module handles the human eye picture to obtain the human eye generation figure in thick stage for distorting network by coarse adjustment
Picture；
Correction module generates the defects of image region for detecting the human eye by fine corrective networks, and lacks to described
Sunken region is modified.
7. the human eye sight correction system according to claim 6 based on deep learning, which is characterized in that further include: it is raw
At module,
The generation module, for training one thick to generate a two-dimensional compensation figure to the convolutional neural networks of fine texture
Spectrum, the substitution operation that the compensation map carries out Pixel-level to original image are generated training image, are schemed using generation training
Mean square error between picture and the original image trains the coarse adjustment distortion network as loss function.
8. the human eye sight correction system according to claim 6 based on deep learning, which is characterized in that described finely to repair
Positive network, comprising: the corrective networks of cyclic policy network and part；
The correction module, comprising: detection unit and amending unit；
The detection unit detects the human eye for the cyclic policy network and generates the defects of image region；
The amending unit is modified the defect area by convolutional layer for the local corrective networks.
9. the human eye sight correction system according to claim 8 based on deep learning, which is characterized in that
The detection unit, is specifically used for:
The image I of given t stept-1, the cyclic policy network by selection one part block coordinate position ltTo select
One block
lt=fr(st-1)
Wherein, st-1It is the state feature by coding of the cyclic policy network, by input picture It-1History after coding
Hidden state ht-1The position l for the block chosen with previous stept-1Common building, g indicates cut operation, by image It-1In to positioning
Set ltBlock cut down as a result.
10. the human eye sight correction system according to claim 9 based on deep learning, which is characterized in that
The amending unit, is specifically used for:
To be modified piece of each step selectionWith the local correction network feIt is modified, obtains one by modified
Block, then use the revised piece of block directly replaced before amendment as the amendment image I of this stept, after T step, obtained final
Image IT。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910175164.0A CN109978804B (en) | 2019-03-08 | 2019-03-08 | Human eye sight line correction method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910175164.0A CN109978804B (en) | 2019-03-08 | 2019-03-08 | Human eye sight line correction method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109978804A true CN109978804A (en) | 2019-07-05 |
| CN109978804B CN109978804B (en) | 2021-02-26 |
Family
ID=67078291
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910175164.0A Active CN109978804B (en) | 2019-03-08 | 2019-03-08 | Human eye sight line correction method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109978804B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111008929A (en) * | 2019-12-19 | 2020-04-14 | 维沃移动通信（杭州）有限公司 | Image correction method and electronic equipment |
| CN111339928A (en) * | 2020-02-25 | 2020-06-26 | 苏州科达科技股份有限公司 | Eye spirit adjusting method and device and storage medium |
| CN113343931A (en) * | 2021-07-05 | 2021-09-03 | Oppo广东移动通信有限公司 | Training method for generating countermeasure network, image sight correction method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110090303A1 (en) * | 2009-10-16 | 2011-04-21 | Apple Inc. | Facial Pose Improvement with Perspective Distortion Correction |
| CN107944415A (en) * | 2017-12-06 | 2018-04-20 | 董伟 | A kind of human eye notice detection method based on deep learning algorithm |
| CN108022213A (en) * | 2017-11-29 | 2018-05-11 | 天津大学 | Video super-resolution algorithm for reconstructing based on generation confrontation network |
| CN108492273A (en) * | 2018-03-28 | 2018-09-04 | 深圳市唯特视科技有限公司 | A kind of image generating method based on from attention model |
| CN108765340A (en) * | 2018-05-29 | 2018-11-06 | Oppo（重庆）智能科技有限公司 | Fuzzy image processing method, apparatus and terminal device |
| CN108885784A (en) * | 2016-04-22 | 2018-11-23 | 英特尔公司 | It is corrected using the real-time eye contact of machine learning neural network based |
| CN109102532A (en) * | 2017-06-20 | 2018-12-28 | 西门子保健有限责任公司 | The metaplasia of deep learning for medical imaging |
-
2019
- 2019-03-08 CN CN201910175164.0A patent/CN109978804B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110090303A1 (en) * | 2009-10-16 | 2011-04-21 | Apple Inc. | Facial Pose Improvement with Perspective Distortion Correction |
| CN108885784A (en) * | 2016-04-22 | 2018-11-23 | 英特尔公司 | It is corrected using the real-time eye contact of machine learning neural network based |
| CN109102532A (en) * | 2017-06-20 | 2018-12-28 | 西门子保健有限责任公司 | The metaplasia of deep learning for medical imaging |
| CN108022213A (en) * | 2017-11-29 | 2018-05-11 | 天津大学 | Video super-resolution algorithm for reconstructing based on generation confrontation network |
| CN107944415A (en) * | 2017-12-06 | 2018-04-20 | 董伟 | A kind of human eye notice detection method based on deep learning algorithm |
| CN108492273A (en) * | 2018-03-28 | 2018-09-04 | 深圳市唯特视科技有限公司 | A kind of image generating method based on from attention model |
| CN108765340A (en) * | 2018-05-29 | 2018-11-06 | Oppo（重庆）智能科技有限公司 | Fuzzy image processing method, apparatus and terminal device |
Non-Patent Citations (3)
| Title |
|---|
| RUIGANG YANG等: "《Eye Gaze Correction with Stereovision for Video-Teleconferencing》", 《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》 * |
| 孙兴华等: "《基于Hough变换和梯度信息的人眼视线方向估计》", 《小型微型计算机系统》 * |
| 尹苓琳等: "《虚拟视角自适应的视线校正方法》", 《计算机辅助设计与图形学学报》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111008929A (en) * | 2019-12-19 | 2020-04-14 | 维沃移动通信（杭州）有限公司 | Image correction method and electronic equipment |
| CN111008929B (en) * | 2019-12-19 | 2023-09-26 | 维沃移动通信（杭州）有限公司 | Image correction method and electronic equipment |
| CN111339928A (en) * | 2020-02-25 | 2020-06-26 | 苏州科达科技股份有限公司 | Eye spirit adjusting method and device and storage medium |
| WO2021169325A1 (en) * | 2020-02-25 | 2021-09-02 | 苏州科达科技股份有限公司 | Gaze adjustment method and apparatus, and storage medium |
| CN113343931A (en) * | 2021-07-05 | 2021-09-03 | Oppo广东移动通信有限公司 | Training method for generating countermeasure network, image sight correction method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109978804B (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhao et al. | RefineDNet: A weakly supervised refinement framework for single image dehazing | |
| CN109978804A (en) | Human eye sight antidote and system based on deep learning | |
| CN104408701B (en) | A kind of large scene video image joining method | |
| CN111292264B (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN111080511B (en) | End-to-end face exchange method for high-resolution multi-feature extraction | |
| Asif et al. | Shape from focus using multilayer feedforward neural networks | |
| CN108537743A (en) | A kind of face-image Enhancement Method based on generation confrontation network | |
| CN108022213A (en) | Video super-resolution algorithm for reconstructing based on generation confrontation network | |
| Li et al. | Dual-scale single image dehazing via neural augmentation | |
| CN103955891B (en) | Image restoration method based on block matching | |
| CN111754928B (en) | Display screen layered correction method and device | |
| CN103945207B (en) | A kind of stereo-picture vertical parallax removing method based on View Synthesis | |
| CN108537747A (en) | A kind of image repair method based on the convolutional neural networks with symmetrical parallel link | |
| CN111062899B (en) | Guidance-based blink video generation method for generating confrontation network | |
| CN116542932A (en) | Injection molding surface defect detection method based on improved YOLOv5s | |
| CN103093426A (en) | Method recovering texture and illumination of calibration plate sheltered area | |
| Swami et al. | Candy: Conditional adversarial networks based end-to-end system for single image haze removal | |
| CN104284090B (en) | The method that varifocal lighting apparatus links with Varifocal zoom lens video camera | |
| Li et al. | Line drawing guided progressive inpainting of mural damages | |
| Hovhannisyan et al. | AED-Net: A single image dehazing | |
| CN110569881B (en) | Data augmentation method and device based on convolutional neural network | |
| Goncalves et al. | Guidednet: Single image dehazing using an end-to-end convolutional neural network | |
| US20210374916A1 (en) | Storage medium storing program, image processing apparatus, and training method of machine learning model | |
| US20210398293A1 (en) | Motion vector generation apparatus, projection image generation apparatus, motion vector generation method, and program | |
| CN113724168B (en) | Airplane image defogging method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |