CN106530227B - Image recovery method and device - Google Patents

Image recovery method and device Download PDF

Info

Publication number
CN106530227B
CN106530227B CN201610958087.2A CN201610958087A CN106530227B CN 106530227 B CN106530227 B CN 106530227B CN 201610958087 A CN201610958087 A CN 201610958087A CN 106530227 B CN106530227 B CN 106530227B
Authority
CN
China
Prior art keywords
module
mosaic
image
picture material
resolution ratio
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610958087.2A
Other languages
Chinese (zh)
Other versions
CN106530227A (en
Inventor
陈志军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Xiaomi Mobile Software Co Ltd
Original Assignee
Beijing Xiaomi Mobile Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Xiaomi Mobile Software Co Ltd filed Critical Beijing Xiaomi Mobile Software Co Ltd
Priority to CN201610958087.2A priority Critical patent/CN106530227B/en
Publication of CN106530227A publication Critical patent/CN106530227A/en
Application granted granted Critical
Publication of CN106530227B publication Critical patent/CN106530227B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4015Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The disclosure is directed to a kind of image recovery method and devices.The described method includes: determining the mosaic area on video image;The first picture material of the mosaic area is input in the convolutional neural networks trained, process of convolution is carried out to the first image content by the positive convolution module of the convolutional neural networks, the fisrt feature for obtaining the first image content indicates;The fisrt feature is indicated by the warp volume module in the convolutional neural networks to carry out deconvolution processing, the mosaic area is obtained by the second picture material before mosaic, the resolution ratio of second picture material is identical as the resolution ratio of the first image content;Second picture material is shown in the mosaic area.Disclosed technique scheme can make to can show that the original contents on video image on video image, it is ensured that no longer promote video quality since the presence of mosaic area influences the visual experience of user on video image.

Description

Image recovery method and device
Technical field
This disclosure relates to technical field of image processing more particularly to a kind of image recovery method and device.
Background technique
When the electricity that user watches some TV stations by video website or video application or video platform provides When shadow or TV play, it will usually stamp mosaic in the upper left corner of entire video pictures or the logo in the upper right corner, thus lead It causes the image information in video pictures to be blurred, reduces video quality.
Summary of the invention
To overcome the problems in correlation technique, the embodiment of the present disclosure provides a kind of image recovery method and device, uses To ensure to promote video quality no longer since the presence of mosaic area influences the visual experience of user on video image.
According to the first aspect of the embodiments of the present disclosure, a kind of image recovery method is provided, comprising:
Determine the mosaic area on video image;
The first picture material of the mosaic area is input in the convolutional neural networks trained, the volume is passed through The positive convolution module of product neural network carries out process of convolution to the first image content, obtains the of the first image content One character representation;
The fisrt feature is indicated by the warp volume module in the convolutional neural networks to carry out deconvolution processing, is obtained To the mosaic area by the second picture material before mosaic, the resolution ratio of second picture material and first figure As the resolution ratio of content is identical;
Second picture material is shown in the mosaic area.
In one embodiment, the method also includes:
By setting the mosaic sample of quantity to the positive convolution module and the corresponding spy of warp volume module Sign parameter is trained, and obtains the second of the corresponding fisrt feature parameter sets of the positive convolution module and the warp volume module Set of characteristic parameters;
Based on the fisrt feature parameter sets that training obtains, by the positive convolution module to the setting quantity Mosaic sample carries out process of convolution, obtains second feature expression;
Based on the second feature parameter sets that training obtains, by the warp volume module to the second feature table Show and carry out deconvolution processing, obtains the reconstructed image of the mosaic sample of the setting quantity;
Determine the reconstructed image of the setting quantity and the corresponding original graph of mosaic sample of the setting quantity Reset error as between;
When the reset error reaches the condition of convergence, control the convolutional neural networks deconditioning, obtain it is described Trained convolutional neural networks.
In one embodiment, the output dimension of the warp volume module is identical as the input dimension of the positive convolution module.
In one embodiment, the mosaic area on the determining video image, comprising:
Video image is input to the disaggregated model trained, the disaggregated model is led to by the mosaic sample of setting quantity It crosses and default sorting algorithm is trained to obtain;
The mosaic area on the video image is detected based on the disaggregated model trained.
In one embodiment, the method also includes:
Determine the resolution ratio of the first picture material of mosaic area and the positive convolution module input dimension whether phase Together;
If the resolution ratio of the first image content and the input dimension be not identical, according to the first image content Resolution ratio and the input dimension, zoom in and out processing to the first image content;
It is described to show second picture material before the mosaic area, the method also includes:
According to the resolution ratio of the first image content and the input dimension, by second picture material zoom to The resolution ratio of the first image content not being scaled before processing.
According to the second aspect of an embodiment of the present disclosure, a kind of image restoration device is provided, comprising:
First determining module, the mosaic area being configured to determine that on video image;
First processing module is configured as the first image for the mosaic area for determining first determining module Content is input in the convolutional neural networks trained, by the positive convolution module of the convolutional neural networks to first figure As content progress process of convolution, the fisrt feature for obtaining the first image content is indicated;
Second processing module is configured as through the warp volume module in the convolutional neural networks to first processing The fisrt feature that module obtains indicates progress deconvolution processing, obtains the mosaic area by the second figure before mosaic As content, the resolution ratio of second picture material is identical as the resolution ratio of the first image content;
Display module, second picture material for being configured as obtaining the Second processing module are shown in the horse Match gram region.
In one embodiment, described device may also include that
Training module is configured as the mosaic sample by setting quantity to the positive convolution module and the deconvolution The corresponding characteristic parameter of module is trained, and obtains corresponding fisrt feature parameter sets of the positive convolution module and described The second feature parameter sets of warp volume module;
Third processing module is configured as the fisrt feature parameter sets obtained based on training module training, Process of convolution is carried out by mosaic sample of the positive convolution module to the setting quantity, obtains second feature expression;
Fourth processing module is configured as the second feature parameter sets obtained based on training, passes through the warp Volume module indicates progress deconvolution processing to the second feature that the third processing module obtains, and obtains the setting quantity Mosaic sample reconstructed image;
Second determining module is configured to determine that the reconstructed image for the setting quantity that the fourth processing module obtains Reset error between the corresponding original image of mosaic sample of the setting quantity;
Control module is configured as when the reset error that second determining module determines reaches the condition of convergence, The convolutional neural networks deconditioning is controlled, the convolutional neural networks trained are obtained.
In one embodiment, the output dimension of the warp volume module is identical as the input dimension of the positive convolution module.
In one embodiment, first determining module can include:
Input submodule is configured as video image being input to the disaggregated model trained, and the disaggregated model is by setting The mosaic sample of fixed number amount to default sorting algorithm by being trained to obtain;
Detection sub-module is configured as detecting the institute of the input submodule input based on the disaggregated model trained State the mosaic area on video image.
In one embodiment, described device may also include that
Third determining module is configured to determine that the resolution ratio and the positive convolution of the first picture material of mosaic area Whether the input dimension of module is identical;
Zoom module, if be configured as the third determining module determine the resolution ratio of the first image content with it is described It is not identical to input dimension, according to the resolution ratio of the first image content and the input dimension, to the first image content Zoom in and out processing;
The Zoom module is additionally configured to resolution ratio and the input dimension according to the first image content, by institute State the second picture material zoom to the first image content be not scaled processing before resolution ratio.
According to the third aspect of an embodiment of the present disclosure, a kind of image restoration device is provided, comprising:
Processor;
Memory for storage processor executable instruction;
Wherein, the processor is configured to:
Determine the mosaic area on video image;
The first picture material of the mosaic area is input in the convolutional neural networks trained, the volume is passed through The positive convolution module of product neural network carries out process of convolution to the first image content, obtains the of the first image content One character representation;
The fisrt feature is indicated by the warp volume module in the convolutional neural networks to carry out deconvolution processing, is obtained To the mosaic area by the second picture material before mosaic, the resolution ratio of second picture material and first figure As the resolution ratio of content is identical;
Second picture material is shown in the mosaic area.
The technical scheme provided by this disclosed embodiment can include the following benefits:
Process of convolution is carried out to the first picture material of mosaic area by the positive convolution module of CNN, passes through the anti-of CNN Convolution module carries out deconvolution processing to the character representation of the first picture material after convolution, obtains mosaic area by mosaic The second preceding picture material shows the second picture material in mosaic area, so as to so as to can show on video image Original contents on video image out, it is ensured that no longer since the presence of mosaic area influences the vision body of user on video image It tests, improves video quality.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not The disclosure can be limited.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows and meets implementation of the invention Example, and be used to explain the principle of the present invention together with specification.
Figure 1A is the flow chart of image recovery method shown according to an exemplary embodiment.
Figure 1B is the frame diagram of the convolutional neural networks shown in Figure 1A illustrated embodiment.
Fig. 1 C is the schematic diagram for not breaking mosaic on the video image shown in Figure 1A illustrated embodiment.
Fig. 1 D is the schematic diagram of the mosaic on the video image shown in Figure 1A illustrated embodiment.
Fig. 1 E is the schematic diagram after restoring shown in Figure 1A illustrated embodiment to video image.
Fig. 2A is the flow chart of training convolutional neural networks shown according to an exemplary embodiment.
Fig. 2 B used frame diagram when being training convolutional neural networks shown according to an exemplary embodiment.
Fig. 3 is the flow chart of the image recovery method shown according to another exemplary embodiment.
Fig. 4 is a kind of block diagram of image restoration device shown according to an exemplary embodiment.
Fig. 5 is the block diagram of another image restoration device shown according to an exemplary embodiment.
Fig. 6 is a kind of block diagram suitable for image restoration device shown according to an exemplary embodiment.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment Described in embodiment do not represent all embodiments consistented with the present invention.On the contrary, they be only with it is such as appended The example of device and method being described in detail in claims, some aspects of the invention are consistent.
Figure 1A is the flow chart of image recovery method shown according to an exemplary embodiment, and Figure 1B is implemented shown in Figure 1A The architecture diagram of the convolutional neural networks exemplified, Fig. 1 C are not break mosaic on the video image shown in Figure 1A illustrated embodiment Schematic diagram, Fig. 1 D is the schematic diagram of the mosaic on the video image shown in Figure 1A illustrated embodiment, and Fig. 1 E is shown in Figure 1A Implement exemplify video image is restored after schematic diagram;The image recovery method can be applied in electronic equipment (such as: smart phone, tablet computer) on, it can be realized by way of Video Applications are installed on an electronic device, such as Figure 1A Shown, which includes the following steps S101-S104:
In step s101, the mosaic area on video image is determined.
In one embodiment, can be by carrying out rough detection to the region where the logo on video image, positioning is gone into action Approximate region where match gram, for example, mosaic would generally detect substantially on the upper left side or upper right side of screen image Behind region, further according to features such as contrast, the clarity of mosaic itself, mosaic area is identified.In another embodiment, The method that pattern-recognition can be used, identifies mosaic area by the disaggregated model trained.
In step s 102, the first picture material of mosaic area is input in the convolutional neural networks trained, Process of convolution is carried out to the first picture material by the positive convolution module of convolutional neural networks, obtains the first of the first picture material Character representation.
In one embodiment, as shown in Figure 1B, the first picture material is output to the positive convolution module 11 in CNN10, passes through The process of convolution of positive convolution module 11, obtains the character representation of the first picture material.It will be appreciated by persons skilled in the art that Positive convolution module 11 may include multiple convolutional layers and multiple sub-sampling layers, each convolutional layer and each sub-sampling in CNN The parameter of layer is what CNN10 was trained to, and the fisrt feature expression obtained by positive 11 process of convolution of convolution module can Accurate to indicate the first picture material, the specific structure about multiple convolutional layers and multiple sub-sampling layers in CNN can be found in phase The description of pass technology, it is not described here in detail for the disclosure.
In step s 103, fisrt feature is indicated to carry out at deconvolution by the warp volume module in convolutional neural networks Reason obtains mosaic area by the second picture material before mosaic, the resolution ratio of the second picture material and the first picture material Resolution ratio it is identical.
In one embodiment, as shown in Figure 1B, the character representation of the first picture material can be input to warp volume module 12, it is handled by the deconvolution of warp volume module 12, restores mosaic area by the second picture material before mosaic.This field Technical staff is it is understood that warp volume module 12 may include multiple warp laminations and multiple up-sampling layers, up-sampling layer Character representation for handling deconvolution carries out up-sampling reconstruction, to ensure the second picture material of the output of warp volume module 12 Resolution ratio it is identical as the resolution ratio of the first picture material.
In step S104, the second picture material is shown in mosaic area.
In an exemplary scene, video image shown in Fig. 1 C has platform logo " 56 I find pleasure in ", is located at entire video The upper right side of image.In video display process, the logo in the video image of each frame is labeled with mosaic, if mosaic area Domain has important picture material, influences whether that the experience of user's viewing video image can be by horse by the embodiment of the present disclosure The picture material for not including have logo out is restored in match gram region, so as to ensure the integrality of video image.As showing , the resolution ratio of the video image in Fig. 1 C is 1400*1000, the mosaic area in Fig. 1 C identified by step S101 Size be 200*200, at this time the first picture material of the 200*200 of mosaic area can be input to and to have been trained CNN10 carries out process of convolution to the first picture material by the positive convolution module 11 in CNN10, obtains the first picture material Fisrt feature indicates, fisrt feature is indicated to carry out deconvolution by warp volume module 12, exports the second picture material, the second figure As the resolution ratio of content is 200*200, by show the second picture material in mosaic area, horse of the realization to video image The image restoration in gram region is matched, so as to allow users to watch the normal picture of not mosaic, promotes screen quality.
In the present embodiment, process of convolution is carried out to the first picture material of mosaic area by the positive convolution module of CNN, The fisrt feature of the first picture material after convolution is indicated to carry out deconvolution processing by the warp volume module of CNN, obtains horse Gram region is matched by the second picture material before mosaic, the second picture material is shown in mosaic area, so as to so that view It can show that the original contents on video image on frequency image, it is ensured that no longer due to the presence of mosaic area on video image The visual experience for influencing user, improves video quality.
In one embodiment, method further include:
By set the mosaic sample of quantity to positive convolution module and the corresponding characteristic parameter of warp volume module into Row training, obtains the second feature parameter sets of positive convolution module corresponding fisrt feature parameter sets and warp volume module;
Based on the obtained fisrt feature parameter sets of training, by positive convolution module to the mosaic sample of setting quantity into Row process of convolution obtains second feature expression;
Based on the second feature parameter sets that training obtains, second feature is indicated by warp volume module to carry out deconvolution Processing obtains the reconstructed image of the mosaic sample of setting quantity;
It determines between the reconstructed image of setting quantity and the corresponding original image of mosaic sample of setting quantity Reset error;
When reset error reaches the condition of convergence, convolutional neural networks deconditioning is controlled, the convolution mind trained Through network.
In one embodiment, the output dimension of warp volume module and the input dimension of positive convolution module are identical.
In one embodiment, the mosaic area on video image is determined, comprising:
Video image is input to the disaggregated model trained, disaggregated model by setting quantity mosaic sample by pair Default sorting algorithm is trained to obtain;
Based on the mosaic area on the disaggregated model detection video image trained.
In one embodiment, method further include:
Whether the resolution ratio for determining the first picture material of mosaic area is identical as the input dimension of positive convolution module;
If the resolution ratio of the first picture material and input dimension be not identical, according to the resolution ratio and input of the first picture material Dimension zooms in and out processing to the first picture material.
So far, the above method that the embodiment of the present disclosure provides, can make to can show that on video image on video image Original contents, it is ensured that on video image no longer due to mosaic area presence influence user visual experience, promoted video Quality.
The technical solution of embodiment of the present disclosure offer is provided below with specific embodiment.
Fig. 2A is the flow chart of training convolutional neural networks shown according to an exemplary embodiment, and Fig. 2 B is shown according to one Example property implements used frame diagram when the training convolutional neural networks exemplified;The present embodiment is provided using the embodiment of the present disclosure The above method as shown in Figure 2 A, included the following steps: by how to illustrate for training convolutional neural networks
In step s 201, positive convolution module and warp volume module are respectively corresponded to by setting the mosaic sample of quantity Characteristic parameter be trained, obtain the second feature of positive convolution module corresponding fisrt feature parameter sets and warp volume module Parameter sets.
In one embodiment, the figure which is obtained after mosaic for the logo of TV station or the network platform Piece sample.In one embodiment, batch training can be carried out to CNN, participate in the setting quantity of the mosaic sample of training every time Specific number with no restrictions, for example, setting quantity as 128,64, etc..In one embodiment, in fisrt feature parameter sets Each convolutional layer for being positive in convolution module 11 of characteristic parameter and sub-sampling layer be in weight parameter, second feature parameter set Characteristic parameter in conjunction is the weight parameter in each warp lamination and up-sampling layer in warp volume module 12, fisrt feature ginseng Manifold is closed can be according to each convolutional layer and sub-sampling layer, each warp with the quantity of parameter included in second feature parameter sets The quantity of lamination and the neuron in up-sampling layer determines, the disclosure to the particular number of parameter with no restrictions.
In step S202, based on the fisrt feature parameter sets that training obtains, by positive convolution module to setting quantity Mosaic sample carry out process of convolution, obtain second feature expression.
In step S203, based on the second feature parameter sets that training obtains, by warp volume module to second feature It indicates to carry out deconvolution processing, obtains the reconstructed image of the mosaic sample of setting quantity.
Positive convolution module carries out process of convolution and warp volume module to second feature to the mosaic sample of setting quantity Indicate that the description for carrying out deconvolution processing may refer to the description of above-mentioned Figure 1A illustrated embodiment, this will not be detailed here.
In step S204, the reconstructed image of setting quantity and the corresponding original of mosaic sample of setting quantity are determined Reset error between beginning image executes step S205 when reset error reaches the condition of convergence, when reset error is not up to received When holding back condition, executes step S201 and CNN is continued to train.
In one embodiment, the recovery between the pre- original image of reconstructed image can be calculated by the loss function of CNN10 The specific formula of error, the loss function of CNN10 may refer to description of related art, and this will not be detailed here.
In step S205, when reset error reaches the condition of convergence, convolutional neural networks deconditioning is controlled, is obtained Trained convolutional neural networks.
In an exemplary scene, illustrated by setting quantity as 128,128 mosaic samples pair The original image for answering 128 logos not comprising TV station or the network platform, is input to when by 128 mosaic samples After CNN10,128 mosaic samples can to positive convolution module 11 and the corresponding characteristic parameter of warp volume module 12 into Row training, obtains the second feature parameter set of positive convolution module 11 corresponding fisrt feature parameter sets and warp volume module 12 It closes.Later, pass through the fisrt feature parameter sets of positive convolution module 11 and the second feature parameter sets pair of warp volume module 12 128 mosaic samples carry out convolution and deconvolution, obtain 128 corresponding restored images of mosaic sample, lead to It crosses reset error computing module 13 and calculates 128 corresponding restored images of mosaic sample and corresponding original image Reset error controls CNN10 deconditioning, the convolutional neural networks trained when reset error reaches the condition of convergence.
In the present embodiment, by being trained to CNN, CNN is made to be provided with the function of positive convolution and reversed convolution, when When reset error reaches the condition of convergence, the CNN accurate recovery after can enabling training goes out the original image of mosaic area, really Protect the integrality of video image.
Fig. 3 is the flow chart of the image recovery method shown according to another exemplary embodiment;The present embodiment utilizes this public affairs The above method that embodiment offer is provided, how to determine the mosaic area on video image and to the first figure of mosaic area It is illustrated for being pre-processed as content, as shown in figure 3, including the following steps:
In step S301, video image is input to the disaggregated model trained, disaggregated model by setting quantity horse Gram sample is matched by being trained to obtain to default sorting algorithm.
In one embodiment, default sorting algorithm can be adaboost, support vector machines (Support Vector Machine, referred to as SVM), CNN etc., default sorting algorithm is instructed by mosaic sample and non-mosaic sample Practice, obtains the disaggregated model for detecting mosaic area.
In step s 302, based on the mosaic area on the disaggregated model detection video image trained.
In one embodiment, mosaic area may include mosaic area center and mosaic area The resolution ratio of one picture material.
In step S303, the resolution ratio of the first picture material of mosaic area and the input of convolutional neural networks are determined Whether dimension is identical.
In step s 304, if the resolution ratio of the first picture material and input dimension be not identical, according to the first picture material Resolution ratio and input dimension, processing is zoomed in and out to the first picture material.
In step S305, after recovering to the second picture material, tieed up according to the resolution ratio of the first picture material and input Second picture material is zoomed to the resolution ratio not being scaled before handling with the first picture material by degree.
For example, S302 determines that the resolution ratio of the first picture material is 200*200 through the above steps, the input of CNN10 is tieed up Degree is 100*100, then the first picture material can be carried out to 1/4 diminution processing, so as to by real shown in above-mentioned Figure 1A The first picture material is restored in the description for applying example, is restoring the second picture material out by above-mentioned Figure 1A illustrated embodiment Afterwards, the resolution ratio of the second picture material is 100*100, it is therefore desirable to be amplified the progress of the second picture material for 2*2 times, thus will Second picture material of 100*100 size is amplified to the resolution ratio not being scaled before handling with the first picture material, i.e., 200*200。
In the present embodiment, the mosaic area in video image is identified by the disaggregated model trained, it can be with Improve the efficiency of mosaic area's detection;By zooming in and out processing to mosaic area, it can be ensured that the Marseille of arbitrary size Gram region can carry out image restoration by CNN.
Fig. 4 is a kind of block diagram of image restoration device shown according to an exemplary embodiment, as shown in figure 4, image is multiple Original device includes:
First determining module 41, the mosaic area being configured to determine that on video image;
First processing module 42 is configured as the first picture material for the mosaic area for determining the first determining module 41 It is input in the convolutional neural networks trained, the first picture material is rolled up by the positive convolution module of convolutional neural networks Product processing, the fisrt feature for obtaining the first picture material indicate;
Second processing module 43 is configured as through the warp volume module 42 in convolutional neural networks to first processing module Obtained fisrt feature indicates to carry out deconvolution processing, obtains mosaic area by the second picture material before mosaic, second The resolution ratio of picture material is identical as the resolution ratio of the first picture material;
Display module 44, the second picture material for being configured as obtaining Second processing module 43 are shown in mosaic area Domain.
Fig. 5 is the block diagram of another image restoration device shown according to an exemplary embodiment, as shown in figure 5, upper On the basis of stating embodiment illustrated in fig. 4, in one embodiment, device may also include that
Training module 45 is configured as each to positive convolution module and warp volume module by the mosaic sample for setting quantity Self-corresponding characteristic parameter is trained, and obtains the of the corresponding fisrt feature parameter sets of positive convolution module and warp volume module Two set of characteristic parameters;
Third processing module 46 is configured as the fisrt feature parameter sets obtained based on the training of training module 45, passed through Positive convolution module carries out process of convolution to the mosaic sample of setting quantity, obtains second feature expression;
Fourth processing module 47 is configured as the second feature parameter sets obtained based on training, passes through warp volume module The second feature that third processing module 46 obtains is indicated to carry out deconvolution processing, obtains the weight of the mosaic sample of setting quantity Composition picture;
Second determining module 48, is configured to determine that the reconstructed image of the setting quantity that fourth processing module 47 obtains and sets Reset error between the corresponding original image of mosaic sample of fixed number amount;
Control module 49 is configured as when the reset error that the second determining module determines reaches the condition of convergence, control volume Product neural network deconditioning, the convolutional neural networks trained.
In one embodiment, the output dimension of warp volume module and the input dimension of positive convolution module are identical.
In one embodiment, the first determining module 41 can include:
Input submodule 411 is configured as video image being input to the disaggregated model trained, and disaggregated model is by setting The mosaic sample of quantity to default sorting algorithm by being trained to obtain;
Detection sub-module 412 is configured as the video inputted based on the disaggregated model detection input submodule 411 trained Mosaic area on image.
In one embodiment, device may also include that
Third determining module 50 is configured to determine that in the first image for the mosaic area that the first determining module 41 determines Whether the resolution ratio of appearance is identical as the input dimension of positive convolution module;
Zoom module 51, if being configured as resolution ratio and input dimension that third determining module 50 determines the first picture material It is not identical, according to the resolution ratio of the first picture material and input dimension, processing is zoomed in and out to the first picture material.
Zoom module 51 is additionally configured to show the second picture material before mosaic area in display module 44, root According to the resolution ratio and input dimension of the first picture material, the second picture material is zoomed to and is not scaled with the first picture material Resolution ratio before processing.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method Embodiment in be described in detail, no detailed explanation will be given here.
Fig. 6 is a kind of block diagram suitable for image restoration device shown according to an exemplary embodiment.For example, device 600 can be mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, and medical treatment is set It is standby, body-building equipment, personal digital assistant etc..
Referring to Fig. 6, device 600 may include following one or more components: processing component 602, memory 604, power supply Component 606, multimedia component 608, audio component 610, the interface 612 of input/output (I/O), sensor module 614, and Communication component 616.
The integrated operation of the usual control device 600 of processing component 602, such as with display, telephone call, data communication, phase Machine operation and record operate associated operation.Processing element 602 may include that one or more processors 620 refer to execute It enables, to perform all or part of the steps of the methods described above.In addition, processing component 602 may include one or more modules, just Interaction between processing component 602 and other assemblies.For example, processing component 602 may include multi-media module, it is more to facilitate Interaction between media component 608 and processing component 602.
Memory 604 is configured as storing various types of data to support the operation in equipment 600.These data are shown Example includes the instruction of any application or method for operating on device 600, contact data, and telephone book data disappears Breath, picture, video etc..Memory 604 can be by any kind of volatibility or non-volatile memory device or their group It closes and realizes, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM) is erasable to compile Journey read-only memory (EPROM), programmable read only memory (PROM), read-only memory (ROM), magnetic memory, flash Device, disk or CD.
Electric power assembly 606 provides electric power for the various assemblies of device 600.Electric power assembly 606 may include power management system System, one or more power supplys and other with for device 600 generate, manage, and distribute the associated component of electric power.
Multimedia component 608 includes the screen of one output interface of offer between described device 600 and user.One In a little embodiments, screen may include liquid crystal display (LCD) and touch panel (TP).If screen includes touch panel, screen Curtain may be implemented as touch screen, to receive input signal from the user.Touch panel includes one or more touch sensings Device is to sense the gesture on touch, slide, and touch panel.The touch sensor can not only sense touch or sliding action Boundary, but also detect duration and pressure associated with the touch or slide operation.In some embodiments, more matchmakers Body component 608 includes a front camera and/or rear camera.When equipment 600 is in operation mode, such as screening-mode or When video mode, front camera and/or rear camera can receive external multi-medium data.Each front camera and Rear camera can be a fixed optical lens system or have focusing and optical zoom capabilities.
Audio component 610 is configured as output and/or input audio signal.For example, audio component 610 includes a Mike Wind (MIC), when device 600 is in operation mode, when such as call mode, recording mode, and voice recognition mode, microphone is matched It is set to reception external audio signal.The received audio signal can be further stored in memory 604 or via communication set Part 616 is sent.In some embodiments, audio component 610 further includes a loudspeaker, is used for output audio signal.
I/O interface 612 provides interface between processing component 602 and peripheral interface module, and above-mentioned peripheral interface module can To be keyboard, click wheel, button etc..These buttons may include, but are not limited to: home button, volume button, start button and lock Determine button.
Sensor module 614 includes one or more sensors, and the state for providing various aspects for device 600 is commented Estimate.For example, sensor module 614 can detecte the state that opens/closes of equipment 600, and the relative positioning of component, for example, it is described Component is the display and keypad of device 600, and sensor module 614 can be with 600 1 components of detection device 600 or device Position change, the existence or non-existence that user contacts with device 600,600 orientation of device or acceleration/deceleration and device 600 Temperature change.Sensor module 614 may include proximity sensor, be configured to detect without any physical contact Presence of nearby objects.Sensor module 614 can also include optical sensor, such as CMOS or ccd image sensor, at As being used in application.In some embodiments, which can also include acceleration transducer, gyro sensors Device, Magnetic Sensor, pressure sensor or temperature sensor.
Communication component 616 is configured to facilitate the communication of wired or wireless way between device 600 and other equipment.Device 600 can access the wireless network based on communication standard, such as WiFi, 2G or 3G or their combination.In an exemplary implementation In example, communication component 616 receives broadcast singal or broadcast related information from external broadcasting management system via broadcast channel. In one exemplary embodiment, the communication component 616 further includes near-field communication (NFC) module, to promote short range communication.Example Such as, NFC module can be based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies are realized.
In the exemplary embodiment, device 600 can be believed by one or more application specific integrated circuit (ASIC), number Number processor (DSP), digital signal processing appts (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic components are realized, for executing the above method.
In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instruction, example are additionally provided It such as include the memory 604 of instruction, above-metioned instruction can be executed by the processor 620 of device 600 to complete the above method.For example, The non-transitorycomputer readable storage medium can be ROM, random access memory (RAM), CD-ROM, tape, floppy disk With optical data storage devices etc..
Those skilled in the art will readily occur to its of the disclosure after considering specification and practicing disclosure disclosed herein Its embodiment.This application is intended to cover any variations, uses, or adaptations of the disclosure, these modifications, purposes or Person's adaptive change follows the general principles of this disclosure and including the undocumented common knowledge in the art of the disclosure Or conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the disclosure are by following Claim is pointed out.
It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and And various modifications and changes may be made without departing from the scope thereof.The scope of the present disclosure is only limited by the accompanying claims.

Claims (11)

1. a kind of image recovery method, which is characterized in that the described method includes:
Determine the mosaic area on video image;
The first picture material of the mosaic area is input in the convolutional neural networks trained, the convolution mind is passed through Positive convolution module through network carries out process of convolution to the first image content, and obtain the first image content first is special Sign indicates;
The fisrt feature is indicated by the warp volume module in the convolutional neural networks to carry out deconvolution processing, obtains institute Mosaic area is stated by the second picture material before mosaic, in the resolution ratio and the first image of second picture material The resolution ratio of appearance is identical;
Second picture material is shown in the mosaic area.
2. the method according to claim 1, wherein the method also includes:
Mosaic sample by setting quantity joins the positive convolution module and the corresponding feature of the warp volume module Number is trained, and obtains the second feature of the positive convolution module corresponding fisrt feature parameter sets and the warp volume module Parameter sets;
Based on the fisrt feature parameter sets that training obtains, by the positive convolution module to the Marseille of the setting quantity Gram sample carries out process of convolution, obtains second feature expression;
Based on the obtained second feature parameter sets of training, by the warp volume module to the second feature indicate into Row deconvolution processing obtains the reconstructed image of the mosaic sample of the setting quantity;
Determine it is described setting quantity reconstructed image and it is described setting quantity the corresponding original image of mosaic sample it Between reset error;
When the reset error reaches the condition of convergence, the convolutional neural networks deconditioning is controlled, obtains described trained Convolutional neural networks.
3. according to the method described in claim 2, it is characterized in that, the output dimension of the warp volume module and the positive convolution The input dimension of module is identical.
4. the method according to claim 1, wherein the mosaic area on the determining video image, comprising:
Video image is input to the disaggregated model trained, the disaggregated model by setting quantity mosaic sample by pair Default sorting algorithm is trained to obtain;
The mosaic area on the video image is detected based on the disaggregated model trained.
5. the method according to claim 1, wherein the method also includes:
Determine whether the resolution ratio of the first picture material of mosaic area is identical as the input dimension of the positive convolution module;
If the resolution ratio of the first image content and the input dimension be not identical, according to the resolution of the first image content Rate and the input dimension, zoom in and out processing to the first image content;
It is described to show second picture material before the mosaic area, the method also includes:
According to the resolution ratio of the first image content and the input dimension, by second picture material zoom to it is described The resolution ratio of first picture material not being scaled before processing.
6. a kind of image restoration device, which is characterized in that described device includes:
First determining module, the mosaic area being configured to determine that on video image;
First processing module is configured as the first picture material for the mosaic area for determining first determining module It is input in the convolutional neural networks trained, by the positive convolution module of the convolutional neural networks in the first image Hold and carry out process of convolution, the fisrt feature for obtaining the first image content indicates;
Second processing module is configured as through the warp volume module in the convolutional neural networks to the first processing module The obtained fisrt feature indicates progress deconvolution processing, obtains the mosaic area by the second image before mosaic Hold, the resolution ratio of second picture material is identical as the resolution ratio of the first image content;
Display module, second picture material for being configured as obtaining the Second processing module are shown in the mosaic Region.
7. device according to claim 6, which is characterized in that described device further include:
Training module is configured as the mosaic sample by setting quantity to the positive convolution module and the warp volume module Corresponding characteristic parameter is trained, and obtains the corresponding fisrt feature parameter sets of the positive convolution module and the warp The second feature parameter sets of volume module;
Third processing module is configured as the fisrt feature parameter sets obtained based on training module training, passed through The positive convolution module carries out process of convolution to the mosaic sample of the setting quantity, obtains second feature expression;
Fourth processing module is configured as the second feature parameter sets obtained based on training, passes through the warp product module Block indicates progress deconvolution processing to the second feature that the third processing module obtains, and obtains the horse of the setting quantity Match the reconstructed image of gram sample;
Second determining module is configured to determine that reconstructed image and the institute of the setting quantity that the fourth processing module obtains State the reset error between the corresponding original image of mosaic sample of setting quantity;
Control module is configured as the control when the reset error that second determining module determines reaches the condition of convergence The convolutional neural networks deconditioning obtains the convolutional neural networks trained.
8. device according to claim 7, which is characterized in that the output dimension of the warp volume module and the positive convolution The input dimension of module is identical.
9. device according to claim 6, which is characterized in that first determining module includes:
Input submodule is configured as video image being input to the disaggregated model trained, and the disaggregated model is by setting number The mosaic sample of amount to default sorting algorithm by being trained to obtain;
Detection sub-module is configured as detecting the view of the input submodule input based on the disaggregated model trained Mosaic area on frequency image.
10. device according to claim 6, which is characterized in that described device further include:
Third determining module is configured to determine that the resolution ratio and the positive convolution module of the first picture material of mosaic area Input dimension it is whether identical;
Zoom module, if being configured as resolution ratio and the input that the third determining module determines the first image content Dimension is not identical, according to the resolution ratio of the first image content and the input dimension, carries out to the first image content Scaling processing;
The Zoom module is additionally configured to show second picture material in the mosaic area in the display module Before domain, according to the resolution ratio of the first image content and the input dimension, by second picture material zoom to The resolution ratio of the first image content not being scaled before processing.
11. a kind of image restoration device, which is characterized in that described device includes:
Processor;
Memory for storage processor executable instruction;
Wherein, the processor is configured to:
Determine the mosaic area on video image;
The first picture material of the mosaic area is input in the convolutional neural networks trained, the convolution mind is passed through Positive convolution module through network carries out process of convolution to the first image content, and obtain the first image content first is special Sign indicates;
The fisrt feature is indicated by the warp volume module in the convolutional neural networks to carry out deconvolution processing, obtains institute Mosaic area is stated by the second picture material before mosaic, in the resolution ratio and the first image of second picture material The resolution ratio of appearance is identical;
Second picture material is shown in the mosaic area.
CN201610958087.2A 2016-10-27 2016-10-27 Image recovery method and device Active CN106530227B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610958087.2A CN106530227B (en) 2016-10-27 2016-10-27 Image recovery method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610958087.2A CN106530227B (en) 2016-10-27 2016-10-27 Image recovery method and device

Publications (2)

Publication Number Publication Date
CN106530227A CN106530227A (en) 2017-03-22
CN106530227B true CN106530227B (en) 2019-08-06

Family

ID=58325864

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610958087.2A Active CN106530227B (en) 2016-10-27 2016-10-27 Image recovery method and device

Country Status (1)

Country Link
CN (1) CN106530227B (en)

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107016657B (en) * 2017-04-07 2019-05-28 河北工业大学 The restorative procedure of the face picture covered by reticulate pattern
CN108734667B (en) * 2017-04-14 2022-01-18 Tcl科技集团股份有限公司 Image processing method and system
CN108230253B (en) * 2017-05-08 2020-11-27 北京市商汤科技开发有限公司 Image restoration method and device, electronic equipment and computer storage medium
CN107392189B (en) * 2017-09-05 2021-04-30 百度在线网络技术(北京)有限公司 Method and device for determining driving behavior of unmanned vehicle
CN107578054A (en) * 2017-09-27 2018-01-12 北京小米移动软件有限公司 Image processing method and device
CN107992894B (en) * 2017-12-12 2022-02-08 北京小米移动软件有限公司 Image recognition method, image recognition device and computer-readable storage medium
CN108876726A (en) * 2017-12-12 2018-11-23 北京旷视科技有限公司 Method, apparatus, system and the computer storage medium of image procossing
CN108010538B (en) * 2017-12-22 2021-08-24 北京奇虎科技有限公司 Audio data processing method and device and computing equipment
CN109564638B (en) * 2018-01-15 2023-05-26 深圳鲲云信息科技有限公司 Artificial intelligence processor and processing method applied by same
CN108805884A (en) * 2018-06-13 2018-11-13 北京搜狐新媒体信息技术有限公司 A kind of mosaic area's detection method, device and equipment
CN109410123B (en) * 2018-10-15 2023-08-18 深圳市能信安科技股份有限公司 Deep learning-based mosaic removing method and device and electronic equipment
US11017499B2 (en) * 2018-12-21 2021-05-25 Here Global B.V. Method, apparatus, and computer program product for generating an overhead view of an environment from a perspective image
US11082757B2 (en) 2019-03-25 2021-08-03 Rovi Guides, Inc. Systems and methods for creating customized content
CN110111282B (en) * 2019-05-09 2021-05-11 杭州电子科技大学上虞科学与工程研究院有限公司 Video deblurring method based on motion vector and CNN
US11562016B2 (en) 2019-06-26 2023-01-24 Rovi Guides, Inc. Systems and methods for generating supplemental content for media content
US11256863B2 (en) 2019-07-19 2022-02-22 Rovi Guides, Inc. Systems and methods for generating content for a screenplay
US11145029B2 (en) 2019-07-25 2021-10-12 Rovi Guides, Inc. Automated regeneration of low quality content to high quality content
CN111432286B (en) 2019-12-31 2022-05-20 杭州海康威视数字技术股份有限公司 Video processing method, device and system
US11604827B2 (en) 2020-02-21 2023-03-14 Rovi Guides, Inc. Systems and methods for generating improved content based on matching mappings
TWI715448B (en) * 2020-02-24 2021-01-01 瑞昱半導體股份有限公司 Method and electronic device for detecting resolution
CN111401453A (en) * 2020-03-18 2020-07-10 西安电子科技大学 Mosaic image classification and identification method and system
CN112329634B (en) * 2020-11-05 2024-04-02 华中师范大学 Classroom behavior identification method and device, electronic equipment and storage medium
CN112184597A (en) * 2020-11-05 2021-01-05 温州大学大数据与信息技术研究院 Image restoration device and method
CN113905276B (en) * 2021-09-24 2023-11-14 威视芯半导体(杭州)有限公司 Video coding processing method, system, equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102131079A (en) * 2011-04-20 2011-07-20 杭州华三通信技术有限公司 Method and device for eliminating motion blur of image
CN104200224A (en) * 2014-08-28 2014-12-10 西北工业大学 Valueless image removing method based on deep convolutional neural networks
CN105096279A (en) * 2015-09-23 2015-11-25 成都融创智谷科技有限公司 Digital image processing method based on convolutional neural network

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9373160B2 (en) * 2013-12-18 2016-06-21 New York University System, method and computer-accessible medium for restoring an image taken through a window

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102131079A (en) * 2011-04-20 2011-07-20 杭州华三通信技术有限公司 Method and device for eliminating motion blur of image
CN104200224A (en) * 2014-08-28 2014-12-10 西北工业大学 Valueless image removing method based on deep convolutional neural networks
CN105096279A (en) * 2015-09-23 2015-11-25 成都融创智谷科技有限公司 Digital image processing method based on convolutional neural network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于卷积神经网络的正则化方法;吕国豪;《计算机研究与发展》;20140604;第51卷(第9期);1891-1900

Also Published As

Publication number Publication date
CN106530227A (en) 2017-03-22

Similar Documents

Publication Publication Date Title
CN106530227B (en) Image recovery method and device
CN106339680B (en) Face key independent positioning method and device
EP3996379A1 (en) Video cover determining method and device, and storage medium
CN105100609B (en) The adjusting method of mobile terminal and acquisition parameters
CN106375772B (en) Video broadcasting method and device
CN110060215B (en) Image processing method and device, electronic equipment and storage medium
CN109257645B (en) Video cover generation method and device
CN105204808B (en) Projective techniques, device and the terminal device of picture
CN110517185A (en) Image processing method, device, electronic equipment and storage medium
CN104090709B (en) Picture switching method and device
CN109831636A (en) Interdynamic video control method, terminal and computer readable storage medium
CN109889724A (en) Image weakening method, device, electronic equipment and readable storage medium storing program for executing
CN104361558B (en) Image processing method, device and equipment
CN104933419B (en) The method, apparatus and red film for obtaining iris image identify equipment
WO2020114236A1 (en) Keypoint detection method and apparatus, electronic device, and storage medium
CN105120301B (en) Method for processing video frequency and device, smart machine
CN105653032A (en) Display adjustment method and apparatus
CN105957037B (en) Image enchancing method and device
CN109788268A (en) Terminal and its white balance correction control method and computer readable storage medium
CN107832746A (en) Expression recognition method and device
CN110580688A (en) Image processing method and device, electronic equipment and storage medium
CN110276418A (en) Character recognition method, device, electronic equipment and storage medium based on picture
CN110807769B (en) Image display control method and device
CN106713656B (en) Shooting method and mobile terminal
CN105426904B (en) Photo processing method, device and equipment

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant