CN108257144A

CN108257144A - Stingy drawing method, device, equipment, storage medium and program based on neural network

Info

Publication number: CN108257144A
Application number: CN201810073359.XA
Authority: CN
Inventors: 张佳维; 任思捷; 潘金山
Original assignee: Shenzhen Sensetime Technology Co Ltd
Current assignee: Shenzhen Sensetime Technology Co Ltd
Priority date: 2018-01-25
Filing date: 2018-01-25
Publication date: 2018-07-06

Abstract

The application embodiment disclose a kind of stingy drawing method based on neural network, neural network training method, device, electronic equipment, computer readable storage medium and computer program, stingy drawing method therein includes：At least by pending image and the prospect constraints graph of the pending image, as input information, neural network is supplied to carry out foreground detection, obtains foreground detection result；The prospect constraints graph is the constraint information of notable object in the pending image；According to foreground detection as a result, taking out foreground image from pending image；Wherein, the neural network includes：Coding network unit and decoding network element, and the connection mode between coding network unit and decoding network element includes：The output of last non-network layer of the coding network unit be supplied to the decoding network unit any network layer and/or, the output of last network layer of the coding network unit is supplied to the non-first network layer of the decoding network unit.

Description

Stingy drawing method, device, equipment, storage medium and program based on neural network

Technical field

This application involves computer vision technique, more particularly, to a kind of stingy drawing method based on neural network, based on god Stingy map device, the training method of neural network, the training device of neural network, electronic equipment through network computer-readable are deposited Storage media and computer program.

Background technology

Stingy figure typically refers to：Prospect in image is stripped out from image.

Stingy figure can be applied in the fields such as film making, net cast and photo disposal, for example, in Making Movies spy During skill, by the way that prospect is stripped out from a series of video frame, and by the prospect being stripped out addition with the corresponding back of the body In a series of video frame of scape, so as to form a kind of special effects.

It is how quick and be accurately stripped out prospect from image, be in computer vision field one merit attention The technical issues of.

Invention content

The application embodiment provides the technical solution of a kind of stingy figure based on neural network and training neural network.

According to the application embodiment wherein on the one hand, a kind of stingy drawing method based on neural network, the method are provided Including：At least by pending image and the prospect constraints graph of the pending image, as input information, it is supplied to nerve net Network carries out foreground detection, obtains foreground detection result；The prospect constraints graph is the significantly constraint of object in the pending image Information；According to the foreground detection as a result, taking out foreground image from the pending image；Wherein, the neural network Including：Coding network unit and decoding network element, and the connection side between the coding network unit and decoding network element Formula includes：The output of last non-network layer of the coding network unit is supplied to any network of the decoding network unit Layer and/or, the output of last network layer of the coding network unit is supplied to the non-first net of the decoding network unit Network layers.

In one embodiment of the application, the coding network unit includes：At least two layers of convolutional layer；The coding network The characteristic pattern of at least one layer of convolutional layer output in unit is supplied to the non-first layer in decoding network unit, and is supplied to decoding The characteristic pattern of network element is identical with the size of characteristic pattern that the last layer of the non-first layer exports.

In the another embodiment of the application, at least one layer of convolutional layer in the coding network unit is with the first predetermined step It is long, down-sampling is carried out to the characteristic pattern of the pending image of the output of a convolutional layer thereon, to form resolution ratio reduction, and number Measure increased characteristic pattern.

In the application a further embodiment, the network layer of the decoding network unit includes：At least one layer of convolutional layer with And at least one layer of warp lamination；The input of at least one layer warp lamination includes：Convolutional layer output in coding network unit The characteristic pattern of pending image and the warp lamination in decoding network unit a upper convolutional layer output.

In the application a further embodiment, at least one layer of warp lamination in the decoding network unit is predetermined with second Step-length up-samples the characteristic pattern of the pending image of the output of a convolutional layer thereon, to form resolution ratio increase, and The characteristic pattern of quantity reduction.

In the application a further embodiment, it is described at least by the prospect of pending image and the pending image about Shu Tu as input information, is supplied to neural network to carry out foreground detection and includes：At least by pending image and described wait to locate The initial prospect constraints graph of image is managed, as input information, neural network is supplied to carry out foreground detection；At least by pending figure Picture and the iv-th iteration prospect constraints graph for the pending image as input information, are supplied to neural network to carry out Foreground detection；Wherein, the N is more than 1, and the iv-th iteration prospect constraints graph for the pending image includes：Nerve Network is in the N-1 times iterative process for the pending image, the foreground detection knot of the pending image of output Fruit.

In the application a further embodiment, the initial prospect constraints graph of the pending image includes：Pending image Three components；And/or the iv-th iteration prospect constraints graph of the pending image includes：Neural network is in the N-1 times iteration In the process, the α of the pending image of output covers figure.

In the application a further embodiment, the input information of the neural network further includes：The pending image Gradient map.

In the application a further embodiment, the neural network is to utilize the image sample with prospect constraint markup information What this training formed.

In the application a further embodiment, the process of the training neural network includes：It concentrates and obtains from training data Image pattern；At least by described image sample and the prospect constraints graph of described image sample, information is inputted as training, is provided Foreground detection is carried out to neural network to be trained, obtains the foreground detection result of described image sample；With described image sample Foreground detection result and described image sample prospect constraint markup information between difference for tutorial message, wait to instruct to described Experienced neural network exercises supervision study.

In the application a further embodiment, it is described at least by the prospect of described image sample and described image sample about Shu Tu inputs information as training, and neural network to be trained is supplied to carry out foreground detection and is included：At least by image pattern with And the initial prospect constraints graph of described image sample, information is inputted as training, before neural network to be trained is supplied to carry out Scape detects；At least by image pattern and the iv-th iteration prospect constraints graph for described image sample, believe as training input Breath is supplied to neural network to be trained to carry out foreground detection.

In the application a further embodiment, the initial prospect constraints graph of described image sample includes：The three of image pattern Component；And/or the iv-th iteration prospect constraints graph for described image sample includes：Neural network is for the figure In the N-1 times iterative process of decent, the α of the described image sample of output covers figure.

In the application a further embodiment, the training input information of the neural network to be trained further includes：It is described The gradient map of image sample.

According to the wherein another aspect of the application embodiment, a kind of training method of neural network, the training are provided Method includes：It is concentrated from training data and obtains image pattern；At least by the prospect of described image sample and described image sample Constraints graph inputs information as training, neural network to be trained is supplied to carry out foreground detection, obtains described image sample Foreground detection result；Between the foreground detection result of described image sample and the prospect of described image sample constraint markup information Difference for tutorial message, exercise supervision to the neural network to be trained study.

It is described at least to constrain the prospect of described image sample and described image sample in one embodiment of the application Figure inputs information as training, and neural network to be trained is supplied to carry out foreground detection and is included：At least by described image sample And the prospect constraints graph of described image sample, information is inputted as training, neural network to be trained is supplied to carry out prospect Detection includes：At least by image pattern and the initial prospect constraints graph of described image sample, information is inputted as training, is provided Foreground detection is carried out to neural network to be trained；At least by image pattern and the iv-th iteration for described image sample Prospect constraints graph inputs information as training, neural network to be trained is supplied to carry out foreground detection.

In the another embodiment of the application, the initial prospect constraints graph of described image sample includes：The three of image pattern Component；And/or the iv-th iteration prospect constraints graph for described image sample includes：Neural network is for the figure In the N-1 times iterative process of decent, the α of the described image sample of output covers figure.

According to the application embodiment wherein in another aspect, a kind of stingy map device based on neural network of offer, described Device includes：Testing result module is obtained, at least by pending image and the prospect constraints graph of the pending image, As input information, neural network is supplied to carry out foreground detection, obtains foreground detection result；The prospect constraints graph is described The constraint information of notable object in pending image；Take foreground image module, for according to the foreground detection as a result, from described Foreground image is taken out in pending image；Wherein, the neural network includes：Coding network unit and decoding network element, And the connection mode between the coding network unit and decoding network element includes：The coding network unit it is non-last The output of network layer be supplied to the decoding network unit any network layer and/or, the coding network unit last The output of network layer is supplied to the non-first network layer of the decoding network unit.

In the application a further embodiment, the network side of the decoding network unit includes：At least one layer of convolutional layer with And at least one layer of warp lamination；The input of at least one layer warp lamination includes：Convolutional layer output in coding network unit The characteristic pattern of pending image and the warp lamination in decoding network unit a upper convolutional layer output.

In the application a further embodiment, the acquisition testing result module is specifically used for：At least by pending image And the initial prospect constraints graph of the pending image, as input information, neural network is supplied to carry out foreground detection；Extremely The pending image of major general and the iv-th iteration prospect constraints graph for the pending image as input information, are supplied to Neural network carries out foreground detection；Wherein, the N is more than 1, and the iv-th iteration prospect for the pending image is about Beam figure includes：Neural network is in the N-1 times iterative process for the pending image, the pending image of output Foreground detection result.

In the application a further embodiment, described device further includes：Sample module is obtained, for being concentrated from training data Obtain image pattern；Pattern detection object module is obtained, at least will be before described image sample and described image sample Scape constraints graph inputs information as training, neural network to be trained is supplied to carry out foreground detection, obtains described image sample Foreground detection result；Module is supervised, for the prospect of the foreground detection result of described image sample and described image sample The difference constrained between markup information is tutorial message, and exercise supervision to the neural network to be trained study.

In the application a further embodiment, the acquisition pattern detection object module is specifically used for：At least by image sample The initial prospect constraints graph of this and described image sample inputs information as training, be supplied to neural network to be trained into Row foreground detection；It is defeated as training at least by image pattern and the iv-th iteration prospect constraints graph for described image sample Enter information, neural network to be trained is supplied to carry out foreground detection.

According to the application embodiment wherein in another aspect, provide a kind of training device of neural network, including：It obtains Sample module obtains image pattern for being concentrated from training data；Pattern detection object module is obtained, at least by the figure Decent and the prospect constraints graph of described image sample, as training input information, be supplied to neural network to be trained into Row foreground detection obtains the foreground detection result of described image sample；Module is supervised, for being examined with the prospect of described image sample The difference surveyed between result and the prospect constraint markup information of described image sample is tutorial message, to the nerve to be trained Network exercises supervision study.

In one embodiment of the application, the acquisition pattern detection object module is specifically used for：At least by image pattern And the initial prospect constraints graph of described image sample, information is inputted as training, neural network to be trained is supplied to carry out Foreground detection；At least by image pattern and the iv-th iteration prospect constraints graph for described image sample, inputted as training Information is supplied to neural network to be trained to carry out foreground detection.

According to the application embodiment another aspect, a kind of electronic equipment is provided, including：Memory, based on storing Calculation machine program；Processor, for performing the computer program stored in the memory, and the computer program is performed When, realize the application either method embodiment.

According to the application embodiment another aspect, a kind of computer readable storage medium is provided, is stored thereon with meter Calculation machine program when the computer program is executed by processor, realizes the application either method embodiment.

According to another aspect of the application embodiment, a kind of computer program is provided, including computer instruction, works as institute When stating computer instruction and being run in the processor of equipment, the application either method embodiment is realized.

Based on the application provide the stingy drawing method based on neural network, neural network training method, based on nerve net The stingy map device of network, the training device of neural network, electronic equipment, computer readable storage medium and computer program, this Shen Please by using neural network, foreground detection is carried out for pending image, is conducive to improve the speed of foreground detection.Passing through will The output of last non-network layer of coding network unit be supplied to decoding network unit any network layer and/or, pass through by The output of last network layer of coding network unit is supplied to the non-first network layer of decoding network unit, makes coding network list More output informations in member can be supplied to decoding network unit, and more image detail letters are provided for decoding network unit Breath is conducive to decoding network unit and the foreground detection of pending image is handled, so as to be conducive to improve neural network output The accuracy of foreground detection result.It follows that the technical solution that the application provides is conducive to improve stingy figure efficiency and scratches figure Accuracy.

Below by drawings and embodiments, the technical solution of the application is described in further detail.

Description of the drawings

The attached drawing of a part for constitution instruction describes presently filed embodiment, and is used to solve together with description Release the principle of the application.

With reference to attached drawing, according to following detailed description, the application can be more clearly understood, wherein：

Fig. 1 is the flow chart of one embodiment of stingy drawing method based on neural network of the application；

Fig. 2 is the schematic diagram of an embodiment of the neural network for being used to implement stingy figure of the application；

Fig. 3 is the flow chart of the stingy drawing method another embodiment based on neural network of the application；

Fig. 4 is the flow chart of the stingy drawing method yet another embodiment based on neural network of the application；

Fig. 5 is the flow chart of one embodiment of training method of the neural network of the application；

Fig. 6 is the structure diagram of one embodiment of stingy map device based on neural network of the application；

Fig. 7 is the structure diagram of one embodiment of training device of the neural network of the application；

Fig. 8 is the block diagram for the example devices for realizing the application embodiment.

Specific embodiment

The various exemplary embodiments of the application are described in detail now with reference to attached drawing.It should be noted that：Unless in addition have Body illustrates that the unlimited system of component and the positioned opposite of step, numerical expression and the numerical value otherwise illustrated in these embodiments is originally The range of application.

Simultaneously, it should be appreciated that for ease of description, the size of the various pieces shown in attached drawing is not according to reality Proportionate relationship draw.

It is illustrative to the description only actually of at least one exemplary embodiment below, is never used as to the application And its application or any restrictions that use.

Technology, method and equipment known to person of ordinary skill in the relevant may be not discussed in detail, but In the case of appropriate, the technology, method and apparatus should be considered as part of specification.

It should be noted that：Similar label and letter represents similar terms in following attached drawing, therefore, once a certain item exists It is defined in one attached drawing, then in subsequent attached drawing does not need to that it is further discussed.

The embodiment of the present application can be applied to the electronic equipments such as terminal device, computer system and server, can be with crowd Mostly other general either dedicated computing system environments or configuration operate together.Suitable for terminal device, computer system with And the example of well-known terminal device, computing system, environment and/or configuration that the electronic equipments such as server are used together, Including but not limited to：It is personal computer system, server computer system, thin client, thick client computer, hand-held or above-knee set System standby, based on microprocessor, set-top box, programmable consumer electronics, NetPC Network PC, little types Ji calculate machine Xi Tong ﹑ Large computer system and distributed cloud computing technology environment including any of the above described system, etc..

The electronic equipments such as terminal device, computer system and server can be in the computer performed by computer system It is described under the general linguistic context of system executable instruction (such as program module).In general, program module can include routine, program, Target program, component, logic and data structure etc., they perform specific task or realize specific abstract data class Type.Computer system/server can be implemented in distributed cloud computing environment, in distributed cloud computing environment, task be by What the remote processing devices being linked through a communication network performed.In distributed cloud computing environment, program module can be located at packet It includes on the Local or Remote computing system storage medium of storage device.

Exemplary embodiment

Fig. 1 is the flow chart of one embodiment of the stingy drawing method based on neural network of the application.The application passes through profit Stingy figure is realized with neural network, is conducive to improve the efficiency of stingy figure, is conducive to that stingy diagram technology is enable preferably to apply in electricity In the fields such as shadow making, net cast and photo batch processing.

As shown in Figure 1, the embodiment method mainly includes：Step S100 and step S110.Below to each step in Fig. 1 Suddenly it is described in detail.

S100, at least by the prospect constraints graph of pending image and pending image, as input information, be supplied to god Foreground detection is carried out through network, obtains foreground detection result.

In an optional example, the pending image in the application can be that the figures such as static picture or photo are presented Picture, or video frame in dynamic video etc. is presented.The pending image can be coloured image (as based on RGB Coloured image etc.) or gray level image etc..Prospect in the pending image is usually the notable object in image, for example, waiting to locate It can be human body, face and animal etc. to manage the prospect in image.The application does not limit the specific manifestation form of pending image, The specific manifestation form of the notable object in pending image is not limited yet.

In an optional example, the input information of the neural network of the application can include：It pending image and waits to locate Manage the prospect constraints graph of image.The input information of the neural network of the application can also including：Pending image and pending On the basis of the prospect constraints graph of image, further include：The other information for the characteristics of pending image can be embodied, for example, treating Handle gradient map of image etc..Three width figures of the leftmost side of Fig. 2 are the input information of neural network, and this three width figure is from top to bottom Respectively：The prospect constraints graph of pending image, the gradient map of pending image and pending image.The application will be by that will treat Handle input information of the gradient map as neural network of image, be conducive to raising neural network foreground detection result it is accurate Property.

In an optional example, the prospect constraints graph of the pending image in the application is mainly used for auxiliary nervous network Foreground detection is carried out, prospect constraints graph may be considered a kind of constraint information of the notable object in pending image, prospect constraint Figure can be the initial marking setting of the notable object in pending image.The prospect constraints graph of pending image generally includes Three parts, this three parts are respectively：For identify the notable object in pending image part, for identifying pending figure The part of background as in and the part for identifying the uncertain content in pending image；For example, prospect constraints graph In 1 for identifying notable object in pending image, in prospect constraints graph 0 for identifying the back of the body in pending image Scape, other numerical value in prospect constraints graph are used to identify the uncertain content in pending image.

In an optional example, the application can be by the Trimap (three components) or alpha of pending image Matting (α covers figure) etc. is used as prospect constraints graph.The application does not limit the specific manifestation form of prospect constraints graph, as long as can The constraint information of the notable object of pending image is provided for neural network.

In an optional example, the situation for inputting information and being iterated processing can be directed in the neural network of the application Under, for a pending image, in first iterative process, the application can be by pending image and pending image Initial prospect constraints graph, as input information, be supplied to neural network；In subsequent iterative process, the application can incite somebody to action The foreground detection result that neural network exports in a preceding iterative process is incited somebody to action as the prospect constraints graph during current iteration Prospect constraints graph during the current iteration of pending image and pending image as input information, is supplied to nerve Network.

In an optional example, the situation for inputting information and being iterated processing can be directed in the neural network of the application Under, for the pending image, in first iterative process, the application can be by pending image, pending image The initial prospect constraints graph of gradient map and pending image as input information, is supplied to neural network；In subsequent iteration In the process, the foreground detection result that the application can export neural network in a preceding iterative process is as current iteration process In prospect constraints graph, and by the current iteration process of pending image, the gradient map of pending image and pending image In the prospect constraints graph arrow (Iterative Refinement) of the iterative refinement in such as Fig. 2 (indicated), as input Information is supplied to neural network.

Since neural network for pending image exports the accuracy of foreground detection result for the first time, it will usually better than first Prospect constraints graph in iterative process, therefore, the application are iterated processing by being directed to pending image, and in iterative processing In the process, foreground detection result neural network currently exported is believed as the input of the neural network during next iteration Prospect constraints graph in breath is conducive to improve the accuracy of the finally formed foreground detection result of neural network.

In an optional example, the initial prospect constraints graph of the pending image in the application can be specially：It waits to locate Manage the Trimap of image.The Trimap of pending image used in this application can be obtained using various ways, for example, The Trimap of pending image used in this application can be：Using based on QCUT (Quantum Cuts, quantum are cut), HDCT (High Dimensional Color Transform, higher-dimension color conversion) or ST (Saliency Tree, significantly Tree) notable object detecting method obtain.The application does not limit the specific acquisition pattern of the Trimap of pending image.In addition, The foreground detection result of neural network output can be specially in a preceding iterative process in the application：Pending image alpha matting。

In an optional example, the gradient map of pending image used in this application can be obtained using various ways , for example, the gradient map of pending image used in this application can be：Utilize the image gradient based on Sobel operators Algorithm, the image gradient algorithm based on Robinson operators are obtained based on the image gradient algorithm of Laplace operators Gradient map.The application does not limit the specific implementation for the gradient map for obtaining pending image.

Seen from the above description, the prospect constraints graph and gradient map of the pending image in the application, can be not It is carried out on the basis of further detecting dependent on for pending image, it is more convenient efficiently to obtain, so as to make the application's Neural network can become neural network end to end.

In an optional example, the neural network in the application includes：Coding network unit and decoding network element.This Coding network unit in application is referred to as Encoder (encoder) or coding network etc..Decoding net in the application Network unit is referred to as Decoder (decoder) or decoding network etc..Coding network unit in the application generally includes： At least two layers of convolutional layer, for example, the number of plies for the convolutional layer that coding network unit is included is no less than four layers.Decoding in the application Network element generally includes：At least one layer of convolutional layer and at least one layer of warp lamination, for example, what decoding network unit was included The number of plies of convolutional layer and warp lamination is no less than two layers.The number of plies that neural network is included is more, then neural network is deeper. The application does not limit the coding network unit in neural network and decodes convolutional layer and warp lamination that network element is included The number of plies.

In an optional example, in each convolutional layer and decoding network unit in the coding network unit of the application The size of the convolution kernel of each convolutional layer is usually identical, for example, the size of the convolution kernel of all convolutional layers is 3 × 3.The application's The size of the convolution kernel of each warp lamination in decoding network unit is usually identical, for example, the convolution kernel of all warp laminations Size is 4 × 4.

In an optional example, the connection mode between coding network unit and decoding network element in the application can To include：Connection (Skip Link) mode is jumped, for example, between the coding network unit and decoding network element in the application It is connected to including on the basis of normal connection mode, further including：Jump connection.Normal connection mode in the application typically refers to, The output of the last one network layer (such as convolutional layer) in coding network unit, first be provided in decoding network unit Network layer (such as first warp lamination or convolutional layer).Jump connection mode in the application generally includes：Coding network unit The last one non-network layer output, be provided to decoding network unit any network layer (such as any warp lamination or Any convolutional layer)；Above-mentioned jump connection can generally also include：The output of the last one network layer of coding network unit is carried Supply non-first network layer (any warp lamination or any volume such as other than first network layer of decoding network unit Lamination).It is needing the network layer in the network layer in coding network unit and decoding network unit, using jump connection side When formula is connected, the size of characteristic pattern of network layer output and decoding network unit in coding network unit should ensure that In the network layer input characteristic pattern size it is identical.

In one example, the output of at least two layers convolutional layer in the coding network unit of the application, can provide respectively To in decoding network unit different layers (for example, different warp laminations, for another example, different convolutional layers, for another example, a deconvolution Layer and a convolutional layer etc.).That is, the input packet of at least one layer (warp lamination or convolutional layer) in decoding network unit It includes：The characteristic pattern and this layer in decoding network unit of the pending image of convolutional layer output in coding network unit The output of last layer (convolutional layer or warp lamination in such as decoding network unit).Relative to the layer in decoding network unit For the characteristic pattern of (such as convolutional layer or warp lamination) output, the characteristic pattern of the convolutional layer output in coding network unit is often More details can be retained, i.e., some details lost in the characteristic pattern of the layer output in decoding network unit often retain In the characteristic pattern of convolutional layer output in coding network unit, the application is by using connection mode is jumped, by coding network list The characteristic pattern of convolutional layer output in member is supplied to warp lamination and/or convolutional layer in decoding network unit, is conducive to make Corresponding details is remained in warp lamination and/or formation characteristic pattern, so as to be conducive to improve the prospect of neural network output The accuracy of testing result.

In an optional example, each convolutional layer in coding network unit is for respectively forming the feature of pending image Figure, and at least part convolutional layer in coding network unit, the feature of pending image that a convolutional layer thereon can be exported Figure carries out down-sampling, and resolution ratio is lower, and more characteristic pattern so as to being formed, for example, in coding network unit M1 (M1 is more than 1) layer convolutional layer, using step-length as the pending image of a 2 pairs of convolutional layer (M1-1 layers of convolutional layer) output thereon Characteristic pattern, carry out down-sampling, can be the so as to the quantity of the characteristic pattern of pending image that M1 layer convolutional layer export Twice of the quantity of the characteristic pattern of M1-1 layers of convolutional layer output, however, the spy of the pending image of the M1 layers of convolutional layer output The resolution ratio of figure is levied, can be the half of the resolution ratio of the characteristic pattern of M1-1 layers of convolutional layer output.Down-sampling in the application Step-length can also use 3 or other values.The application does not limit the specific value of the step-length of down-sampling.

In an optional example, each warp lamination and convolution in addition to last layer in decoding network unit Layer, is for respectively forming the characteristic pattern of pending image, last layer in decoding network unit is typically formed alpha matting.At least part warp lamination in the decoding network unit, the pending image that a convolutional layer thereon can be exported Characteristic pattern up-sampled, so as to form resolution ratio higher, and the characteristic pattern that quantity is less, for example, decoding network list M2 (M2 is more than or equal to 1) layer warp lamination in member, using step-length as a 2 pairs of convolutional layer (M2-1 layers of convolutional layer) outputs thereon The characteristic pattern of pending image up-sampled, the characteristic pattern of pending image exported so as to the M2 layer warp lamination Quantity can be the half of the quantity of the characteristic pattern of M2-1 layers of convolutional layer output, however, the M2 layers of warp lamination output The resolution ratio of the characteristic pattern of pending image can be twice of the resolution ratio of the characteristic pattern of M1-1 layers of convolutional layer output.This The step-length of up-sampling in application can also use 3 or other values.The application does not limit the specific value of the step-length of up-sampling.

In an optional example, a specific example of the neural network of the application is as shown in Figure 2.In Fig. 2, in box Conv represent convolutional layer, the Deconv in box represents warp lamination, and 11 Conv in the left side of Fig. 2 belong to coding network list Member, 4 Deconv and 4 Conv on the right side of Fig. 2 belong to decoding network unit.11 Conv reconciliation in coding network unit The convolution kernel size of 4 Conv in code network element is 3 × 3.The convolution kernel of 4 Deconv in decoding network unit is big Small is 4 × 4.The output of the second layer convolutional layer in coding network unit in Fig. 2 is connected by jumping, and is provided to decoding net The 4th Deconv in network unit；The output of the 4th layer of convolutional layer in coding network unit is connected by jumping, and is provided to Third Deconv in decoding network unit；The output of layer 6 convolutional layer in coding network unit is connected by jumping, quilt Second Deconv being supplied in decoding network unit；The output of the 8th layer of convolutional layer in coding network unit passes through the company of jump It connects, first Deconv being provided in decoding network unit.

Conv 3 × 3 × 32 in Fig. 2 represents that the convolution kernel size of the convolutional layer is 3 × 3, which exports 32 spies Sign figure.Conv 3 × 3 × 64 ↓ 2 represents that the convolution kernel size of the convolutional layer is 3 × 3, which carries out down-sampling to characteristic pattern (resolution ratio reduction), and the step-length of down-sampling is 2, which can export 64 characteristic patterns.Deconv4 × 4 × 256 ↑ 2 represent The convolution kernel size of the warp lamination be 4 × 4, which up-samples characteristic pattern (resolution ratio raising), and on adopt The step-length of sample is 2, which can export 256 characteristic patterns.Deconv4 × 4 × 128 ↑ 2 represent the convolution of the warp lamination Core size is 4 × 4, which up-samples characteristic pattern, and the step-length up-sampled is 2, which can export 128 characteristic patterns.And so on, no longer other convolutional layers in Fig. 2 and warp lamination are illustrated one by one.

It should be strongly noted that above-mentioned neural network shown in Fig. 2 is a specific example, the god in the application Other forms can also be shown as through network, for example, one layer of increase/reduction or multilayer convolution in neural network in fig. 2 Layer and/or warp lamination etc..

In an optional example, the neural network of the application can be directed to the pending image of input and pending image Prospect constraints graph (such as Trimap, for another example, the alpha matting of preceding an iteration are by as prospect constraints graph) formation Alpha matting, and export.

In an optional example, the neural network of the application can also be directed to the pending image of input, pending figure It the prospect constraints graph (such as Trimap, for another example, the alpha matting of preceding an iteration are by as prospect constraints graph) of picture and treats The gradient profile of image is handled into alpha matting, and is exported.

S110, according to foreground detection as a result, taking out foreground image from pending image.

In an optional example, the application can determine to wait to locate according to the alpha matting that neural network exports The foreground area in image is managed, so as to take out foreground image from pending image.The application, which does not limit, utilizes nerve The alpha matting of network output take out the specific implementation of foreground image from pending image.

Fig. 3 is the flow chart of another embodiment of the stingy drawing method based on neural network of the application.As shown in figure 3, This method mainly includes：Step S300, step S310, step S320, step S330, step S340 and step S350.Below Each step in Fig. 3 is described in detail.

S300, the stingy drawing method for starting the application, and it is 1 to set current iteration number.To step S310.

S310, the Trimap for obtaining pending image and pending image, and by the pending image got and The Trimap of pending image is supplied to neural network as input information.To step S320.

S320, using neural network, based on current input information, foreground detection is carried out to pending image, so as to be formed The alpha matting of pending image, and export.To step S330.

S330, judge whether current iteration number meets default iterations (such as 2 or 3 or 4), change if meeting agreement For condition, then to step S340, if being unsatisfactory for predetermined iterated conditional, to step S350.

The alpha matting of S340, the pending image currently exported according to neural network are scratched from pending image Take out foreground image.To step S360.

S350, current iteration number is added to 1, and the pending image that pending image and neural network are currently exported Alpha matting as input information, be supplied to neural network, return to step S320.

S360, the application stingy drawing method terminate.

Fig. 4 is the flow chart of another embodiment of the stingy drawing method based on neural network of the application.As shown in figure 4, This method mainly includes：Step S400, step S410, step S420, step S430, step S440 and step S450.Below Each step in Fig. 4 is described in detail.

S400, the stingy drawing method for starting the application, and it is 1 to set current iteration number.To step S410.

S410, pending image, the gradient map of pending image and the Trimap of pending image are obtained, and will obtained The Trimap of the pending image, the gradient map of pending image and the pending image that arrive is supplied to god as input information Through network.To step S420.

S420, using neural network, based on current input information, foreground detection is carried out to pending image, so as to be formed The alpha matting of pending image, and export.To step S430.

S430, judge whether current iteration number meets default iterations (such as 2 or 3 or 4), change if meeting agreement For condition, then to step S440, if being unsatisfactory for predetermined iterated conditional, to step S450.

The alpha matting of S440, the pending image currently exported according to neural network are scratched from pending image Take out foreground image.To step S460.

S450, current iteration number adds to 1, and by the gradient map and neural network of pending image, pending image The alpha matting of the pending image currently exported are supplied to neural network, return to step S420 as input information.

S460, the application stingy drawing method terminate.

Fig. 5 is the flow chart of one embodiment that the application trains neural network.As shown in figure 5, the embodiment method packet It includes：Step S500, step S510 and step S520.Each step in Fig. 5 is described in detail below.

S500, acquisition image pattern is concentrated from training data.

In an optional example, the training data concentration in the application includes multiple images for being used to train neural network Sample, image pattern can be the coloured image sample based on RGB, or gray level image sample etc..Under normal conditions, often A image pattern is both provided with prospect constraint markup information (such as prospect constraint mark figure)；For example, each image pattern is respectively provided with Have：Alpha matting are marked.Training data concentrate image pattern prospect constraint markup information can be before training, It is obtained by artificial notation methods.In addition, the application can also use the existing training with alpha matting marks Data set.The application can according to random read take mode or according to image pattern ordering sequence reading manner, once from Training data, which is concentrated, reads one or more image pattern.

S510, at least by the prospect constraints graph of image pattern and image pattern, input information as training, be supplied to and treat Trained neural network carries out foreground detection, obtains the foreground detection result of image pattern.

In an optional example, the input information of the neural network to be trained of the application can include：Image pattern With the prospect constraints graph of image pattern.The input information of the neural network to be trained of the application can also including：Image sample On the basis of the prospect constraints graph of this and image pattern, further include：The other information for the characteristics of image pattern can be embodied, example Such as, gradient map of image sample etc..

In an optional example, the prospect constraints graph of image pattern generally includes three parts, this three parts is respectively：With In the part for identifying the notable object in image pattern, for the part that identifies the background in image pattern and for identifying Go out the part of the uncertain content in image pattern；For example, 1 in prospect constraints graph is notable in image pattern for identifying Object, in prospect constraints graph 0 for identifying the background in image pattern, other numerical value in prospect constraints graph are for identifying Uncertain content in image pattern.The application can be using Trimap alpha matting of image pattern etc. as before Scape constraints graph.The application does not limit the specific manifestation form of prospect constraints graph, as long as can be provided for neural network to be trained The constraint information of the notable object of image pattern.

In an optional example, it can be directed to input information in the neural network to be trained of the application and be iterated place In the case of reason, for an image pattern, in first iterative process, the application can be by image pattern and image sample This initial prospect constraints graph as input information, is supplied to neural network to be trained；In subsequent iterative process, this The foreground detection result that application can export neural network to be trained in a preceding iterative process is as current iteration process In prospect constraints graph, and by the prospect constraints graph during the current iteration of image pattern and image pattern, as input Information is supplied to neural network to be trained.

In an optional example, it can be directed to input information in the neural network to be trained of the application and be iterated place In the case of reason, for an image pattern, in first iterative process, the application can be by image pattern, image pattern Gradient map and image pattern initial prospect constraints graph, as input information be supplied to neural network to be trained；Rear In continuous iterative process, the application can be by the foreground detection result of neural network output to be trained in a preceding iterative process As the prospect constraints graph during current iteration, and by image pattern, the gradient map of image pattern and image sample this Prospect constraints graph in iterative process as input information, is supplied to neural network to be trained.

In an optional example, the initial prospect constraints graph of the image pattern in the application can be specially：Image sample This Trimap.The Trimap of image pattern used in this application can be obtained using various ways, specific as above-mentioned Description in method embodiment, this will not be repeated here.The specific acquisition of the Trimap of the unlimited imaged sample of the application Mode.In addition, the foreground detection result of neural network output to be trained in a preceding iterative process in the application can have Body is：The alpha matting of image pattern.

In an optional example, the gradient map of image pattern used in this application can be obtained using various ways , the specific description as in above method embodiment, this will not be repeated here.The application, which does not limit, obtains image pattern The specific implementation of gradient map.

In an optional example, the neural network to be trained of the application can be directed to the image pattern and image of input Prospect constraints graph (such as Trimap, for another example, the alpha matting of preceding an iteration are by as the prospect constraints graph) formation of sample Alpha matting, and export.

In an optional example, the neural network to be trained of the application can also be directed to the image pattern of input, figure The prospect constraints graph (such as Trimap, for another example, the alpha matting of preceding an iteration are by as prospect constraints graph) of decent with And the gradient profile of image pattern is into alpha matting, and exports.

The prospect of S520, the foreground detection result exported with neural network to be trained and image pattern constrain markup information Between difference for tutorial message, treat trained neural network and exercise supervision study.

In an optional example, the application can with reduce foreground detection result that neural network to be trained exports with For the purpose of difference between the prospect constraint markup information of image pattern, by adjusting the network ginseng in neural network to be trained Number (weights of such as convolution kernel) exercises supervision study so as to fulfill trained neural network is treated.One optional example, for Multiple images sample in batch process, the application can utilize loss function (such as L1 loss functions), be treated with reducing The foreground detection result of each image pattern of trained neural network output constrains mark letter with the prospect of corresponding image pattern For the purpose of difference between breath, calculated accordingly, so as to form a back-propagation process, in the back-propagation process, Adjust the network parameter in neural network to be trained.

In an optional example, for when the training for the neural network trained reaches predetermined iterated conditional, this Training process terminates.Predetermined iterated conditional in the application can include：The foreground detection knot of neural network output to be trained Difference between the prospect of fruit and image pattern constraint markup information meets predetermined difference requirement.Meet the predetermined difference in difference In the case of it is required that, this is treated trained neural network and successfully trains completion.Predetermined iterated conditional in the application can also Including：To this, neural network to be trained is trained, and the quantity of used image pattern reaches predetermined quantity requirement etc.. The quantity of the image pattern used reaches predetermined quantity requirement, however, in the case that difference does not meet predetermined difference requirement, this It is secondary to treat trained neural network and do not train successfully.Success training complete neural network can be used for pending image into Row FIG pull handle.

Fig. 6 is the structure diagram of stingy map device one embodiment based on neural network of the application.As shown in fig. 6, The device of the embodiment mainly includes：It obtains testing result module 600 and takes foreground image module 610.Optionally, the reality Applying the device of example can also include：Sample module 620 is obtained, obtain pattern detection object module 630 and supervises module 640.

Testing result module 600 is obtained to be mainly used at least constraining the prospect of pending image and pending image Figure as input information, is supplied to neural network to carry out foreground detection, to obtain foreground detection result.Obtain testing result mould For retouching in S100, S310, S320, S410 and S420 in the operation such as above method embodiment that block 600 specifically performs It states, this will not be repeated here.

Take foreground image module 610 be mainly used for according to obtain testing result module 600 obtain foreground detection as a result, Foreground image is taken out from pending image.The operation such as above method that foreground image module 610 specifically performs is taken to implement For the description in S110, S330, S340, S350, S430, S440 and S450 in mode, this will not be repeated here.

Sample module 620 is obtained to be mainly used for concentrating acquisition image pattern from training data.It is specific to obtain sample module 620 For the description in S500 in the operation such as above method embodiment of execution, this will not be repeated here.

Pattern detection object module 630 is obtained to be mainly used at least constraining the prospect of image pattern and image pattern Figure inputs information as training, neural network to be trained is supplied to carry out foreground detection, obtains the foreground detection of image pattern As a result.It obtains in the operation such as above method embodiment that pattern detection object module 630 specifically performs for retouching in S510 It states, this will not be repeated here.

Supervision module 640 is mainly used for the foreground detection result of image pattern and the prospect of image pattern constraint mark letter Difference between breath is tutorial message, treats trained neural network and exercises supervision study.The behaviour that supervision module 640 specifically performs Make as, for the description in S520, this will not be repeated here in above method embodiment.

Fig. 7 is the structure diagram of training device one embodiment of the neural network of the application.As shown in fig. 7, the reality The device for applying example mainly includes：Sample module 620 is obtained, obtain pattern detection object module 630 and supervises module 640.

Example devices

Fig. 8 shows the example devices 800 for being adapted for carrying out the application, and equipment 800 can be the control being configured in automobile System/electronic system, mobile terminal (for example, intelligent mobile phone etc.), personal computer (PC, for example, desktop computer or Notebook computer etc.), tablet computer and server etc..In Fig. 8, equipment 800 includes one or more processor, communication Portion etc., one or more of processors can be：One or more central processing unit (CPU) 801 and/or, one Or multiple image processors (GPU) 813 that the stingy figure based on neural network is carried out using neural network etc., processor can root Random access storage device is loaded into according to the executable instruction being stored in read-only memory (ROM) 802 or from storage section 808 (RAM) executable instruction in 803 and perform various appropriate actions and processing.Communication unit 812 can include but is not limited to net Card, the network interface card can include but is not limited to IB (Infiniband) network interface card.Processor can with read-only memory 802 and/or with Machine accesses communication in memory 830 and, to perform executable instruction, is connected by bus 804 with communication unit 812 and through communication unit 812 communicate with other target devices, so as to complete the corresponding steps in the application.

Operation performed by above-mentioned each instruction may refer to the associated description in above method embodiment, herein no longer in detail Explanation.

In addition, in RAM 803, various programs and data needed for device operation can also be stored with.CPU801、 ROM802 and RAM803 is connected with each other by bus 804.In the case where there is RAM803, ROM802 is optional module. RAM803 stores executable instruction or executable instruction is written into ROM802 at runtime, and executable instruction makes central processing Unit 801 performs the step included by above-mentioned method for segmenting objects.Input/output (I/O) interface 805 is also connected to bus 804. Communication unit 812 can be integrally disposed, may be set to be with multiple submodule (for example, multiple IB network interface cards), and respectively with always Line connects.

I/O interfaces 805 are connected to lower component：Importation 806 including keyboard, mouse etc.；It is penetrated including such as cathode The output par, c 807 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 808 including hard disk etc.； And the communications portion 809 of the network interface card including LAN card, modem etc..Communications portion 809 via such as because The network of spy's net performs communication process.Driver 810 is also according to needing to be connected to I/O interfaces 805.Detachable media 811, such as Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 810, as needed in order to be read from thereon Computer program be installed in as needed in storage section 808.

It should be strongly noted that framework as shown in Figure 8 is only a kind of optional realization method, in concrete practice process In, can the component count amount and type of above-mentioned Fig. 8 be selected, be deleted, be increased or be replaced according to actual needs；In different function Component setting on, can also be used it is separately positioned or integrally disposed and other implementations, for example, GPU and CPU separate setting, for another example GPU, can be integrated on CPU, communication unit separates setting, also can be integrally disposed in CPU or GPU is first-class by reason.These are replaceable Embodiment each fall within the protection domain of the application.

Particularly, it according to presently filed embodiment, may be implemented as calculating below with reference to the process of flow chart description Machine software program, for example, the application embodiment includes a kind of computer program product, it can it includes machine is tangibly embodied in The computer program on medium is read, computer program was included for the program code of the step shown in execution flow chart, program generation Code may include the corresponding instruction of step in the corresponding method for performing the application and providing.

In such embodiment, which can be downloaded and pacified from network by communications portion 809 It fills and/or is mounted from detachable media 811.When the computer program is performed by central processing unit (CPU) 801, perform The instruction of the above-mentioned corresponding steps of realization described in the application.

In one or more optional embodiments, the embodiment of the present disclosure additionally provides a kind of computer program program production Product, for storing computer-readable instruction, described instruction is performed so that computer is performed described in above-mentioned any embodiment Stingy drawing method based on neural network or neural network training method.

The computer program product can be realized especially by hardware, software or its mode combined.In an alternative embodiment In son, the computer program product is embodied as computer storage media, in another optional example, the computer Program product is embodied as software product, such as software development kit (SoftwareDevelopment Kit, SDK) etc..

In one or more optional embodiments, the embodiment of the present disclosure additionally provides another scratching based on neural network The training method and its corresponding device of drawing method and neural network and electronic equipment, computer storage media, computer program And computer program product, wherein, this method includes：First device sends the stingy figure based on neural network to second device and refers to Show or neural network trained to indicate, the instruction cause second device perform in any of the above-described possible embodiment based on nerve The stingy drawing method of network or training neural network method；First device receives the scratching based on neural network that second device is sent Figure result or neural metwork training result.

In some embodiments, the instruction of stingy figure or training neural network instruction that should be based on neural network can be specially Call instruction, first device can be indicated by way of calling second device perform stingy graphic operation based on neural network or Training neural network operation, accordingly, in response to call instruction is received, second device can perform above-mentioned based on neural network Stingy drawing method or training neural network method in any embodiment in step and/or flow.

It should be understood that the terms such as " first " in the embodiment of the present disclosure, " second " are used for the purpose of distinguishing, and be not construed as Restriction to the embodiment of the present disclosure.

It should also be understood that in the disclosure, " multiple " can refer to two or more, " at least one " can refer to one, Two or more.

It should also be understood that for the either component, data or the structure that are referred in the disclosure, clearly limited or preceding no In the case of opposite enlightenment given hereinlater, one or more may be generally understood to.

It should also be understood that the disclosure highlights the description of each embodiment the difference between each embodiment, Same or similar part can be referred to mutually, for sake of simplicity, no longer repeating one by one.

The present processes and device, electronic equipment and computer-readable storage medium may be achieved in many ways Matter.For example, can by any combinations of software, hardware, firmware or software, hardware, firmware come realize the present processes and Device, electronic equipment and computer readable storage medium.The said sequence of the step of for method merely to illustrate, The step of the present processes, is not limited to sequence described in detail above, unless specifically stated otherwise.In addition, at some In embodiment, the application can be also embodied as recording program in the recording medium, these programs include being used to implement basis The machine readable instructions of the present processes.Thus, the application also covers storage for performing the journey according to the present processes The recording medium of sequence.

The description of the present application provides for the sake of example and description, and is not exhaustively or by the application It is limited to disclosed form.Many modifications and variations are obvious for the ordinary skill in the art.It selects and retouches It states embodiment and is to more preferably illustrate the principle of the application and practical application, and enable those of ordinary skill in the art Enough understand that the embodiment of the present application can be so that design the various embodiments with various modifications for being suitable for special-purpose.

Claims

1. a kind of stingy drawing method based on neural network, which is characterized in that including：

At least by pending image and the prospect constraints graph of the pending image, as input information, it is supplied to nerve net Network carries out foreground detection, obtains foreground detection result；The prospect constraints graph is the significantly constraint of object in the pending image Information；

According to the foreground detection as a result, taking out foreground image from the pending image；

Wherein, the neural network includes：Coding network unit and decoding network element, and the coding network unit and decoding Connection mode between network element includes：

The output of last non-network layer of the coding network unit is supplied to any network layer of the decoding network unit, And/or

The output of last network layer of the coding network unit is supplied to the non-first network layer of the decoding network unit.

2. according to the method described in claim 1, it is characterized in that, the coding network unit includes：

At least two layers of convolutional layer；

The characteristic pattern of at least one layer of convolutional layer output in the coding network unit is supplied to non-the in decoding network unit One layer, and the size of characteristic pattern that the characteristic pattern of decoding network unit is supplied to be exported with the last layer of the non-first layer It is identical.

3. according to the method described in claim 2, it is characterized in that, at least one layer of convolutional layer in the coding network unit with First pre- fixed step size carries out down-sampling to the characteristic pattern of the pending image of the output of a convolutional layer thereon, is differentiated with being formed Rate reduces, and the increased characteristic pattern of quantity.

4. according to the method in any one of claims 1 to 3, which is characterized in that the network layer of the decoding network unit Including：

At least one layer of convolutional layer and at least one layer of warp lamination；

The input of at least one layer warp lamination includes：The spy of the pending image of convolutional layer output in coding network unit The output of a upper convolutional layer for sign figure and the warp lamination in decoding network unit.

5. a kind of training method of neural network, which is characterized in that the method includes：

It is concentrated from training data and obtains image pattern；

At least by described image sample and the prospect constraints graph of described image sample, information is inputted as training, is supplied to and treats Trained neural network carries out foreground detection, obtains the foreground detection result of described image sample；

Using the foreground detection result of described image sample and the prospect of described image sample constraint markup information between difference as Tutorial message, exercise supervision to the neural network to be trained study.

6. a kind of stingy map device based on neural network, which is characterized in that including：

Testing result module is obtained, at least by pending image and the prospect constraints graph of the pending image, as Information is inputted, neural network is supplied to carry out foreground detection, obtains foreground detection result；The prospect constraints graph is waited to locate to be described Manage the constraint information of notable object in image；

Take foreground image module, for according to the foreground detection as a result, taking out foreground picture from the pending image Picture；

7. a kind of training device of neural network, which is characterized in that including：

Sample module is obtained, image pattern is obtained for being concentrated from training data；

Pattern detection object module is obtained, at least by described image sample and the prospect constraints graph of described image sample, Information is inputted as training, neural network to be trained is supplied to carry out foreground detection, obtains the prospect inspection of described image sample Survey result；

Module is supervised, for constraining markup information with the prospect of the foreground detection result of described image sample and described image sample Between difference for tutorial message, exercise supervision to the neural network to be trained study.

8. a kind of electronic equipment, including：

Memory, for storing computer program；

Processor, for performing the computer program stored in the memory, and the computer program is performed, and is realized Method described in any one of the claims 1-5.

9. a kind of computer readable storage medium, is stored thereon with computer program, when which is executed by processor, Realize the method described in any one of the claims 1-5.

10. a kind of computer program, including computer instruction, when the computer instruction is run in the processor of equipment, Realize the method described in any one of the claims 1-5.