CN108231190A

CN108231190A - Handle the method for image and nerve network system, equipment, medium, program

Info

Publication number: CN108231190A
Application number: CN201711326277.3A
Authority: CN
Inventors: 陈�胜; 王哲; 石建萍
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2017-12-12
Filing date: 2017-12-12
Publication date: 2018-06-29
Anticipated expiration: 2037-12-12
Also published as: CN108231190B

Abstract

The embodiment of the present disclosure discloses a kind of processing method of image and nerve network system, equipment, medium, program, wherein, network system includes：Shared network, sorter network, segmentation network and transmission network, wherein, it shares network and is used to carry out feature extraction to pending image, obtain the initial characteristics of pending image, and initial characteristics are input to sorter network and divide network；Sorter network is used for according to initial characteristics, is carried out classification processing to pending image, is obtained the classification results of pending image；Divide network to be used for according to initial characteristics, image dividing processing is carried out to pending image, obtains the image segmentation result of pending image；Network is transmitted to be used to transmit information between sorter network and segmentation network.The embodiment of the present disclosure makes the feature after being transmitted improve the effect of current task with more characterization force information by transmitting network transmission sorter network and dividing the information between network.

Description

Handle the method for image and nerve network system, equipment, medium, program

Technical field

This disclosure relates to computer vision technique, especially a kind of method for handling image and nerve network system, equipment, Medium, program.

Background technology

With the development of computer vision technique, realized extensively by neural fusion image classification and image segmentation Using in computer vision field, image segmentation (Segmentation) refers to digital picture being subdivided into multiple images The process in region (set of pixel) (also referred to as super-pixel).The purpose of image segmentation is to simplify or change the expression shape of image Formula so that image is easier to understand and analyzes.Image segmentation is commonly used in the object and boundary (line, curve in positioning image Deng).Image classification according to the different characteristic reflected in each comfortable image information, distinguishes different classes of target Image processing method.It carries out quantitative analysis using computer to image, and each pixel in image or image or region are drawn A certain kind being classified as in several classifications, to replace the vision interpretation of people.

Invention content

A kind of image processing techniques that the embodiment of the present disclosure provides.

According to the one side of the embodiment of the present disclosure, what is provided is a kind of for handling the nerve network system of image, including：

Shared network, sorter network, segmentation network and transmission network, wherein,

The shared network is used to carry out feature extraction to pending image, obtains the initial spy of the pending image Sign, and the initial characteristics are input to the sorter network and segmentation network；

The sorter network is used for according to the initial characteristics, is carried out classification processing to the pending image, is obtained institute State the classification results of pending image；

The segmentation network is used for according to the initial characteristics, is carried out image dividing processing to the pending image, is obtained To the image segmentation result of the pending image；

The transmission network is used to transmit information between the sorter network and the segmentation network.

In another embodiment based on above system of the present invention, the transmission network is used in the sorter network Information is transmitted between second network layer of first network layer and the segmentation network.

In another embodiment based on above system of the present invention, the transmission network is used for：Described the will be derived from The fisrt feature of one network layer is transferred to second network layer, and will be transmitted from the second feature of second network layer To the first network layer, wherein, the fisrt feature is used to determine the image of the pending image by the segmentation network Segmentation result, the second feature are used to determine the classification results of the pending image by the sorter network.

In another embodiment based on above system of the present invention, the transmission network, including at least one convolutional layer, Activation primitive layer and threshold function layer；

The convolutional layer, the feature for being exported to the first network layer perform convolution operation, obtain described in corresponding to Second network layer exports the first intermediate features of feature；Convolution operation is performed to the feature of second network layer output, is obtained Corresponding to the second intermediate features of first network layer output feature；

The activation primitive layer is converted to fisrt feature, to described for carrying out type to first intermediate features Second intermediate features carry out type and are converted to second feature；

The threshold function layer is controlled for being based on threshold function between the first network layer and second network layer Information transmit.

In another embodiment based on above system of the present invention, the threshold function layer is specifically used for：

Meet the first preset condition in response to the fisrt feature, the fisrt feature is transferred to second network Layer；And/or

Meet the second preset condition in response to the second feature, the second feature is transferred to the first network Layer.

In another embodiment based on above system of the present invention, the first network layer includes：

Processing module, for the information inputted to the previous network layer for being located at the first network layer in the sorter network It is handled, obtains the first original output feature；

Fusion Module for merging the second feature and the first original output feature, obtains fusion feature.

In another embodiment based on above system of the present invention, the Fusion Module is used for original defeated by described first Go out feature to be added by element with the second feature, obtain the fusion feature.

In another embodiment based on above system of the present invention, the segmentation network includes at least one convolution with holes Layer and up-sampling layer；

The convolutional layer with holes for performing convolution operation to the initial characteristics, obtains the corresponding pending image Characteristic pattern；

The up-sampling layer, for handling the characteristic pattern, output and the pending image same size Feature vector chart.

In another embodiment based on above system of the present invention, the segmentation network further includes result determination unit, For using the cross entropy loss function of Weight, determining the image segmentation result of the pending image.

In another embodiment based on above system of the present invention, the pending image is dermoscopy detection image；

The sorter network is used to determine the classification of the corresponding skin disease of the dermoscopy detection image；

The segmentation network is used to determine the corresponding pathological regions of the dermoscopy detection image.

In another embodiment based on above system of the present invention, the transmission network is completed to the shared net What the training after training of network, the sorter network and the segmentation network obtained.

In another embodiment based on above system of the present invention, the segmentation network is completed to the shared net Training obtains after the joint training of network and the sorter network.

According to the other side of the embodiment of the present disclosure, a kind of method of processing image provided, including：

Image is handled by shared network handles and carries out feature extraction, obtains the initial characteristics of the pending image, and The initial characteristics are input to the sorter network and segmentation network；

The information that network layer exports in the sorter network and the segmentation network is mutually transmitted based on network is transmitted；

Using the sorter network, based on the initial characteristics and the information transmitted network and transmitted, wait to locate to described Reason image carries out classification processing, obtains the classification results of the pending image；

Using the segmentation network, based on the initial characteristics and the information transmitted network and transmitted, wait to locate to described It manages image and carries out image dividing processing, obtain the image segmentation result of the pending image.

In another embodiment based on the above method of the present invention, based on transmitting network by the sorter network and described The information that network layer exports in segmentation network is mutually transmitted, including：

Based on the transmission network in the first network layer of the sorter network and the second network layer of the segmentation network Between transmit information.

In another embodiment based on the above method of the present invention, based on the transmission network in the sorter network Information is transmitted between second network layer of first network layer and the segmentation network, including：

Second network layer will be transferred to from the fisrt feature of the first network layer, and will derive from described the The second feature of two network layers is transferred to the first network layer, wherein, the fisrt feature is by the segmentation network for true The image segmentation result of the fixed pending image, the second feature are used to determine the pending figure by the sorter network The classification results of picture.

In another embodiment based on the above method of the present invention, the fisrt feature that the first network layer is obtained passes Second network layer is handed to, and the second feature that second network layer is obtained is transferred to the first network layer, including：

Convolution operation is performed to the feature of first network layer output, is obtained special corresponding to second network layer output First intermediate features of sign；Convolution operation is performed to the feature of second network layer output, obtains and corresponds to first net Network layers export the second intermediate features of feature；

Type is carried out to first intermediate features and is converted to fisrt feature, type is carried out to second intermediate features It is converted to second feature；

The information between the first network layer and second network layer is controlled to transmit based on threshold function.

It is described that the first network is controlled based on threshold function in another embodiment based on the above method of the present invention Information between layer and second network layer is transmitted, including：

In another embodiment based on the above method of the present invention, passed based on the initial characteristics and the transmission network The information passed carries out classification processing to the pending image, including：

The information inputted to the previous network layer for being located at the first network layer in the sorter network is handled, and is obtained First original output feature；The information of the previous network layer input is obtained based on the initial characteristics；

The second feature and the first original output feature are merged, obtains fusion feature；

Based on the fusion feature, the classification results of the pending image are obtained.

In another embodiment based on the above method of the present invention, the second feature and described first original defeated is merged Go out feature, obtain fusion feature, including：

Described first original output feature with the second feature by element is added, obtains the fusion feature.

In another embodiment based on the above method of the present invention, passed based on the initial characteristics and the transmission network The information passed carries out image dividing processing to the pending image, including：

Convolution operation is performed to the initial characteristics, obtains the characteristic pattern of the corresponding pending image；

The characteristic pattern is handled, output and the feature vector chart of the pending image same size.

In another embodiment based on the above method of the present invention, further include：Utilize the intersection entropy loss letter of Weight Number determines the image segmentation result of the pending image.

In another embodiment based on the above method of the present invention, the pending image is dermoscopy detection image；

Using the sorter network, based on the initial characteristics and the information transmitted network and transmitted, wait to locate to described Reason image carries out classification processing, obtains the classification results of the pending image, including：

Using sorter network, the classification of the corresponding skin disease of the dermoscopy detection image is determined；

Using the segmentation network, based on the initial characteristics and the information transmitted network and transmitted, wait to locate to described It manages image and carries out image dividing processing, obtain the image segmentation result of the pending image, including：

Using the segmentation network, the corresponding pathological regions of the dermoscopy detection image are determined.

It is described to handle image progress by shared network handles in another embodiment based on the above method of the present invention Feature extraction before obtaining the initial characteristics of the pending image, further includes：

The shared network, sorter network and segmentation network are trained based on sample data, until meeting preset stopping condition； Shared network, sorter network and segmentation network after being trained；The sample data is labeled with mark classification results and mark Segmentation result.

It is described to handle image progress by shared network handles in another embodiment based on the above method of the present invention Feature extraction, before obtaining the initial characteristics of the pending image or based on transmitting network by the sorter network and described Before the information that network layer exports in segmentation network is mutually transmitted, further include：

Based on shared network, sorter network and the segmentation network after the training, the transmission is trained with reference to sample data Network.

It is described that the shared net is trained based on sample data in another embodiment based on the above method of the present invention Network, sorter network and segmentation network, including：

Under the conditions of the parameter for keeping the segmentation network is fixed, the shared network and institute are trained based on sample data State sorter network；Until meet the first preset stopping condition, shared network and sorter network after being trained；

Under the conditions of the parameter of the sorter network after keeping the training is fixed, the training is trained based on sample data Shared network afterwards and the segmentation network, until meeting the second preset stopping condition, the shared network after being trained, classification Network and segmentation network.

In another embodiment based on the above method of the present invention, the parameter of the sorter network after the training is kept Under the conditions of fixed, the shared network after the training and the segmentation network are trained based on sample data, further included：

Initialization operation is performed to the segmentation network based on the parameter in the sorter network after the training.

In another embodiment based on the above method of the present invention, further include：

Increase in the appended sample data to initial sample data of preset quantity and form the sample data.

In another embodiment based on the above method of the present invention, the appended sample data for increasing preset quantity arrive The sample data is formed in initial sample data, including：

Prestore the distance between the sample data and the initial sample data in database are calculated, is obtained based on the distance Obtain the additional data of preset quantity；The database includes at least one sample data that prestores.

In another embodiment based on the above method of the present invention, it is described calculate database in prestore sample data with The distance between described initial sample data obtains the appended sample data of preset quantity based on the distance, including：

Being averaged between sample data and at least one initial sample data that prestore in the database is calculated respectively Distance value；

It is less than or equal to preset value in response to the average distance value, using the sample data that prestores as appended sample number According to.

According to the other side of the embodiment of the present disclosure, a kind of electronic equipment provided, including processor, the processor Nerve network system including being used to handle image as described above.

According to the other side of the embodiment of the present disclosure, a kind of electronic equipment provided, including：Memory, for storing Executable instruction；

And processor, it completes to locate as described above to perform the executable instruction for communicating with the memory The method for managing image.

According to the other side of the embodiment of the present disclosure, a kind of computer storage media provided, for storing computer The instruction that can be read, described instruction are performed the method for performing processing image as described above.

According to the other side of the embodiment of the present disclosure, a kind of computer program provided, including computer-readable code, When the computer-readable code in equipment when running, the processor execution in the equipment is used to implement processing as described above The instruction of the method for image.

Another aspect according to embodiments of the present invention, a kind of computer program product provided, for storing computer Readable instruction, described instruction is performed so that computer performs the processing image described in any of the above-described possible realization method Method.

In an optional embodiment, the computer program product is specially computer storage media, at another In optional embodiment, the computer program product is specially software product, such as SDK etc..

Based on disclosure above-described embodiment provide a kind of processing image method and nerve network system, equipment, medium, Program handles image zooming-out initial characteristics by shared network handles, and the initial characteristics are common for sorter network and segmentation network The feature needed is once extracted by shared network, avoids repetition extraction feature, improves the efficiency of image procossing；Respectively The classification results and image segmentation result of pending image are exported by sorter network and segmentation network, are realized through a net Network completes the classification and segmentation to image simultaneously；Information is transmitted between sorter network and segmentation network by transmitting network, is filled Divide and sorter network is utilized and divides the relevance between network, segmentation network useful feature will be transmitted in sorter network Segmentation network is given, sorter network will be passed to sorter network useful feature in segmentation network, make the feature after being transmitted With more characterization force information, the effect of current task is improved.

Below by drawings and examples, the technical solution of the disclosure is described in further detail.

Description of the drawings

The attached drawing of a part for constitution instruction describes embodiment of the disclosure, and is used to explain together with description The principle of the disclosure.

With reference to attached drawing, according to following detailed description, the disclosure can be more clearly understood, wherein：

Fig. 1 is the structure diagram for being used to handle the nerve network system of image that the embodiment of the present disclosure provides.

Fig. 2 is the schematic flow diagram for being used to handle the method for image that the embodiment of the present disclosure provides.

Fig. 3 is the structure diagram for realizing the terminal device of the embodiment of the present application or the electronic equipment of server.

Specific embodiment

The various exemplary embodiments of the disclosure are described in detail now with reference to attached drawing.It should be noted that：Unless in addition have Body illustrates that the unlimited system of component and the positioned opposite of step, numerical expression and the numerical value otherwise illustrated in these embodiments is originally Scope of disclosure.

Simultaneously, it should be appreciated that for ease of description, the size of the various pieces shown in attached drawing is not according to reality Proportionate relationship draw.

It is illustrative to the description only actually of at least one exemplary embodiment below, is never used as to the disclosure And its application or any restrictions that use.

Technology, method and apparatus known to person of ordinary skill in the relevant may be not discussed in detail, but suitable In the case of, the technology, method and apparatus should be considered as part of specification.

It should be noted that：Similar label and letter represents similar terms in following attached drawing, therefore, once a certain Xiang Yi It is defined in a attached drawing, then in subsequent attached drawing does not need to that it is further discussed.

It should also be understood that in the embodiments of the present disclosure, " A is connect with B " can refer to that A is directly connected to B or A and B is by one Other a or multiple units/components are indirectly connected with, and the embodiment of the present disclosure does not limit this.

The embodiment of the present disclosure can be applied to computer system/server, can be with numerous other general or specialized calculating System environments or configuration operate together.Suitable for be used together with computer system/server well-known computing system, ring The example of border and/or configuration includes but not limited to：Personal computer system, server computer system, thin client, thick client Machine, hand-held or laptop devices, the system based on microprocessor, set-top box, programmable consumer electronics, NetPC Network PC, Minicomputer system, large computer system and distributed cloud computing technology environment including any of the above described system, etc..

Computer system/server can be in computer system executable instruction (such as journey performed by computer system Sequence module) general linguistic context under describe.In general, program module can include routine, program, target program, component, logic, number According to structure etc., they perform specific task or realize specific abstract data type.Computer system/server can be with Implement in distributed cloud computing environment, in distributed cloud computing environment, task is long-range by what is be linked through a communication network Manage what equipment performed.In distributed cloud computing environment, program module can be located at the Local or Remote meter for including storage device It calculates in system storage medium.

Skin disease is to be happened at the general name of skin and skin accessory organ's disease, be seriously affect people's health it is common simultaneously One of frequently-occurring disease even causes death when serious.If energy early detection simultaneously treats, can effectively alleviate and controls disease Feelings.However, due to the shortage of professional skin disease doctor's human hand, the diagnose and treat of early stage is all difficult to realize.Therefore, by calculating Machine vision technique, development automation skin disease diagnosis is to assist to improve the efficiency and objective that dermatologist diagnoses in clinical practice Property is necessary.In addition, the focal area being partitioned into automatically in medical image, which is alternatively doctor, provides significant examine Disconnected advisory opinion.

During the disclosure is realized, publisher has found, when handling skin disease detection image, utilizes classification and segmentation Potential relationship between task can improve the treatment effeciency and recognition effect of image.

It should be understood that the technical solution that the embodiment of the present disclosure provides can be mainly used in processing skin disease detection image, example Such as dermoscopy picture, but the embodiment of the present disclosure is also in other kinds of image, and the embodiment of the present disclosure does not limit this.

Fig. 1 is the structure diagram for being used to handle the nerve network system of image that the embodiment of the present disclosure provides.Such as Fig. 1 institutes Show, which includes：

Shared network 11 (FeatureNet), sorter network 12 (ClsNet), segmentation network 13 (SegNet) and transmission net Network 14 (such as：Feature Passing Module), wherein, share network 11 output terminal respectively with sorter network 12 and divide Cut the input terminal connection of network 13.Network 14 is transmitted to be connected between sorter network 12 and segmentation network 13.

In one or more alternative embodiments, share network 11 and be used to carry out feature extraction to pending image, obtain The initial characteristics of pending image, and initial characteristics are input to sorter network and segmentation network.

Optionally, pending image can be skin detection image, such as dermoscopy picture or can also be other classes The image of type, the embodiment of the present application do not limit this.

Since image classification task and segmentation task are two relevant tasks, the particularly classification of skin disease image and are divided The degree of correlation between cutting is especially high, and the present embodiment will both can apply to sorter network by shared network 11, can also apply to point The feature extraction of network is cut out as initial characteristics, by the extraction of initial characteristics, improves the efficiency of image procossing.It is optional Ground, initial characteristics may include minutia and/or edge feature, but the embodiment of the present application to the specific implementations of initial characteristics not It limits.

Sorter network 12 is used for according to initial characteristics, is carried out classification processing to pending image, is obtained pending image Classification results.Divide network 13 to be used for according to initial characteristics, image dividing processing is carried out to pending image, obtains pending figure The image segmentation result of picture.

In this way, by the sorter network in nerve network system and segmentation network, can obtain simultaneously image classification and Image segmentation result.

Optionally, sorter network and segmentation network can respectively include at least one network layer, and the embodiment of the present disclosure is to dividing The realization of class network and segmentation network is not construed as limiting.

Optionally, if pending image is specially skin detection image, which can be used to determine skin The classification of the corresponding skin disease of detection image,, then can be with for another example if there is skin disease for example whether there is skin disease Type including skin disease.Segmentation network can determine the corresponding pathological regions of skin detection image.

Network 14 is transmitted to be used to transmit information between sorter network 12 and segmentation network 13.

Shared network 11 can be to the same initial characteristics of sorter network 12 and 13 the input phase of segmentation network, and sorter network 12 can Classified with being based on the initial characteristics, segmentation network can be based on the initial characteristics and carry out image segmentation.Transmitting network 14 can To transmit information between sorter network 12 and segmentation network 13, segmentation network 13 can be with by transmitting information that network 14 transmits Network 12 is classified for classifying, the information that sorter network 12 is transmitted to segmentation network 13 by transmitting network 14 can be by Segmentation network 13 is used to carry out image segmentation, is conducive to improve the information diversity in classification and image segmentation process, improves and divide Class and the accuracy rate of image segmentation.

In one or more alternative embodiments, it is defeated to transmit a side of the network in receiving sorter network and dividing network After the information entered, the information can be transmitted to the opposing party.Wherein, it should be appreciated that being transmitted to the opposing party in the embodiment of the present disclosure The information can refer to the information is directly transferred to the opposing party or can also refer to the information is handled after be transferred to it is another One side, the embodiment of the present application do not limit this.

In one or more alternative embodiments, spy can be transmitted between sorter network and segmentation network by transmitting network 14 Sign, that is to say, that above- mentioned information can include feature.Exist for example, transmitting network 14 and segmentation network being transmitted to sorter network The one or more features that are obtained in image segmentation process are transmitted sorter network to segmentation network and are obtained in assorting process One or more features, the embodiment of the present disclosure do not limit this.

In one or more alternative embodiments, transmitting network 14 can pass to the side in sorter network and segmentation network Pass the feature of one or more network layers output that the opposing party includes.Wherein, the network layer belonging to the feature of transmission and transmission Characteristic type can be by being trained to obtain to nerve network system, the embodiment of the present application does not limit this.

In one or more alternative embodiments, transmitting network 14 can be in the first network layer of sorter network and segmentation net Information is transmitted between second network layer of network.Wherein, first network layer is a network layer in sorter network, the second network layer To divide a network layer in network, and specific first network layer and the second network layer are respectively sorter network and segmentation network Which layer can by specific tasks object and training determine, the embodiment of the present application implements it and does not limit.

As alternative embodiment, transmitting network 14 can be in two or more first network layer and two or two Transmit information between the second above network layer, and the quantity of the quantity of first network layer and the second network layer not necessarily phase Deng there may be that multiple first network layers correspond to second network layers or a first network layer corresponds to multiple second networks The situation of layer.

Based on a kind of nerve network system for being used to handle image that disclosure above-described embodiment provides, pass through and share network To pending image zooming-out initial characteristics, feature of the initial characteristics for sorter network and segmentation network common need, by altogether It enjoys network once to extract, avoids repetition extraction feature, improve the efficiency of image procossing；Pass through sorter network and segmentation respectively Network exports the classification results and image segmentation result of pending image, realizes through a network while completes to image Classification and segmentation；By transmit network sorter network and segmentation network between transmit information, take full advantage of sorter network with Divide the relevance between network, segmentation network will be passed to for segmentation network useful feature in sorter network, will divided Sorter network is passed to sorter network useful feature in network, the feature after being transmitted is made to have more characterization force information, Improve the effect of current task.

In at least one alternative embodiment, transmit network 14 and the fisrt feature that first network layer obtains is transferred to second Network layer, and the second feature that the second network layer is obtained is transferred to first network layer.

Wherein, fisrt feature is divided network for determining the image segmentation result of pending image, and second feature is divided Class network is used to determine the classification results of pending image.

At this point, optionally, initial characteristics that segmentation network can be based on shared network inputs and transmit that network transmits the One feature carries out image dividing processing to pending image.For example, segmentation network can be by fisrt feature with dividing in network Original feature carries out fusion treatment, obtains fusion feature, and obtain image segmentation result based on the fusion feature.

The similar initial characteristics that optionally, sorter network can be based on shared network inputs and the second of transmission network transmission Feature carries out classification processing to pending image.For example, sorter network can be by original spy in second feature and sorter network Sign carries out fusion treatment, obtains fusion feature, and obtain classification results based on the fusion feature.

In one or more alternative embodiments, threshold function layer can be included by transmitting network, for control tactics network Information between segmentation network is transmitted.For example, threshold function layer can be controlled between first network layer and the second network layer Feature is transmitted.

Optionally, the side in sorter network and segmentation network is input in the feature for transmitting network, only to another Square useful feature can just be passed to the opposing party, for example, fisrt feature is to divide task useful feature to image, and second It is characterized in for image classification task useful feature.In this way, by by a network branches (sorter network or segmentation network) Another network branches (segmentation network or sorter network) useful feature is screened and is added in another network branches, The feature that the useful feature of reception and original Fusion Features obtain can preferably be characterized image, and right by another network branches Current task generates positive influence, can enhance the effect of classification and segmentation simultaneously.

Optionally, whether threshold function layer can meet specified conditions according to feature, to determine whether to transmit this feature.Make For an example, threshold function layer can meet the first preset condition in response to fisrt feature, determine fisrt feature being transferred to Second network layer.As another example, threshold function layer can meet the second preset condition in response to second feature, really It is fixed that second feature is transferred to the first network layer.Wherein, optionally, the first preset condition and/or the second preset condition can To be by being obtained to the training of nerve network system, corresponding different task can obtain different default items by training Part, the embodiment of the present application do not limit the specific implementation of first preset condition and/or the second preset condition.

In this way, the feature for only meeting the first preset condition is considered as useful feature and is passed to the second network Layer, and the feature for only meeting the first preset condition is considered as useful feature and is passed to first network layer.

Optionally, threshold function layer can be based on threshold function, and the information between control tactics network and segmentation network passes It passs.Threshold function layer by threshold function can information throughput rate between 0 to 1, these threshold functions can learn, Therefore rate of information transmission can be controlled by the threshold function filter of response particular visual mode and be suitable for individual specimen.By biography Pass network, the feature of (segmentation) network of classifying passes to segmentation (classification) network by threshold function, the feature passed over and The Fusion Features of current network branch can obtain the information with more characterization power, thus can improve current task effect.

Optionally, at least one convolutional layer and activation primitive layer can be included by transmitting network.

In an optional example, convolutional layer performs convolution operation for the feature that is exported to first network layer, obtains the One intermediate features；And convolution operation is performed to the feature of the second network layer output, obtain the second intermediate features；

It specifically, can be by the size for the feature that first network layer exports and the second network by the convolution operation of convolutional layer The dimension of the feature of layer output carries out unification, in order to which subsequently the feature of two networks is merged, specifically, the dimension of feature Degree can include the length and width of characteristic pattern；Characteristic pattern can be zoomed in and out by convolution operation and realize two characteristic dimensions Unification.

Optionally, activation primitive layer can carry out type to the first intermediate features and be converted to fisrt feature, and to second Intermediate features carry out type and are converted to second feature.

Sigmoid functions may be used in activation primitive layer or other kinds of activation primitive is realized.Due to sorter network and Segmentation network corresponds to different tasks, converts the characteristics of image of transmission therefore, it is necessary to activation primitive.

In one or more alternative embodiments, first network layer receive transmit network transmit second feature it Afterwards, can second feature and original feature of the first network layer be subjected to fusion treatment, obtains fusion feature, and export and be somebody's turn to do Fusion feature.

Optionally, original feature here can be in the case where not receiving and transmitting the information that network transmits, this The feature of one network layer input either the feature (i.e. original output feature) of first network layer output or can also be this Intermediate features that one network layer obtains during output feature is obtained, etc., the embodiment of the present application does not limit this It is fixed.

In an optional example, first network layer includes：Processing module, for first network layer in sorter network The information of previous network layer input is handled, and obtains the first original output feature；Fusion Module, for merge second feature and First original output feature, obtains fusion feature.

Specifically, processing module can to the input information of other layers in sorter network (such as：Input feature vector) at Reason obtains the first original output feature, i.e., the feature obtained in the case of the information for not considering to transmit network transmission.It is optional Ground, the information of the previous network layer input can be the features of the previous network layer output of the first network layer in sorter network, Optionally, if the first network layer is the first layer of sorter network, the information of the previous network layer input can also be altogether The initial characteristics of network output are enjoyed, but the embodiment of the present application does not limit this.Fusion Module can merge the first original output Feature and second feature obtain fusion feature, and export the fusion feature.In this way, transmit the second of network transmission receiving After feature, the output of first network layer becomes fusion feature by the first original output feature, wherein, the fusion feature is one In a or multiple optional examples, the first original output feature is added by Fusion Module with second feature by element, and it is special to obtain fusion Sign.

By being added by element, element can include characteristic value or feature vector etc. in the embodiment, by corresponding position Feature is added, and the fusion feature of acquisition is also presented in segmentation network and classification is appointed in addition to the feature of embodiment sorter network acquisition Business useful feature realizes and generates positive influence to classification task by segmentation task, enhances the effect of sorter network.

Similarly, divide the second network layer in network after fisrt feature is received, it can also be to the fisrt feature Fusion treatment is carried out with original feature of the second network layer, obtains fusion feature.Optionally, second network layer is to fisrt feature Specific processing can be similar with first network layer, which is not described herein again.

In an optional example, the fusion feature of first network layer and the output of the second network layer can be by the following formula (1) it determines：

Wherein, x_clsAnd x_segRepresent respectively sorter network be input to first network layer information and segmentation network inputs arrive The information of second network layer,WithOriginal output of first network layer and the second network layer is represented respectively, i.e., based on x_cls And x_segObtained output.WithThe new of first network layer and the second network layer when network is transmitted in addition is represented respectively Output.G_seg2cls(G_cls2seg) represent controlling featureClassification (segmentation) network is transferred to from segmentation (classification) network Threshold function.Sig and σ is Sigmoid functions and ReLU functions respectively.It is convolution operation, and ω and b are the ginsengs of convolution kernel Number, op here represent the operation being added by element.The volume of corresponding segmentation network is represented respectively Accumulate the offset that the weight weight of core, the weight weight of the convolution kernel of corresponding sorter network, correspondence divide the convolution kernel of network Measure the offset bias of bias, the convolution kernel for corresponding to sorter network.

It should be understood that above-mentioned realization method, which is only intended to illustration first network layer and the second network layer, is receiving transmission How to be handled to obtain output feature after the information that network transmits, wherein, for sake of simplicity, having ignored BN letters in formula (1) Number, in the embodiment of the present application, first network layer and the second network layer can also obtain output feature by other processing modes, The embodiment of the present application does not limit this.

In an optional example, the latter half of Resnet-101 networks may be used in sorter network, at this point, corresponding Part before Resnet-101 networks may be used in shared network (before residual error module conv4_10)；The sorter network is in addition to last The output of one layer of full articulamentum is changed to skin disease species number.The sorter network is connected to after shared network (FeatureNet), C dimensional vectors (c represents classification) are exported, probability vector is converted by normalization operation, represents each dermopathic possibility, are led to Cross normalization (such as：Softmax) operation makes classification results be converted into Probability Forms.

Optionally, for the task of extraction skin lens image, shared network (FeatureNet) can be selected from deep remnants Network portion (resnet-101, until conv4 10 pieces).Shared network includes one group of residual error module, wherein each residual error mould Block is superimposed by several network layers and obtained, wherein, network layer can include：Convolutional layer, mass (BN) layer and hot Shandong layer etc..Every Additional jump connection is introduced in a residual error module, to improve information flow, and largely alleviates gradient disappearance problem.

In an optional embodiment of the disclosure, segmentation network includes at least one convolutional layer (dilated with holes Convolution) and up-sampling layer.Convolutional layer with holes can be used for performing convolution operation to initial characteristics, obtains correspondence and waits to locate Manage the characteristic pattern of image.The up-sampling filter that convolution with holes is obtained using zero is inserted between two continuous convolution kernel values Convolution is carried out, under identical design conditions, convolution with holes provides the receptive field of bigger.The ratio of zero is inserted by control Value is adaptively modified the receptive field of wave filter.In this way, the denser output figure of higher resolution can be obtained, for object Edge details processing it is more preferable, thus result is more accurate.

Up-sampling layer can be used for handling characteristic pattern, output and the feature of the pending image same size to Spirogram, and feature based vectogram obtains the differentiation result of each pixel in corresponding pending image.The segmentation network connection After shared network (FeatureNet), output can be the score of c × h/8 × w/8 (h and w represent the height and width of artwork) Figure, the size identical with pending image is restored to by up-sampling, so as to produce a prediction to each pixel, Remain spatial information in pending image simultaneously, finally carried out on the score characteristic pattern of up-sampling prospect pixel-by-pixel, Background class, wherein, therefore the part that foreground pixel representative need to be partitioned into, can be realized pending based on prospect, background class The segmentation of image.

Optionally, segmentation network structure can copy deeplab-ResNet101 to be built.

In one or more alternative embodiments, it is contemplated that the depth injustice that may be present between foreground and background classification Weighing apparatus, segmentation network can result determination unit utilize the cross entropy loss function of Weight, determine the image point of pending image Cut result.

In an optional example, shown in the formula such as formula (2) of the cross entropy loss function of Weight：

Wherein, X is input picture, y_i∈ { 0,1 }, j=| 1 ..., X |, j is single pixel (pixel-wise) binary system of X Label, and Y₊And Y_-It has indicated whether just to mark and negative flag pixel, weight beta is equal to foreground pixel point and the number of background pixel point Amount ratio, i.e. β=| Y_-|/|Y₊|, P () in output layer using Sigmoid functions by being obtained.

Optionally, other forms can also be taken by dividing the loss function of network, and the embodiment of the present application is not construed as limiting this.

Optionally, in a specific example of the disclosure, pending image is dermoscopy detection image；Dermoscopy is also known as Epidermis light transmission microscope, the English name and alias of dermoscopy include Dermatoscope, Dermoscope, Epiluminescence Microscope(ELM)、Incident light microscope、Skin surface microscope.At this point, optionally, sorter network is used to determine the classification of the corresponding skin disease of dermoscopy detection image； In the embodiment of specific detection skin disease, the classification of skin disease can be pre-set.For example, n classification can be set, it is defeated The feature vector gone out can include n+1 value, wherein preceding n can correspond to a kind of skin disease respectively, the last one is corresponded to without skin Skin disease.Optionally, segmentation network is used to determine the corresponding pathological regions of dermoscopy detection image.In point of dermoscopy detection image It cuts in task, divides the correlation higher of task and classification task, therefore, the feature energy of segmentation network integration sorter network transmission Preferably determining foreground area (pathological regions).

In one or more alternative embodiments, transmitting network is completed to shared network, sorter network and segmentation net What the training after training of network obtained.

Specifically, it transmits to ensure to transmit network between sorter network and segmentation network and is characterized in useful feature, It needs to share network, sorter network and divide network transmission network integration to train jointly, to realize transmission useful feature, shielding Useless feature.

In one or more alternative embodiments, segmentation network is to complete to instruct the joint for sharing network and sorter network Training obtains after practicing.

In a specific example, in the shared network of training, sorter network and segmentation network, it is not added with feature and transmits mould Block, the parameter Weight Training network other parts of fixed segmentation network extremely restrain.With sorter network weight initialization at this time point Cut branch's (different part random initializtion of the two)；Fixed cluster network, training network other parts to convergence.

Fig. 2 is the schematic flow diagram of the method for processing image that the embodiment of the present disclosure provides.As shown in Fig. 2, the embodiment Method is applied to nerve network system, and nerve network system includes shared network, sorter network, segmentation network and transmits network.

The embodiment method includes：

Step 201, it handles image by shared network handles and carries out feature extraction, obtain the initial spy of pending image Sign.

Optionally, pending image can be skin detection image, such as dermoscopy picture or can also be other classes The image of type, the embodiment of the present application do not limit this.Since image classification task and segmentation task are two relevant tasks, The degree of correlation particularly between the classification and segmentation of skin disease image is especially high, and the present embodiment can be applied by shared network In sorter network, the feature extraction that can also apply to segmentation network out as initial characteristics, by the extraction of initial characteristics, carry The high efficiency of image procossing.Optionally, initial characteristics may include minutia and/or edge feature, but the application is implemented Example does not limit the specific implementation of initial characteristics.

Step 202, the information that network layer exports in sorter network and segmentation network is mutually transmitted based on transmission network.

Specifically, transmit network can sorter network and segmentation network between transfer characteristic, that is to say, that above- mentioned information It can include feature.For example, segmentation network obtains in image segmentation process one can be transmitted to sorter network by transmitting network A or multiple features or the one or more features obtained in assorting process to segmentation network transmission sorter network, this public affairs Embodiment is opened not limit this.

Optionally, transmit that network can transmit that the opposing party includes to the side in sorter network and segmentation network one or The feature of multiple network layers output.Wherein, the network layer belonging to the feature of transmission and the characteristic type of transmission can be by right Nerve network system is trained to obtain, and the embodiment of the present application does not limit this.

Step 203, using sorter network, based on initial characteristics and transmit the information that network transmits, to pending image into Row classification is handled, and obtains the classification results of pending image.

Step 204, using dividing network, based on initial characteristics and transmit the information that network transmits, to pending image into Row image dividing processing, obtains the image segmentation result of pending image.

There is no sequencings wherein between operation 203 and operation 204, can first carry out operation 203 and redo 204 or first carry out operation 204 redo 203 or be performed simultaneously operation 203 and operation 203.

Method based on a kind of processing image that disclosure above-described embodiment provides handles image by shared network handles Initial characteristics are extracted, feature of the initial characteristics for sorter network and segmentation network common need is once carried by shared network It takes, avoids repetition extraction feature, improve the efficiency of image procossing；It exports by sorter network and segmentation network respectively and waits to locate The classification results and image segmentation result of image are managed, are realized through the classification and segmentation of a network while completion to image； Information is transmitted between sorter network and segmentation network by transmitting network, is taken full advantage of between sorter network and segmentation network Relevance, will in sorter network for segmentation network useful feature pass to segmentation network, will segmentation network in classification Network useful feature passes to sorter network, and the feature after being transmitted is made to be improved with more characterization force information as predecessor The effect of business.

In one or more alternative embodiments, operation 202 can specifically include：

Information is transmitted between the second network layer of the first network layer of sorter network and segmentation network based on transmission network.

Wherein, first network layer is a network layer in sorter network, and the second network layer is one divided in network Network layer, and specifically first network layer and the second network layer are respectively that sorter network can be by specific with which layer for dividing network Task object and training determine that the embodiment of the present application implements it and do not limit.

Network is transmitted after the information of the side input in receiving sorter network and dividing network, it can be to the opposing party Transmit the information.Wherein, it should be appreciated that transmitting the information to the opposing party and can refer in the embodiment of the present disclosure directly passes the information Be handed to the opposing party or can also refer to the information is handled after be transferred to the opposing party, the embodiment of the present application does not do this It limits.

As alternative embodiment, transmit network can two or more first network layers and two or two with On the second network layer between transmit information, and the quantity of the quantity of first network layer and the second network layer is not necessarily equal, There may be multiple first network layers one the second network layer of correspondence or a first network layer corresponds to multiple second network layers Situation.

In one or more alternative embodiments, based on transmission network in the first network layer of sorter network and segmentation network The second network layer between transmit information, including：

The second network layer will be transferred to, and will be from the of the second network layer from the fisrt feature of first network layer Two features are transferred to first network layer, wherein, fisrt feature is divided network for determining that the image of pending image divides knot Fruit, second feature are classified network for determining the classification results of pending image.

Optionally, threshold function layer can be based on threshold function, and the information between control tactics network and segmentation network passes It passs.

In at least one alternative embodiment, the fisrt feature that first network layer obtains is transferred to the second network layer, and The second feature that second network layer obtains is transferred to first network layer, including：

Convolution operation is performed to the feature of first network layer output, obtains and corresponds to the first of the second network layer output feature Intermediate features；Convolution operation is performed to the feature of the second network layer output, obtains and corresponds to the of first network layer output feature Two intermediate features.

Type is carried out to the first intermediate features and is converted to fisrt feature, carrying out type to the second intermediate features is converted to Second feature.

Specifically, type conversion be to be realized by activation primitive, activation primitive may be used Sigmoid functions or other The activation primitive of type is realized.Due to sorter network with segmentation network corresponds to different tasks, therefore, it is necessary to activation primitive will biography The characteristics of image passed is converted.

The information between first network layer and the second network layer is controlled to transmit based on threshold function.

Threshold function can so that these threshold functions can learn information throughput rate, therefore information between 0 to 1 Transport can be controlled by the threshold function filter of response particular visual mode and be suitable for individual specimen.By transmit network, The feature of classification (segmentation) network passes to segmentation (classification) network, the feature and current network passed over by threshold function The Fusion Features of branch can obtain the information with more characterization power, thus can improve current task effect.

In one or more alternative embodiments, operation 203 includes：

The information inputted to the previous network layer for being located at first network layer in sorter network is handled, and it is original to obtain first Export feature；The information of previous network layer input is obtained based on initial characteristics；

Second feature and the first original output feature are merged, obtains fusion feature；Based on fusion feature, pending figure is obtained The classification results of picture.

Specifically, the input information (such as input feature vector) of other layers in sorter network can be handled, obtains One original output feature, i.e., the feature obtained in the case of the information for not considering to transmit network transmission.Optionally, the previous net The information of network layers input can be the feature of the previous network layer output of the first network layer in sorter network, optionally, if The first network layer is the first layer of sorter network, then the information of the previous network layer input can also be that shared network exports Initial characteristics, but the embodiment of the present application does not limit this.Fusion Module can merge the first original output feature and the second spy Sign, obtains fusion feature, and export the fusion feature.In this way, after the second feature for transmitting network transmission is received, first The output of network layer becomes fusion feature by the first original output feature, wherein, the fusion feature is one or more optional In example, the first original output feature is added by Fusion Module with second feature by element, obtains fusion feature.

In one or more alternative embodiments, operation 204 includes：

Convolution operation is performed to initial characteristics, obtains the characteristic pattern of corresponding pending image；

Characteristic pattern is handled, output and the feature vector chart of pending image same size.

Operation 204 is by dividing real-time performance, and specifically, segmentation network includes at least one convolutional layer (dilated with holes Convolution) and up-sampling layer.Convolutional layer with holes can be used for performing convolution operation to initial characteristics, obtains correspondence and waits to locate Manage the characteristic pattern of image.The up-sampling filter that convolution with holes is obtained using zero is inserted between two continuous convolution kernel values Convolution is carried out, under identical design conditions, convolution with holes provides the receptive field of bigger.The ratio of zero is inserted by control Value is adaptively modified the receptive field of wave filter.In this way, the denser output figure of higher resolution can be obtained, for object Edge details processing it is more preferable, thus result is more accurate.

In one or more alternative embodiments, it is contemplated that the depth injustice that may be present between foreground and background classification Weighing apparatus utilizes the cross entropy loss function of Weight in operation 204, determine the image segmentation result of pending image.

In an optional example, the formula of the cross entropy loss function of Weight can as shown in formula above (2), but The embodiment of the present application is without being limited thereto.

Optionally, in a specific example of the disclosure, pending image is dermoscopy detection image；Dermoscopy is also known as Epidermis light transmission microscope, the English name and alias of dermoscopy include Dermatoscope, Dermoscope, Epiluminescence Microscope(ELM)、Incident light microscope、Skin surface microscope。

At this point, optionally, operation 203 includes：Using sorter network, the corresponding skin disease of dermoscopy detection image is determined Classification；In the specifically embodiment of detection skin disease, the classification of skin disease can be pre-set.For example, n can be set A classification, the feature vector of output can include n+1 value, wherein preceding n can correspond to a kind of skin disease respectively, the last one It corresponds to without skin disease.

Optionally, operation 204 includes：Using network is divided, the corresponding pathological regions of dermoscopy detection image are determined.In skin In the segmentation task of skin microscopy altimetric image, divide the correlation higher of task and classification task, therefore, segmentation network integration classification The feature of network transmission can preferably determine foreground area (pathological regions).

In one or more alternative embodiments, before operation 201, further include：

Shared network, sorter network and segmentation network are trained based on sample data, until meeting preset stopping condition；It obtains Shared network, sorter network and segmentation network after training；Sample data is labeled with mark classification results and marks segmentation result.

Specifically, shared network, sorter network and segmentation network are trained based on sample data, including：

Under the conditions of the parameter for keeping segmentation network is fixed, shared network and sorter network are trained based on sample data； Until meet the first preset stopping condition, shared network and sorter network after being trained；

Under the conditions of the parameter of the sorter network after keeping training is fixed, based on shared after sample data training training Network and segmentation network, until meeting the second preset stopping condition, shared network, sorter network and segmentation net after being trained Network.

In one or more alternative embodiments, under the conditions of the parameter of the sorter network after keeping training is fixed, base Shared network and segmentation network after sample data training training, further include：

Initialization operation is performed to segmentation network based on the parameter in the sorter network after training.

Specifically, for sorter network and segmentation network not corresponding part stochastic parameter initialize.

Before training network, Data flipping and superposition Gaussian noise can also be carried out to sample data, to obtain more Mostly and more fully sample data more fully trains network with realizing.

In one or more alternative embodiments, before operation 201 or before operation 202, further include：

Based on shared network, sorter network and the segmentation network after training, trained with reference to sample data and transmit network.

In one or more alternative embodiments, in order to which preferably network, sorter network, segmentation network and biography are shared in training Network is passed, in training during each iteration, increases in the appended sample data to initial sample data of preset quantity and forms sample Data.Network capabilities is made preferably to be trained by increasing attachment sample data.Specifically, the selection of attachment sample data is By calculating prestore the distance between sample data and initial sample data in database, preset quantity is obtained based on distance Additional data.

Specifically, database includes at least one sample data that prestores.It can also include other images in the database, have May major part image include very noisy, if random selection prestores sample data as attachment sample data in the database, Noise data will cause the effect of network training to decline.Therefore, it is proposed to a kind of selection method of attachment sample data, i.e., logical It crosses calculating to prestore the distance between sample data and initial sample data, determines which prestores sample data and initial sample data Similar, network training effect can be improved by adding in the appended sample data similar with initial sample data.

In one or more optional embodiments, calculate in database prestore sample data and initial sample data it Between distance, based on distance obtain preset quantity appended sample data, including：

The average distance value to prestore between sample data and at least one initial sample data in database is calculated respectively；

It is less than or equal to preset value in response to average distance value, the sample data that will prestore is as appended sample data.

Specifically, the sample data that can be prestored by convolutional calculation and initial sample data acquisition prestore sample characteristics and just Beginning sample characteristics specifically can be used with pretrained VGG real-time performance convolution operations, the sample characteristics that prestore based on acquisition And initial sample characteristics calculate each prestore between sample data and initial sample data average distance value (such as：Feature it Between COS distance).When average distance value is less than or during preset value, it is believed that prestore sample data and the initial sample data phase Seemingly, it is believed that the sample data that prestores is suitable for current task, by calculating all sample data and initial sample datas of prestoring Average distance value, multiple sample datas that prestore of the condition of satisfaction will be obtained from database, these meet the pre- storage sample of condition Notebook data can be used as appended sample data, specifically, can all appended sample data once all be added in initial sample number According to；It can also divide several attachment sample data is added in initial sample data；It can also be limited according to quantity, each iteration adds in The appended sample data of preset quantity.

One of ordinary skill in the art will appreciate that：Realizing all or part of step of above method embodiment can pass through The relevant hardware of program instruction is completed, and aforementioned program can be stored in computer read/write memory medium, which exists During execution, step including the steps of the foregoing method embodiments is performed；And aforementioned storage medium includes：ROM, RAM, magnetic disc or CD Etc. the various media that can store program code.

According to the one side of the embodiment of the present disclosure, a kind of electronic equipment provided, including processor, processor includes this The nerve network system for being used to handle image of any of the above-described embodiment is disclosed.

According to the one side of the embodiment of the present disclosure, a kind of electronic equipment provided, including：Memory, can for storing Execute instruction；

And processor, for communicating with memory the side of disclosure processing image is completed to perform executable instruction Any of the above-described embodiment of method.

According to the one side of the embodiment of the present disclosure, a kind of computer storage media provided can for storing computer The instruction of reading, instruction are performed any of the above-described embodiment of method for performing disclosure processing image.

According to the one side of the embodiment of the present disclosure, a kind of computer program provided, including computer-readable code, when For computer-readable code when being run in equipment, the processor execution in the equipment is used to implement any of the above-described embodiment of the disclosure Handle the instruction of the method for image.

In one or more optional embodiments, the embodiment of the present disclosure additionally provides a kind of computer program product, uses In storage computer-readable instruction, which is performed so that being handled in any of the above-described possible realization method of computer execution The method of image.

The computer program product can be realized especially by hardware, software or its mode combined.In an alternative embodiment In son, the computer program product is embodied as computer storage media, in another optional example, the computer Program product is embodied as software product, such as software development kit (Software Development Kit, SDK) etc..

In one or more optional embodiments, the embodiment of the present invention additionally provide it is a kind of handle image and nerve net Network system, electronic equipment, computer storage media, computer program and computer program product, wherein, this method includes： Image is handled by shared network handles and carries out feature extraction, obtains the initial characteristics of pending image；It will based on network is transmitted The information that network layer exports in sorter network and segmentation network is mutually transmitted；Using sorter network, based on initial characteristics and transmission The information that network transmits, carries out classification processing to pending image, obtains the classification results of pending image；Using dividing net Network based on the information that initial characteristics and transmission network transmit, carries out image dividing processing to pending image, obtains pending figure The image segmentation result of picture.

In some embodiments, human body critical point detection instruction can be specially call instruction, and first device can lead to It crosses the mode called and indicates that second device performs the processing of image, accordingly, in response to call instruction is received, second device can To perform step and/or flow in any embodiment in the method for above-mentioned processing image.

The embodiment of the present disclosure additionally provides a kind of electronic equipment, such as can be mobile terminal, personal computer (PC), put down Plate computer, server etc..Below with reference to Fig. 3, it illustrates suitable for being used for realizing the terminal device of the embodiment of the present application or service The structure diagram of the electronic equipment 300 of device：As shown in figure 3, computer system 300 includes one or more processors, communication Portion etc., one or more of processors are for example：One or more central processing unit (CPU) 301 and/or one or more Image processor (GPU) 313 etc., processor can according to the executable instruction being stored in read-only memory (ROM) 302 or From the executable instruction that storage section 308 is loaded into random access storage device (RAM) 303 perform various appropriate actions and Processing.Communication unit 312 may include but be not limited to network interface card, and the network interface card may include but be not limited to IB (Infiniband) network interface card.

Processor can communicate with read-only memory 302 and/or random access storage device 330 to perform executable instruction, It is connected by bus 304 with communication unit 312 and is communicated through communication unit 312 with other target devices, is implemented so as to complete the application The corresponding operation of any one method that example provides for example, handling image by shared network handles carries out feature extraction, is treated Handle the initial characteristics of image；The information that network layer exports in sorter network and segmentation network is mutually passed based on network is transmitted It passs；Using sorter network, based on the information that initial characteristics and transmission network transmit, classification processing is carried out to pending image, is obtained To the classification results of pending image；Using network is divided, based on the information that initial characteristics and transmission network transmit, to pending Image carries out image dividing processing, obtains the image segmentation result of pending image.

In addition, in RAM 303, it can also be stored with various programs and data needed for device operation.CPU301、ROM302 And RAM303 is connected with each other by bus 304.In the case where there is RAM303, ROM302 is optional module.RAM303 is stored Executable instruction is written in executable instruction into ROM302 at runtime, and it is above-mentioned logical that executable instruction performs processor 301 The corresponding operation of letter method.Input/output (I/O) interface 305 is also connected to bus 304.Communication unit 312 can be integrally disposed, It may be set to be with multiple submodule (such as multiple IB network interface cards), and in bus link.

I/O interfaces 305 are connected to lower component：Importation 306 including keyboard, mouse etc.；It is penetrated including such as cathode The output par, c 307 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 308 including hard disk etc.； And the communications portion 309 of the network interface card including LAN card, modem etc..Communications portion 309 via such as because The network of spy's net performs communication process.Driver 310 is also according to needing to be connected to I/O interfaces 305.Detachable media 311, such as Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 310, as needed in order to be read from thereon Computer program be mounted into storage section 308 as needed.

Need what is illustrated, framework as shown in Figure 3 is only a kind of optional realization method, can root during concrete practice The component count amount and type of above-mentioned Fig. 3 are selected, are deleted, increased or replaced according to actual needs；It is set in different function component Put, can also be used it is separately positioned or integrally disposed and other implementations, such as GPU and CPU separate setting or can be by GPU collection Into on CPU, communication unit separates setting, can also be integrally disposed on CPU or GPU, etc..These interchangeable embodiments Each fall within protection domain disclosed in the disclosure.

Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description Software program.For example, embodiment of the disclosure includes a kind of computer program product, it is machine readable including being tangibly embodied in Computer program on medium, computer program are included for the program code of the method shown in execution flow chart, program code It may include the corresponding instruction of corresponding execution method and step provided by the embodiments of the present application, handled for example, passing through shared network handles Image carries out feature extraction, obtains the initial characteristics of pending image；It will be in sorter network and segmentation network based on network is transmitted The information of network layer output is mutually transmitted；Using sorter network, based on the information that initial characteristics and transmission network transmit, place is treated Reason image carries out classification processing, obtains the classification results of pending image；Using network is divided, based on initial characteristics and net is transmitted The information that network transmits carries out image dividing processing to pending image, obtains the image segmentation result of pending image.In this way Embodiment in, which can be downloaded and installed from network by communications portion 309 and/or from detachable Medium 311 is mounted.When the computer program is performed by central processing unit (CPU) 301, perform and limited in the present processes Fixed above-mentioned function.

Disclosed method and device, equipment may be achieved in many ways.For example, software, hardware, firmware can be passed through Or any combinations of software, hardware, firmware realize disclosed method and device, equipment.The step of for method Sequence is stated merely to illustrate, the step of disclosed method is not limited to sequence described in detail above, unless with other Mode illustrates.In addition, in some embodiments, the disclosure can be also embodied as recording program in the recording medium, this A little programs include being used to implement the machine readable instructions according to disclosed method.Thus, the disclosure also covers storage for holding The recording medium gone according to the program of disclosed method.

The description of the disclosure provides for the sake of example and description, and is not exhaustively or by the disclosure It is limited to disclosed form.Many modifications and variations are obvious for the ordinary skill in the art.It selects and retouches Embodiment is stated and be the principle and practical application in order to more preferably illustrate the disclosure, and those of ordinary skill in the art is enable to manage The disclosure is solved so as to design the various embodiments with various modifications suitable for special-purpose.

Claims

1. a kind of nerve network system for being used to handle image, which is characterized in that including：

The shared network is used to carry out feature extraction to pending image, obtains the initial characteristics of the pending image, and The initial characteristics are input to the sorter network and segmentation network；

The sorter network is used to, according to the initial characteristics, classification processing is carried out to the pending image, obtains described treat Handle the classification results of image；

The segmentation network is used for according to the initial characteristics, is carried out image dividing processing to the pending image, is obtained institute State the image segmentation result of pending image；

2. system according to claim 1, which is characterized in that the transmission network is used for the first of the sorter network Information is transmitted between network layer and the second network layer of the segmentation network.

3. system according to claim 2, which is characterized in that the transmission network is used for：First net will be derived from The fisrt feature of network layers is transferred to second network layer, and will be transferred to institute from the second feature of second network layer First network layer is stated, wherein, the fisrt feature is used to determine that the image of the pending image to be divided by the segmentation network As a result, the second feature is used to determine the classification results of the pending image by the sorter network.

4. system according to claim 3, which is characterized in that the transmission network, including at least one convolutional layer, activation Function layer and threshold function layer；

The convolutional layer, the feature for being exported to the first network layer perform convolution operation, obtain and correspond to described second Network layer exports the first intermediate features of feature；Convolution operation is performed to the feature of second network layer output, is corresponded to In the second intermediate features of first network layer output feature；

The activation primitive layer is converted to fisrt feature, to described second for carrying out type to first intermediate features Intermediate features carry out type and are converted to second feature；

The threshold function layer, for controlling the letter between the first network layer and second network layer based on threshold function Breath transmits.

5. system according to claim 4, which is characterized in that the threshold function layer is specifically used for：

Meet the first preset condition in response to the fisrt feature, the fisrt feature is transferred to second network layer；With/ Or

A kind of 6. method for handling image, which is characterized in that applied to nerve network system, the nerve network system is included altogether It enjoys network, sorter network, segmentation network and transmits network, the method includes：

Image is handled by shared network handles and carries out feature extraction, obtains the initial characteristics of the pending image；

Using the sorter network, based on the initial characteristics and the information transmitted network and transmitted, to the pending figure As carrying out classification processing, the classification results of the pending image are obtained；

Using the segmentation network, based on the initial characteristics and the information transmitted network and transmitted, to the pending figure As carrying out image dividing processing, the image segmentation result of the pending image is obtained.

7. a kind of electronic equipment, which is characterized in that including processor, the processor includes claim 1 to 5 any one institute That states is used to handle the nerve network system of image.

8. a kind of electronic equipment, which is characterized in that including：Memory, for storing executable instruction；

And processor, for communicating to perform the executable instruction so as to complete described in claim 6 with the memory The method for handling image.

9. a kind of computer storage media, for storing computer-readable instruction, which is characterized in that described instruction is performed When perform claim requirement 6 it is described processing images methods.

10. a kind of computer program, including computer-readable code, which is characterized in that when the computer-readable code is being set During standby upper operation, the processor execution in the equipment is used to implement the instruction for the method that image is handled described in claim 6.