CN109344839A - Image processing method and device, electronic equipment, storage medium, program product - Google Patents
Image processing method and device, electronic equipment, storage medium, program product Download PDFInfo
- Publication number
- CN109344839A CN109344839A CN201810892869.XA CN201810892869A CN109344839A CN 109344839 A CN109344839 A CN 109344839A CN 201810892869 A CN201810892869 A CN 201810892869A CN 109344839 A CN109344839 A CN 109344839A
- Authority
- CN
- China
- Prior art keywords
- capsule
- characteristic
- main
- data
- feature data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the present application discloses a kind of image processing method and device, electronic equipment, storage medium, program product, wherein, method includes: to carry out feature extraction processing to image to be processed by least two first capsules, obtain at least two fisrt feature data, wherein, each characteristic includes multiple vectors at least two characteristic；Based on at least two fisrt feature data, the main characteristic and side characteristic of the second capsule are determined, wherein network layer belonging to second capsule is located at after the network layer of at least two first capsule；The main characteristic and the side characteristic are handled by second capsule, obtain second feature data；Based on the second feature data, processing result image is obtained, reduces the complexity of image procossing, improves image procossing performance.
Description
Technical field
This application involves computer vision technique, especially a kind of image processing method and device, electronic equipment, storage are situated between
Matter, program product.
Background technique
Neural network has achieved extensive use and development in terms of image procossing.In the recent period, researcher proposes capsule
Network application is in image procossing, and in capsule network, neuron is substituted by capsule, and capsule can be indicated with vector, and general
Logical neural network is compared, and capsule network has better image process performance.
Summary of the invention
The embodiment of the present application provides a kind of image processing techniques.
According to the one aspect of the embodiment of the present application, a kind of image processing method is provided, comprising:
Feature extraction processing is carried out to image to be processed by least two first capsules, obtains at least two fisrt feature
Data, wherein each characteristic includes multiple vectors at least two characteristic；
Based on at least two fisrt feature data, the main characteristic and side characteristic of the second capsule are determined,
In, network layer belonging to second capsule is located at after the network layer of at least two first capsule；
The main characteristic and the side characteristic are handled by second capsule, obtain second feature
Data；
Based on the second feature data, processing result image is obtained.
Optionally, described to be based on at least two fisrt feature data, determine the main characteristic of the second capsule and auxiliary
Help characteristic, comprising:
By the fisrt feature that position main capsule corresponding with second capsule obtains at least two first capsule
Main characteristic of the data as second capsule；
First obtained based at least one side capsule at least two first capsule in addition to the main capsule
Characteristic determines the side characteristic of second capsule.
Optionally, described at least one side capsule based at least two first capsule in addition to the main capsule
Obtained fisrt feature data determine the side characteristic of second capsule, comprising:
Process of convolution is carried out to the fisrt feature data that at least one described side capsule obtains, obtains second capsule
Side characteristic.
It is optionally, described that the main characteristic and the side characteristic are handled by second capsule,
Obtain second feature data, comprising:
Based on the main characteristic and the side characteristic, the input data of second capsule is determined；
The input data is handled by second capsule, obtains second feature data.
Optionally, described to be based on the main characteristic and the side characteristic, determine the input of second capsule
Data, comprising:
It is special based on the main characteristic, the first weight of the main characteristic, the side characteristic and the side
The second weight for levying data, obtains the input data of second capsule.
Optionally, first weight and second weight are obtained by training.
Optionally, the method utilizes capsule network implementations, and the capsule network includes at least two network layers, Mei Gesuo
Stating network layer includes at least one capsule；
It is described that feature extraction processing is carried out to image to be processed by least two first capsules, obtain at least two first
Before characteristic, further includes:
Sample image is inputted into the capsule network；
Characteristic distance between two characteristic patterns based on two at least two network layer network layer output,
Obtain first-loss；
Based on the first-loss training capsule network.
Optionally, the sample image has mark processing result；
Before the capsule network based on first-loss training, further includes:
The sample image is handled based on the capsule network, obtains prediction processing result；
Based on the prediction processing result and the mark processing result, the second loss is determined；
It is described that the capsule network is trained based on the first-loss, comprising:
Based on the first-loss and the second loss training capsule network.
Optionally, between two characteristic patterns based on two at least two network layer network layer output
Characteristic distance, obtain first-loss, comprising:
The first predicted characteristics figure and the second prediction for obtaining two network layers output at least two network layer are special
Sign figure, the first predicted characteristics figure include the first predicted characteristics data of at least one capsule output, and second prediction is special
Sign figure includes the second predicted characteristics data of at least one capsule output；
Up-sampling operation is carried out to the second predicted characteristics figure, the second predicted characteristics figure and institute after making the up-sampling
The first predicted characteristics figure is stated to match；
Based on the characteristic distance between the second predicted characteristics figure and the first predicted characteristics figure after the up-sampling, really
The fixed first-loss.
It is optionally, described to be based on the first-loss and the second loss training capsule network, comprising:
By the first-loss and the second loss weighted sum, the result training capsule net based on weighted sum
Network.
According to the other side of the embodiment of the present application, a kind of image processing apparatus for providing, comprising:
First capsule unit is obtained for carrying out feature extraction processing to image to be processed by least two first capsules
To at least two fisrt feature data, wherein each characteristic includes multiple vectors at least two characteristic；
Main side decomposition unit determines the main characteristic of the second capsule for being based on at least two fisrt feature data
According to side characteristic, wherein network layer belonging to second capsule is located at the network layer of at least two first capsule
Later；
Second capsule unit, for being carried out by second capsule to the main characteristic and the side characteristic
Processing, obtains second feature data；
As a result obtaining unit obtains processing result image for being based on the second feature data.
Optionally, the main side decomposition unit, comprising:
Main characteristic module is used for position main capsule corresponding with second capsule at least two first capsule
Main characteristic of the obtained fisrt feature data as second capsule；
Side characteristic module, for based at least one side at least two first capsule in addition to the main capsule
The fisrt feature data that capsule obtains determine the side characteristic of second capsule.
Optionally, the side characteristic module, specifically for the fisrt feature data obtained at least one described side capsule
Process of convolution is carried out, the side characteristic of second capsule is obtained.
Optionally, the second capsule unit, comprising:
Data computation module determines second capsule for being based on the main characteristic and the side characteristic
Input data；
Data processing module obtains second feature for handling by second capsule the input data
Data.
Optionally, the data computation module, specifically for based on the main characteristic, the main characteristic
Second weight of one weight, the side characteristic and the side characteristic obtains the input data of second capsule.
Optionally, first weight and second weight are obtained by training.
Optionally, described device utilizes capsule network implementations, and the capsule network includes at least two network layers, Mei Gesuo
Stating network layer includes at least one capsule；
Described device, further includes:
Sample reception unit, for sample image to be inputted the capsule network；
First-loss unit, for two features based on two at least two network layer network layer output
Characteristic distance between figure obtains first-loss；
Training unit, for based on the first-loss training capsule network.
Optionally, the sample image has mark processing result；
Described device, further includes:
Predicting unit obtains prediction processing result for handling based on the capsule network the sample image；
Second loss unit determines the second loss for being based on the prediction processing result and the mark processing result；
The training unit is specifically used for based on the first-loss and the second loss training capsule network.
Optionally, the first-loss unit, specifically for obtaining two network layers at least two network layer
The the first predicted characteristics figure and the second predicted characteristics figure of output, the first predicted characteristics figure includes what at least one capsule exported
First predicted characteristics data, the second predicted characteristics figure include the second predicted characteristics data of at least one capsule output；It is right
The second predicted characteristics figure carries out up-sampling operation, the second predicted characteristics figure after making the up-sampling and first prediction
Characteristic pattern matches；Based on the feature between the second predicted characteristics figure and the first predicted characteristics figure after the up-sampling away from
From determining the first-loss.
Optionally, the training unit is specifically used for for the first-loss and the second loss weighted sum being based on
The result training capsule network of weighted sum.
According to the other side of the embodiment of the present application, a kind of electronic equipment provided, including processor, the processor
Including image processing apparatus described in any one as above.
According to the other side of the embodiment of the present application, a kind of electronic equipment that provides, comprising: memory, for storing
Executable instruction；
And processor, it is as above any one to complete that the executable instruction is executed for communicating with the memory
The operation of item described image processing method.
According to the other side of the embodiment of the present application, a kind of computer readable storage medium provided, based on storing
The instruction that calculation machine can be read, described instruction are performed the operation for executing any one described image processing method as above.
According to the other side of the embodiment of the present application, a kind of computer program product provided, including it is computer-readable
Code, when the computer-readable code is run in equipment, the processor in the equipment is executed for realizing such as taking up an official post
The instruction for a described image processing method of anticipating.
According to another aspect of the embodiment of the present application, another computer program product provided is calculated for storing
Machine readable instruction, described instruction is performed so that computer executes at image described in any of the above-described possible implementation
The operation of reason method.
In an optional embodiment, the computer program product is specially computer storage medium, at another
In optional embodiment, the computer program product is specially software product, such as SDK etc..
According to the embodiment of the present application also provides another image processing methods and device, electronic equipment, computer storage
Medium, computer program product, wherein feature extraction processing is carried out to image to be processed by least two first capsules, is obtained
To at least two fisrt feature data, wherein each characteristic includes multiple vectors at least two characteristics；Based on extremely
Few two fisrt feature data, determine the main characteristic and side characteristic of the second capsule, wherein net belonging to the second capsule
Network layers are located at after the network layer of at least two first capsules；Main characteristic and side characteristic are carried out by the second capsule
Processing, obtains second feature data；Based on second feature data, processing result image is obtained.
A kind of image processing method and device, electronic equipment, storage medium, journey provided based on the above embodiments of the present application
Sequence product carries out feature extraction processing to image to be processed by least two first capsules, obtains at least two fisrt feature
Data, wherein each characteristic includes multiple vectors at least two characteristics；Based at least two fisrt feature numbers
According to determining the main characteristic and side characteristic of the second capsule, wherein network layer belonging to the second capsule is located at least two
After the network layer of first capsule；Main characteristic and side characteristic are handled by the second capsule, obtain the second spy
Levy data；Based on second feature data, processing result image is obtained；By the characteristic between capsule be divided into main characteristic and
The transmission of side characteristic, reduces the complexity of image procossing, improves image procossing performance.
Below by drawings and examples, the technical solution of the application is described in further detail.
Detailed description of the invention
The attached drawing for constituting part of specification describes embodiments herein, and together with description for explaining
The principle of the application.
The application can be more clearly understood according to following detailed description referring to attached drawing, in which:
Fig. 1 is the flow chart of the application image processing method one embodiment.
Fig. 2 is the structural schematic diagram of transmission feature data between capsule in the embodiment of the present application.
Fig. 3 is the structural schematic diagram of training capsule network in the embodiment of the present application.
Fig. 4 is the flow diagram for obtaining first-loss in the embodiment of the present application during training capsule network.
Fig. 5 is a structural schematic diagram of the embodiment of the present application image processing apparatus.
Fig. 6 is the structural representation suitable for the electronic equipment of the terminal device or server that are used to realize the embodiment of the present application
Figure.
Specific embodiment
The various exemplary embodiments of the application are described in detail now with reference to attached drawing.It should also be noted that unless in addition having
Body explanation, the unlimited system of component and the positioned opposite of step, numerical expression and the numerical value otherwise illustrated in these embodiments is originally
The range of application.
Simultaneously, it should be appreciated that for ease of description, the size of various pieces shown in attached drawing is not according to reality
Proportionate relationship draw.
Be to the description only actually of at least one exemplary embodiment below it is illustrative, never as to the application
And its application or any restrictions used.
Technology, method and apparatus known to person of ordinary skill in the relevant may be not discussed in detail, but suitable
In the case of, the technology, method and apparatus should be considered as part of specification.
It should also be noted that similar label and letter indicate similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, then in subsequent attached drawing does not need that it is further discussed.
Fig. 1 is the flow chart of the application image processing method one embodiment.As shown in Figure 1, the embodiment method includes:
Step 110, feature extraction processing is carried out to image to be processed by least two first capsules, obtains at least two
Fisrt feature data.
Wherein, each characteristic includes multiple vectors at least two characteristics；Due to capsule network and common mind
The feature of difference through network, each capsule output is expressed in the form of vectors, such as: some entity is measured so that vector field homoemorphism is long
The probability of appearance, modulus value is bigger, and probability is bigger, indicates the angle etc. that some entity turns to the direction of vector；I.e. each feature
Data include the characteristic point of multiple vector form expression,
Step 120, at least two fisrt feature data are based on, determine the main characteristic and side characteristic of the second capsule
According to.
Wherein, network layer belonging to the second capsule is located at after the network layer of at least two first capsules.
Since the connection type of conventional capsule network and the connection type of fully-connected network are identical；Each capsule of preceding layer
All each capsule is connected with later layer, therefore, between capsule during transfer characteristic data later layer capsule input
It is mapped in the space of preceding layer capsule by a transformation matrix first, then routing procedure trial will be similar to high layer information
Specific low layer capsule send information to high level, and activate corresponding capsule characteristics.The defect of this capsule network is exactly to count
It is very high to calculate complexity, and parameter amount is very big.
At least two fisrt feature data by being determined as the main characteristic and side feature of the second capsule by the present embodiment
Data, side characteristic are that a large amount of characteristics are narrowed down to corresponding main characteristic after processing (such as: process of convolution)
Size, reduce data transmission capacity；The characteristic transmittance process between capsule is simplified, computation complexity is reduced, subtracts
Lack parameter amount, accelerates calculating speed.
Step 130, main characteristic and side characteristic are handled by the second capsule, obtains second feature number
According to.
In one or more optional embodiments, it is based on main characteristic and side characteristic, determines the second capsule
Input data；
Input data is handled by the second capsule, obtains second feature data.
Optionally, main characteristic and side characteristic are mapped on the corresponding spatial position of the second capsule respectively, and
The input data of the second capsule is obtained based on main characteristic and side characteristic by weighted summation；Optionally, base
In main characteristic, the first weight of main characteristic, the second weight of side characteristic and side characteristic, the second glue is obtained
The input data of capsule.
Wherein, the first weight corresponds to main characteristic, the second weight respective side characteristic, passes through main characteristic and the
The product and side characteristic of one weight and the second weight sum of products, that is, can determine the input data of the second capsule；This implementation
The first weight and the second weight in example are obtained based on the process of simplification.
Simplify process optionally, include the following: for conventional capsules network, the output of high-rise capsule is to pass through ballot
Parameter c and low layer capsule are specifically shown in formula (1) come what is obtained:
Wherein, i indicates that i-th of capsule in lower layer network layer, j indicate j-th of capsule in upper layer network layer, cijIt indicates
I-th of capsule in lower layer network layer and to the ballot parameter between j-th of capsule in upper layer network layer, sjIndicate high-rise net
The received characteristic information of j-th of capsule in network layers, v indicate the characteristic information of low layer capsule output,Indicate lower layer network layer
In j-th of capsule from i-th of capsule to upper layer network layer transmission characteristic information, can be by becoming to formula (1)
Shape obtains formula (2):
Wherein, the content in formula (1) is splitted into two parts by formula (2), by simplification, can get formula (3):
Wherein, two on the right side of formula (3) respectively represent main split's (main characteristic) and side branch (side characteristic).
Wherein main split is intended to find the identical mode (in the corresponding capsule of characteristic pattern spatial location) between adjacent two layers, and side
Branch is intended to supplement main split using the information in remaining capsule of lower layer network layer；The specific process that simplifies can pass through
The characteristic pattern constituted to the feature of other capsules output in addition to i=j carries out convolution operation, to obtain side feature, wherein side is special
The size of sign can be identical as the size of main feature.
Optionally, the first weight and the second weight are obtained by training.
In capsule network, identical entity can be perceived in different positions by different capsules, and approximate entity is in phase
Same position can be perceived by the same capsule, therefore can identify the position of entity.
In the present embodiment, the first weight and the second weight are by simplifying approximate acquisition, and specifically simplified degree passes through
The network training for including at least one first capsule and at least one the second capsule is obtained, to realize simplified network implementations
With the same or similar treatment effect of network before simplification.As shown in formula (3), the weighted value (the first weight) of main characteristic
M as in formula (3)1, the weighted value (the second weight) of side characteristic is the m in formula (3)2, by capsule network
Training, can get m1And m2Value.
Step 140, second feature data are based on, processing result image is obtained.
Based on a kind of image processing method that the above embodiments of the present application provide, place is treated by least two first capsules
It manages image and carries out feature extraction processing, obtain at least two fisrt feature data, wherein each spy at least two characteristics
Levying data includes multiple vectors；Based at least two fisrt feature data, the main characteristic and side feature of the second capsule are determined
Data, wherein network layer belonging to the second capsule is located at after the network layer of at least two first capsules；Pass through the second capsule pair
Main characteristic and side characteristic are handled, and second feature data are obtained；Based on second feature data, image procossing is obtained
As a result；Characteristic between capsule is divided into main characteristic and the transmission of side characteristic, reduces the complexity of image procossing
Degree improves image procossing performance.
In one or more optional embodiments, step 120 may include:
Using position main capsule corresponding with the second capsule obtains at least two first capsules fisrt feature data as
The main characteristic of second capsule；
Based on the fisrt feature data that at least one side capsule at least two first capsules in addition to main capsule obtains,
Determine the side characteristic of the second capsule.
Optionally, at least two characteristics that at least two first capsules obtain in lower layer network layer may make up a spy
Sign figure, the portion in each first capsule character pair figure, the second capsule also portion in character pair figure will be with
Second capsule corresponds to main characteristic of the fisrt feature data as the second capsule of the first capsule of same position, and each second
Capsule corresponds to the fisrt feature data of different the first capsule output as main characteristic；Not corresponding first capsule in position is defeated
Side characteristic of the characteristic out as second capsule guarantees all low layer capsule outputs to supplement relevant information
Characteristic is applied in the processing of high-rise capsule.
Optionally, the first spy obtained based at least one side capsule at least two first capsules in addition to main capsule
Data are levied, determine the side characteristic of the second capsule, comprising:
The fisrt feature data that capsule obtains at least one side carry out process of convolution, obtain the side characteristic of the second capsule
According to.
In order to simplify the fisrt feature data of multiple first capsule outputs, reduce transmitted data amount, the present embodiment passes through
Process of convolution (such as: realized using at least one convolutional layer), biggish feature can be reduced to by process of convolution smaller
Feature, and the parameter of the convolutional layer pass through in network training process training determine.
Fig. 2 is the structural schematic diagram of transmission feature data between capsule in the embodiment of the present application.As shown in Fig. 2, proposing
A kind of approximate way substituting original capsule network, for the information of low layer capsule by the incoming high level of two-way, a part is main feature
Data, serve as the main source of high layer information, and data source is spatial position low layer glue corresponding with high-rise capsule position
Capsule；Another part is side characteristic, for finding other feature variants, and the supplemental information of main split is served as, in order to protect
The performance of card capsule network does not decline, and all characteristics that low layer capsule is exported are applied in the processing of high-rise capsule,
Therefore, all characteristics other than main characteristic are mapped to the corresponding position of high-rise capsule by side characteristic, with
Simplify data transmission.
In one or more optional embodiments, present implementation utilizes capsule network implementations；Capsule network includes extremely
Few two network layers, each network layer includes at least one capsule；
The present embodiment method prior to step 110, can also include:
Sample image is inputted into the capsule network；
The characteristic distance between two characteristic patterns exported based on two network layers at least two network layers, obtains first
Loss；
Based on first-loss training capsule network.
Capsule network is similar with other neural networks, before realizing image procossing, in order to reach preferable image procossing
Effect needs to be trained capsule network based on the sample image with annotation results.
The present embodiment proposes a kind of first-loss for embodying and unanimously feeding back, by the high-rise feature with low layer of comparison, most
Excellent transmission divergence to carry out regularization to feature, is trained by first-loss to capsule network, accelerates network training
Speed.
Optionally, sample image has mark processing result；
Before first-loss training capsule network, can also include:
Sample image is handled based on capsule network, obtains prediction processing result；
Based on prediction processing result and mark processing result, the second loss is determined；
Based on first-loss training capsule network, comprising:
Training capsule network is lost based on first-loss and second.
It can get network losses by prediction processing result and annotation results, by reversed gradient algorithm, it can be achieved that glue
The training of each network layer in keed network.
Optionally, the characteristic distance between two characteristic patterns exported based on two network layers at least two network layers,
Obtain first-loss, comprising:
The the first predicted characteristics figure and the second predicted characteristics figure of two network layers output at least two network layers are obtained,
First predicted characteristics figure includes the first predicted characteristics data of at least one capsule output, and the second predicted characteristics figure includes at least one
Second predicted characteristics data of a capsule output；
Up-sampling operation is carried out to the second predicted characteristics figure, the second predicted characteristics figure and the first prediction after making up-sampling are special
Sign figure matches；
Based on the characteristic distance between the second predicted characteristics figure and the first predicted characteristics figure after up-sampling, the first damage is determined
It loses.
In the design of previous capsule network, high-rise capsule can just meeting when having good consistency with low layer
It is activated, thus inspires, the present embodiment devises a kind of loss function (the second loss), it is desirable that upper layer network can recover low
The information of layer network.Optimal transmission strategy (Optimal transport) is utilized to measure the distance of two feature spaces.
Fig. 3 is the structural schematic diagram of training capsule network in the embodiment of the present application.As shown in figure 3, capConv is capsule volume
Lamination, capFC are capsule non-linear layer, can also include that a shortcut connects (skip connection), optimal transmission divergence
The part of (Sinkhorn divergence) is as a loss for supervising so that low layer capsule and high-rise capsule information one
It causes, reduces the loss that information is propagated between layers, by minimizing the optimal transmission divergence (Sinkhorn between two layers
Divergence), it can achieve this purpose, therefore, capsule network be trained using transmission divergence as first-loss.
Fig. 4 is the flow diagram for obtaining first-loss in the embodiment of the present application during training capsule network.Such as Fig. 4
It is shown, the process that entire Sinkhorn divergence calculates is illustrated, the feature u of low layer capsule output is calculatedyIt is exported with high-rise capsule
Feature vxThe distance between, due to vxCompare uyIt is small, therefore, by up-sampling vxBe converted to gψ, calculate the feature after up-sampling
F is obtained with the feature calculation distance of low layer capsule outputφ, the form that this feature vector is converted to Q is expressed, Q indicates two spies
Function between sign obtains K based on Q, and then determines Sinkhorn divergence
Optionally, training capsule network is lost based on first-loss and second, comprising:
By first-loss and the second loss weighted sum, the result training capsule network based on weighted sum.
Optionally, training speed can be accelerated by first-loss and the second loss weighted sum, wherein first-loss and the
The corresponding weighted value of two losses can be set by actual task, such as: the weighted value of first-loss is set as 10, the second damage
The weighted value of mistake is corresponding to be set as 1.
Optionally, the first predicted characteristics figure and second of two network layers output at least two network layer is obtained
Predicted characteristics figure, comprising:
At least two first branches spy is exported respectively by least two capsules in the first network layer in two network layers
Data are levied, combines at least two first branching characteristic data according to the corresponding feature space of sample image, obtains the first prediction
Characteristic pattern；
At least two second branches spy is exported respectively by least two capsules in the second network layer in two network layers
Data are levied, combines at least two second branching characteristic data according to the corresponding feature space of sample image, obtains the second prediction
Characteristic pattern；The output end of first network layer is directly or indirectly connected with the input terminal of the second network layer.
Capsule in each network layer respectively corresponds the Partial Feature of sample image, and in order to be lost, needing will be same
The feature that multiple capsules in layer obtain is combined.
In one or more optional embodiments, the present embodiment method can also include:
Using next network layer as a upper network layer, by the network layer of the output end direct or indirect connection of next network layer
As next network layer；
Second characteristics of image is decomposed as main feature and side feature group；
Main feature and side feature group are mapped in the correspondence lower layer capsule in the next network layer for needing to input, under
Network layer is not present in the output end of one network layer.
When in capsule network including 3 layers or 3 layers or more of network layer, the dynamic routing between every two network layer is all logical
Two Mapping implementations are crossed, the received main feature of each capsule and side feature group are transferred in the capsule by two mappings respectively, with
Activate the capsule.
Those of ordinary skill in the art will appreciate that: realize that all or part of the steps of above method embodiment can pass through
The relevant hardware of program instruction is completed, and program above-mentioned can be stored in a computer readable storage medium, the program
When being executed, step including the steps of the foregoing method embodiments is executed；And storage medium above-mentioned includes: ROM, RAM, magnetic disk or light
The various media that can store program code such as disk.
Fig. 5 is a structural schematic diagram of the embodiment of the present application image processing apparatus.The device of the embodiment can be used for reality
The existing above-mentioned each method embodiment of the application.As shown in figure 5, the device of the embodiment includes:
First capsule unit 51, for carrying out feature extraction processing to image to be processed by least two first capsules,
Obtain at least two fisrt feature data.
Wherein, each characteristic includes multiple vectors at least two characteristics.
Main side decomposition unit 52 determines the main characteristic of the second capsule for being based at least two fisrt feature data
According to side characteristic.
Wherein, network layer belonging to the second capsule is located at after the network layer of at least two first capsules.
Second capsule unit 53 is obtained for being handled by the second capsule main characteristic and side characteristic
Second feature data.
As a result obtaining unit 54 obtain processing result image for being based on the second feature data.
Based on a kind of image processing apparatus that the above embodiments of the present application provide, place is treated by least two first capsules
It manages image and carries out feature extraction processing, obtain at least two fisrt feature data, wherein each spy at least two characteristics
Levying data includes multiple vectors；Based at least two fisrt feature data, the main characteristic and side feature of the second capsule are determined
Data, wherein network layer belonging to the second capsule is located at after the network layer of at least two first capsules；Pass through the second capsule pair
Main characteristic and side characteristic are handled, and second feature data are obtained；Based on second feature data, image procossing is obtained
As a result；Characteristic between capsule is divided into main characteristic and the transmission of side characteristic, reduces the complexity of image procossing
Degree improves image procossing performance.
In one or more optional embodiments, main side decomposition unit 52, comprising:
Main characteristic module, for obtaining position main capsule corresponding with the second capsule at least two first capsules
Main characteristic of one characteristic as the second capsule；
Side characteristic module, for being obtained based at least one side capsule at least two first capsules in addition to main capsule
Fisrt feature data, determine the side characteristic of the second capsule.
Optionally, at least two characteristics that at least two first capsules obtain in lower layer network layer may make up a spy
Sign figure, the portion in each first capsule character pair figure, the second capsule also portion in character pair figure will be with
Second capsule corresponds to main characteristic of the fisrt feature data as the second capsule of the first capsule of same position, and each second
Capsule corresponds to the fisrt feature data of different the first capsule output as main characteristic；Not corresponding first capsule in position is defeated
Side characteristic of the characteristic out as second capsule guarantees all low layer capsule outputs to supplement relevant information
Characteristic is applied in the processing of high-rise capsule.
Optionally, side characteristic module carries out convolution specifically for the fisrt feature data obtained at least one side capsule
Processing obtains the side characteristic of the second capsule.
In one or more optional embodiments, the second capsule unit 53, comprising:
Data computation module determines the input data of the second capsule for being based on main characteristic and side characteristic；
Data processing module obtains second feature data for handling by the second capsule input data.
Optionally, main characteristic and side characteristic are mapped on the corresponding spatial position of the second capsule respectively, and
The input data of the second capsule is obtained based on main characteristic and side characteristic by weighted summation；Optionally, base
In main characteristic, the first weight of main characteristic, the second weight of side characteristic and side characteristic, the second glue is obtained
The input data of capsule.
Optionally, data computation module, it is special specifically for the first weight based on main characteristic, main characteristic, side
The second weight for levying data and side characteristic, obtains the input data of the second capsule.
Optionally, the first weight and the second weight are obtained by training.
In one or more optional embodiments, the present embodiment device utilizes capsule network implementations, and capsule network includes
At least two network layers, each network layer include at least one capsule；
The present embodiment device can also include:
Sample reception unit, for sample image to be inputted capsule network；
First-loss unit, for based at least two network layers two network layers export two characteristic patterns between
Characteristic distance obtains first-loss；
Training unit, for based on first-loss training capsule network.
The present embodiment proposes a kind of first-loss for embodying and unanimously feeding back, by the high-rise feature with low layer of comparison, most
Excellent transmission divergence to carry out regularization to feature, is trained by first-loss to capsule network, accelerates network training
Speed.
Optionally, sample image has mark processing result；
The present embodiment device, further includes:
Predicting unit obtains prediction processing result for handling based on capsule network sample image；
Second loss unit, for determining the second loss based on prediction processing result and mark processing result；
Training unit is specifically used for losing training capsule network based on first-loss and second.
Optionally, first-loss unit, specifically for obtaining the of two network layers output at least two network layers
One predicted characteristics figure and the second predicted characteristics figure, the first predicted characteristics figure include the first predicted characteristics of at least one capsule output
Data, the second predicted characteristics figure include the second predicted characteristics data of at least one capsule output；To the second predicted characteristics figure into
Row up-sampling operation, the second predicted characteristics figure after making up-sampling match with the first predicted characteristics figure；After up-sampling
Characteristic distance between second predicted characteristics figure and the first predicted characteristics figure, determines first-loss.
Optionally, training unit is specifically used for first-loss and the second loss weighted sum, the knot based on weighted sum
Fruit trains capsule network.
The course of work and set-up mode of image processing apparatus any embodiment provided by the embodiments of the present application can be with
Referring to the specific descriptions of the above-mentioned correlation method embodiment of the application, as space is limited, details are not described herein.
According to the other side of the embodiment of the present application, a kind of electronic equipment provided, including processor, the processor
Including image processing apparatus described in any one embodiment as above.
According to the other side of the embodiment of the present application, a kind of electronic equipment that provides, comprising: memory, for storing
Executable instruction；
And processor, it is as above any one to complete that the executable instruction is executed for communicating with the memory
The operation of embodiment described image processing method.
According to the other side of the embodiment of the present application, a kind of computer readable storage medium provided, based on storing
The instruction that calculation machine can be read, described instruction are performed the operation for executing as above any one embodiment described image processing method.
According to the other side of the embodiment of the present application, a kind of computer program product provided, including it is computer-readable
Code, when the computer-readable code is run in equipment, the processor in the equipment is executed for realizing such as taking up an official post
The instruction for an embodiment described image processing method of anticipating.
According to another aspect of the embodiment of the present application, another computer program product provided is calculated for storing
Machine readable instruction, described instruction is performed so that computer executes at image described in any of the above-described possible implementation
The operation of reason method.
In one or more optional embodiments, the embodiment of the present application also provides a kind of productions of computer program program
Product, for storing computer-readable instruction, described instruction is performed so that computer executes any of the above-described possible realization side
The operation of image processing method described in formula.
The computer program product can be realized especially by hardware, software or its mode combined.In an alternative embodiment
In son, the computer program product is embodied as computer storage medium, in another optional example, the computer
Program product is embodied as software product, such as software development kit (Software Development Kit, SDK) etc..
According to the embodiment of the present application also provides image processing methods and device, electronic equipment, computer storage medium, meter
Calculation machine program product, wherein feature extraction processing is carried out to image to be processed by least two first capsules, obtains at least two
A fisrt feature data, wherein each characteristic includes multiple vectors at least two characteristics；Based at least two
One characteristic determines the main characteristic and side characteristic of the second capsule, wherein network layer belonging to the second capsule is located at
After the network layer of at least two first capsules；Main characteristic and side characteristic are handled by the second capsule, obtained
To second feature data；Based on second feature data, processing result image is obtained.
In some embodiments, image procossing instruction can be specially call instruction, and first device can pass through calling
Mode indicate second device execute image procossing, accordingly, in response to call instruction is received, second device can be executed
State the step and/or process in any embodiment in image processing method.
It should be understood that the terms such as " first " in the embodiment of the present application, " second " are used for the purpose of distinguishing, and be not construed as
Restriction to the embodiment of the present application.
It should also be understood that in this application, " multiple " can refer to two or more, "at least one" can refer to one,
Two or more.
It should also be understood that clearly being limited or no preceding for the either component, data or the structure that are referred in the application
In the case where opposite enlightenment given hereinlater, one or more may be generally understood to.
It should also be understood that the application highlights the difference between each embodiment to the description of each embodiment,
Same or similar place can be referred to mutually, for sake of simplicity, no longer repeating one by one.
The embodiment of the present application also provides a kind of electronic equipment, such as can be mobile terminal, personal computer (PC), put down
Plate computer, server etc..Below with reference to Fig. 6, it illustrates the terminal device or the services that are suitable for being used to realize the embodiment of the present application
The structural schematic diagram of the electronic equipment 600 of device: as shown in fig. 6, electronic equipment 600 includes one or more processors, communication unit
For example Deng, one or more of processors: one or more central processing unit (CPU) 601, and/or one or more figures
As processor (GPU) 613 etc., processor can according to the executable instruction being stored in read-only memory (ROM) 602 or from
Executable instruction that storage section 608 is loaded into random access storage device (RAM) 603 and execute various movements appropriate and place
Reason.Communication unit 612 may include but be not limited to network interface card, and the network interface card may include but be not limited to IB (Infiniband) network interface card.
Processor can with communicate in read-only memory 602 and/or random access storage device 603 to execute executable instruction,
It is connected by bus 604 with communication unit 612 and is communicated through communication unit 612 with other target devices, to completes the application implementation
The corresponding operation of any one method that example provides mentions for example, carrying out feature to image to be processed by least two first capsules
Take processing, obtain at least two fisrt feature data, wherein at least two characteristics each characteristic include it is multiple to
Amount；Based at least two fisrt feature data, the main characteristic and side characteristic of the second capsule are determined, wherein the second glue
Network layer belonging to capsule is located at after the network layer of at least two first capsules；It is special to main characteristic and side by the second capsule
Sign data are handled, and second feature data are obtained；Based on second feature data, processing result image is obtained.
In addition, in RAM 603, various programs and data needed for being also stored with device operation.CPU601,ROM602
And RAM603 is connected with each other by bus 604.In the case where there is RAM603, ROM602 is optional module.RAM603 storage
Executable instruction, or executable instruction is written into ROM602 at runtime, executable instruction executes central processing unit 601
The corresponding operation of above-mentioned communication means.Input/output (I/O) interface 605 is also connected to bus 604.Communication unit 612 can integrate
Setting, may be set to be with multiple submodule (such as multiple IB network interface cards), and in bus link.
I/O interface 605 is connected to lower component: the importation 606 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 608 including hard disk etc.；
And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net executes communication process.Driver 610 is also connected to I/O interface 605 as needed.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 610, in order to read from thereon
Computer program be mounted into storage section 608 as needed.
It should be noted that framework as shown in FIG. 6 is only a kind of optional implementation, it, can root during concrete practice
The component count amount and type of above-mentioned Fig. 6 are selected, are deleted, increased or replaced according to actual needs；It is set in different function component
It sets, separately positioned or integrally disposed and other implementations, such as the separable setting of GPU613 and CPU601 or can also be used
GPU613 is integrated on CPU601, the separable setting of communication unit, can also be integrally disposed on CPU601 or GPU613, etc..
These interchangeable embodiments each fall within protection scope disclosed in the present application.
Particularly, according to an embodiment of the present application, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiments herein includes a kind of computer program product comprising be tangibly embodied in machine readable
Computer program on medium, computer program include the program code for method shown in execution flow chart, program code
It may include the corresponding instruction of corresponding execution method and step provided by the embodiments of the present application, for example, passing through at least two first capsules
Feature extraction processing is carried out to image to be processed, obtains at least two fisrt feature data, wherein at least two characteristics
Each characteristic includes multiple vectors；Based at least two fisrt feature data, determine the second capsule main characteristic and
Side characteristic, wherein network layer belonging to the second capsule is located at after the network layer of at least two first capsules；Pass through second
Capsule handles main characteristic and side characteristic, obtains second feature data；Based on second feature data, figure is obtained
As processing result.In such embodiments, which can be downloaded and be pacified from network by communications portion 609
Dress, and/or be mounted from detachable media 611.When the computer program is executed by central processing unit (CPU) 601, execute
The operation for the above-mentioned function of being limited in the present processes.
The present processes and device may be achieved in many ways.For example, can by software, hardware, firmware or
Software, hardware, firmware any combination realize the present processes and device.The said sequence of the step of for the method
Merely to be illustrated, the step of the present processes, is not limited to sequence described in detail above, special unless otherwise
It does not mentionlet alone bright.In addition, in some embodiments, also the application can be embodied as to record program in the recording medium, these programs
Including for realizing according to the machine readable instructions of the present processes.Thus, the application also covers storage for executing basis
The recording medium of the program of the present processes.
The description of the present application is given for the purpose of illustration and description, and is not exhaustively or by the application
It is limited to disclosed form.Many modifications and variations are obvious for the ordinary skill in the art.It selects and retouches
Embodiment is stated and be the principle and practical application in order to more preferably illustrate the application, and those skilled in the art is enable to manage
Solution the application is to design various embodiments suitable for specific applications with various modifications.
Claims (10)
1. a kind of image processing method characterized by comprising
Feature extraction processing is carried out to image to be processed by least two first capsules, obtains at least two fisrt feature numbers
According to, wherein each characteristic includes multiple vectors at least two characteristic；
Based on at least two fisrt feature data, the main characteristic and side characteristic of the second capsule are determined, wherein institute
Network layer belonging to the second capsule is stated to be located at after the network layer of at least two first capsule；
The main characteristic and the side characteristic are handled by second capsule, obtain second feature number
According to；
Based on the second feature data, processing result image is obtained.
2. the method according to claim 1, wherein described be based on at least two fisrt feature data, really
The main characteristic and supplemental characteristic data of fixed second capsule, comprising:
By the fisrt feature data that position main capsule corresponding with second capsule obtains at least two first capsule
Main characteristic as second capsule；
The fisrt feature obtained based at least one side capsule at least two first capsule in addition to the main capsule
Data determine the side characteristic of second capsule.
3. according to the method described in claim 2, it is characterized in that, described described based on being removed at least two first capsule
The fisrt feature data that at least one side capsule except main capsule obtains determine the side characteristic of second capsule, packet
It includes:
Process of convolution is carried out to the fisrt feature data that at least one described side capsule obtains, the side for obtaining second capsule is special
Levy data.
4. method according to claim 1 to 3, which is characterized in that it is described by second capsule to the main spy
Sign data and the side characteristic are handled, and second feature data are obtained, comprising:
Based on the main characteristic and the side characteristic, the input data of second capsule is determined；
The input data is handled by second capsule, obtains second feature data.
5. according to the method described in claim 4, it is characterized in that, described be based on the main characteristic and the side characteristic
According to determining the input data of second capsule, comprising:
Based on the main characteristic, the first weight of the main characteristic, the side characteristic and the side characteristic
According to the second weight, obtain the input data of second capsule.
6. a kind of image processing apparatus characterized by comprising
First capsule unit, for by least two first capsules to image to be processed carry out feature extraction processing, obtain to
Few two fisrt feature data, wherein each characteristic includes multiple vectors at least two characteristic；
Main side decomposition unit, for be based on at least two fisrt feature data, determine the second capsule main characteristic and
Side characteristic, wherein network layer belonging to second capsule is located at after the network layer of at least two first capsule；
Second capsule unit, for by second capsule to the main characteristic and the side characteristic at
Reason, obtains second feature data；
As a result obtaining unit obtains processing result image for being based on the second feature data.
7. a kind of electronic equipment, which is characterized in that including processor, the processor includes at image as claimed in claim 6
Manage device.
8. a kind of electronic equipment characterized by comprising memory, for storing executable instruction；
And processor, for being communicated with the memory to execute the executable instruction to complete claim 1 to 5 times
The operation for a described image processing method of anticipating.
9. a kind of computer readable storage medium, for storing computer-readable instruction, which is characterized in that described instruction quilt
Perform claim requires the operation of 1 to 5 any one described image processing method when execution.
10. a kind of computer program product, including computer-readable code, which is characterized in that when the computer-readable code
When running in equipment, the processor in the equipment is executed for realizing the processing of claim 1 to 5 any one described image
The instruction of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810892869.XA CN109344839B (en) | 2018-08-07 | 2018-08-07 | Image processing method and apparatus, electronic device, storage medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810892869.XA CN109344839B (en) | 2018-08-07 | 2018-08-07 | Image processing method and apparatus, electronic device, storage medium, and program product |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109344839A true CN109344839A (en) | 2019-02-15 |
| CN109344839B CN109344839B (en) | 2020-11-27 |
Family
ID=65296522
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810892869.XA Active CN109344839B (en) | 2018-08-07 | 2018-08-07 | Image processing method and apparatus, electronic device, storage medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344839B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110119449A (en) * | 2019-05-14 | 2019-08-13 | 湖南大学 | A kind of criminal case charge prediction technique based on sequence enhancing capsule net network |
| CN110414317A (en) * | 2019-06-12 | 2019-11-05 | 四川大学 | Full-automatic Arneth's count method based on capsule network |
| CN111325259A (en) * | 2020-02-14 | 2020-06-23 | 武汉大学 | Remote sensing image classification method based on deep learning and binary coding |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104812288A (en) * | 2012-11-29 | 2015-07-29 | 奥林巴斯株式会社 | Image processing device, image processing method, and image processing program |
-
2018
- 2018-08-07 CN CN201810892869.XA patent/CN109344839B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104812288A (en) * | 2012-11-29 | 2015-07-29 | 奥林巴斯株式会社 | Image processing device, image processing method, and image processing program |
Non-Patent Citations (3)
| Title |
|---|
| SARA SABOUR等: "Dynamic Routing Between CapsulesDynamic Routing Between Capsules", 《31ST CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS》 * |
| 机器之心: "先读懂CapsNet架构然后用TensorFlow实现，这应该是最详细的教程了", 《HTTPS://ZHUANLAN.ZHIHU.COM/P/30753326》 * |
| 机器之心PRO: "教程 | 可视化CapsNet，详解Hinton等人提出的胶囊概念与原理", 《HTTPS://WWW.SOHU.COM/A/227482961_129720HTTPS://WWW.SOHU.COM/A/227482961_129720》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110119449A (en) * | 2019-05-14 | 2019-08-13 | 湖南大学 | A kind of criminal case charge prediction technique based on sequence enhancing capsule net network |
| CN110119449B (en) * | 2019-05-14 | 2020-12-25 | 湖南大学 | Criminal case criminal name prediction method based on sequence-enhanced capsule network |
| CN110414317A (en) * | 2019-06-12 | 2019-11-05 | 四川大学 | Full-automatic Arneth's count method based on capsule network |
| CN110414317B (en) * | 2019-06-12 | 2021-10-08 | 四川大学 | Full-automatic leukocyte classification counting method based on capsule network |
| CN111325259A (en) * | 2020-02-14 | 2020-06-23 | 武汉大学 | Remote sensing image classification method based on deep learning and binary coding |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109344839B (en) | 2020-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110046537A (en) | The system and method for carrying out dynamic face analysis using recurrent neural network | |
| US20220284638A1 (en) | Method for image processing, computer device, and storage medium | |
| CN107578099A (en) | Computing device and method | |
| CN108229343A (en) | Target object critical point detection method, deep learning neural network and device | |
| CN110309856A (en) | Image classification method, the training method of neural network and device | |
| CN110378381A (en) | Object detecting method, device and computer storage medium | |
| CN109902548A (en) | A kind of object properties recognition methods, calculates equipment and system at device | |
| CN108229591A (en) | Neural network adaptive training method and apparatus, equipment, program and storage medium | |
| CN109344839A (en) | Image processing method and device, electronic equipment, storage medium, program product | |
| CN107844832A (en) | A kind of information processing method and Related product | |
| CN110009705A (en) | Image is created using the mapping for indicating variety classes pixel | |
| WO2022068623A1 (en) | Model training method and related device | |
| CN109800821A (en) | Method, image processing method, device, equipment and the medium of training neural network | |
| CN108830221A (en) | The target object segmentation of image and training method and device, equipment, medium, product | |
| CN108416436A (en) | The method and its system of neural network division are carried out using multi-core processing module | |
| CN109800789A (en) | Diabetic retinopathy classification method and device based on figure network | |
| CN108280451A (en) | Semantic segmentation and network training method and device, equipment, medium, program | |
| CN110443222A (en) | Method and apparatus for training face's critical point detection model | |
| CN109241988A (en) | Feature extracting method and device, electronic equipment, storage medium, program product | |
| CN109685068A (en) | A kind of image processing method and system based on generation confrontation neural network | |
| CN105574808B (en) | A kind of pipeline texture textures cellular system | |
| CN107004253A (en) | The application programming interface framework based on figure with equivalence class for enhanced image procossing concurrency | |
| CN110059793A (en) | The gradually modification of production confrontation neural network | |
| CN108491872A (en) | Target recognition methods and device, electronic equipment, program and storage medium again | |
| US11605001B2 (en) | Weight demodulation for a generative neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |