CN110942090B - Model training method, image processing device, electronic equipment and storage medium - Google Patents
Model training method, image processing device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110942090B CN110942090B CN201911097179.6A CN201911097179A CN110942090B CN 110942090 B CN110942090 B CN 110942090B CN 201911097179 A CN201911097179 A CN 201911097179A CN 110942090 B CN110942090 B CN 110942090B
- Authority
- CN
- China
- Prior art keywords
- network model
- parameters
- preset
- network structure
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 82
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000012549 training Methods 0.000 title claims abstract description 58
- 238000004590 computer program Methods 0.000 claims description 16
- 238000003672 processing method Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 9
- 239000004744 fabric Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 27
- 238000010586 diagram Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Abstract
The invention discloses a model training method, an image processing device, electronic equipment and a storage medium, wherein the model training method comprises the following steps: acquiring an initial network model and a target limiting condition; under the target limiting condition, based on a preset alternate updating mode, an initial network structure super parameter and a conventional parameter of an initial network model, carrying out iterative updating on the network structure super parameter and the conventional parameter of the network model until both the network structure super parameter and the conventional parameter are converged to obtain a target network model, wherein the preset alternate updating mode comprises: and carrying out iterative updating of the super-parameters of the T times of network structures based on a preset evolution strategy every S times of iterative updating of the conventional parameters. When the method is implemented, when the image processing model for realizing a specific purpose is trained, the network structure superparameter in the network model can be automatically and iteratively updated based on the preset evolution strategy, so that the cost of the model training process is reduced, the model training efficiency is improved, the cost of the whole image processing process is further reduced, and the image processing efficiency is improved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a model training method, an image processing device, an electronic device, and a storage medium.
Background
With the development of science and technology and the continuous progress of human society, image processing is widely applied in various industries, and the use of image processing technology is becoming particularly important. Currently, in some image processing scenarios, an image processing model for implementing a specific purpose is generally trained based on an initial network model, and an image to be processed is processed by means of the image processing model, wherein the model training process mainly involves updating network structure super-parameters and conventional parameters of the model.
In the prior art, when the network structure super-parameters of the network model are updated, the model is mainly realized manually, so that the model training process has higher cost and lower efficiency, and the whole image processing process has higher cost and lower efficiency.
Disclosure of Invention
The embodiment of the invention provides a model training and image processing method, device, electronic equipment and storage medium, which are used for solving the technical problems of higher cost and lower efficiency in the image processing process in the prior art.
According to a first aspect of the present invention, a model training method is disclosed, the method comprising:
acquiring a preset initial network model and a target limiting condition, wherein the preset initial network model comprises an initial network structure super-parameter and an initial conventional parameter;
under the target limiting condition, based on a preset alternate updating mode, the initial network structure super-parameters and the initial conventional parameters, carrying out iterative updating on the network structure super-parameters and the conventional parameters of the network model until the network structure super-parameters and the conventional parameters of the network model are converged to obtain a target network model;
wherein, the preset alternate updating mode comprises the following steps: and each time of iterative updating of the conventional parameters is carried out, iterative updating of the super parameters of the network structure is carried out for T times based on a preset evolution strategy, T is smaller than S, and the target network model comprises the super parameters of the network structure and the conventional parameters of the target.
Optionally, as an embodiment, the updating process of the network structure super parameter includes:
extracting an initial network structure superparameter a of the preset initial network model 1 、a 2 ,…,a N Wherein a is i The i-th initial network structure superparameter in the preset initial network model is equal to or more than 1 and equal to or less than N, wherein N is the number of the initial network structure superparameters in the preset initial network model;
Based on the a 1 、a 2 ,…,a N Generating a network structure super-parameter vector p, wherein p= (a) 1 ,a 2 ,…,a N )；
Based on the p, generating a directional derivative g for updating the network structure super-parameters P Wherein, the method comprises the steps of, wherein,loss is a preset loss function, and Deltap is disturbance quantity;
obtaining disturbance quantity set { DELTAp 1 ,△p 2 ,…,△p M -wherein Δp j J is more than or equal to 1 and less than or equal to M, which is the number of disturbance variables in the disturbance variable set;
based on the { DELTAp 1 ,△p 2 ,…,△p M And said g P Generating a set of directional derivatives { g P1 ,g P2 ,…,g PM And } wherein,g Pj a j-th directional derivative of the set of directional derivatives;
based on a preset gradient descent algorithm, the { g } P1 ,g P2 ,…,g PM Processing to obtain said p with respect to said { DELTAp 1 ,△p 2 ,…,△p M Update direction set { D } 1 ,D 2 ,…,D M }, wherein D j For said p relative to said Δp j Is updated in the update direction;
based on { D } 1 ,D 2 ,…,D M And determining the super-parameters of the target network structure.
Optionally, as an embodiment, the step of determining is based on the { D } 1 ,D 2 ,…,D M -determining target network structure superparameters, comprising:
determining the { D } 1 ,D 2 ,…,D M An update direction satisfying a preset condition in };
and calculating the average value of the network structure super-parameters corresponding to the updating direction meeting the preset conditions, and determining the average value as the target network structure super-parameters.
Optionally, as an embodiment, the network structure super parameter includes: the number of output channels per layer in the network model, the resolution of the input image of the network model, and the network depth of the network model.
Optionally, as an embodiment, the target defining condition includes any one of the following:
the total parameter number of the network model is lower than a preset number threshold, the total calculated amount of the network model is lower than a preset calculated amount threshold, and the running time of the network model on the specific equipment is lower than a preset time threshold.
According to a second aspect of the present invention, there is also disclosed an image processing method for performing image processing based on a target network model obtained by training by the above model training method, the method comprising:
receiving an image to be processed;
converting the image to be processed into input data matched with the target network model;
inputting the input data into the target network model for processing to obtain an output result of the target network model;
and determining an output result of the target network model as an image processing result of the image to be processed.
According to a third aspect of the present invention, there is also disclosed a model training apparatus, the apparatus comprising:
The acquisition module is used for acquiring a preset initial network model and target limiting conditions, wherein the preset initial network model comprises an initial network structure super-parameter and an initial conventional parameter;
the training module is used for carrying out iterative updating on the network structure super-parameters and the conventional parameters of the network model based on a preset alternate updating mode, the initial network structure super-parameters and the initial conventional parameters under the target limiting condition until the network structure super-parameters and the conventional parameters of the network model are converged to obtain a target network model;
wherein, the preset alternate updating mode comprises the following steps: and each time of iterative updating of the conventional parameters is carried out, iterative updating of the super parameters of the network structure is carried out for T times based on a preset evolution strategy, T is smaller than S, and the target network model comprises the super parameters of the network structure and the conventional parameters of the target.
Optionally, as an embodiment, the training module includes: the network structure super-parameter updating sub-module comprises:
an extracting unit, configured to extract an initial network structure superparameter a of the preset initial network model 1 、a 2 ,…,a N Wherein a is i The i-th initial network structure superparameter in the preset initial network model is equal to or more than 1 and equal to or less than N, wherein N is the number of the initial network structure superparameters in the preset initial network model;
A first generation unit for based on the a 1 、a 2 ,…,a N Generating a network structure super-parameter vector p, wherein p= (a) 1 ,a 2 ,…,a N )；
A second generation unit for generating a directional derivative g for updating the network structure super-parameters based on the p P Wherein, the method comprises the steps of, wherein,loss is a preset loss function, and Deltap is disturbance quantity;
an acquisition unit for acquiring a disturbance variable set { DELTAp 1 ,△p 2 ,…,△p M -wherein Δp j J is more than or equal to 1 and less than or equal to M, which is the number of disturbance variables in the disturbance variable set;
a third generation unit for generating a third generation result based on the { Δp 1 ,△p 2 ,…,△p M And said g P Generating a set of directional derivatives { g P1 ,g P2 ,…,g PM And } wherein,g Pj a j-th directional derivative of the set of directional derivatives;
a processing unit for based onA preset gradient descent algorithm is used for the { g } P1 ,g P2 ,…,g PM Processing to obtain said p with respect to said { DELTAp 1 ,△p 2 ,…,△p M Update direction set { D } 1 ,D 2 ,…,D M }, wherein D j For said p relative to said Δp j Is updated in the update direction;
a determining unit for based on { D } 1 ,D 2 ,…,D M And determining the super-parameters of the target network structure.
Optionally, as an embodiment, the determining unit includes:
a first determination subunit for determining the { D } 1 ,D 2 ,…,D M An update direction satisfying a preset condition in };
a calculating subunit, configured to calculate a mean value of network structure super parameters corresponding to the update direction that satisfies the preset condition;
And the second determination subunit is used for determining the average value as the target network structure super-parameter.
Optionally, as an embodiment, the network structure super parameter includes: the number of output channels per layer in the network model, the resolution of the input image of the network model, and the network depth of the network model.
Optionally, as an embodiment, the target defining condition includes any one of the following:
the total parameter number of the network model is lower than a preset number threshold, the total calculated amount of the network model is lower than a preset calculated amount threshold, and the running time of the network model on the specific equipment is lower than a preset time threshold.
According to a fourth aspect of the present invention, there is also disclosed an image processing apparatus for performing image processing based on a target network model trained by any one of the model training apparatuses described above, the image processing apparatus comprising:
the receiving module is used for receiving the image to be processed;
the input module is used for converting the image to be processed into input data matched with the target network model;
the processing module is used for inputting the input data into the target network model for processing to obtain an output result of the target network model;
And the determining module is used for determining the output result of the target network model as the image processing result of the image to be processed.
According to a fifth aspect of the present invention, there is also disclosed an electronic device comprising: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the computer program realizes the steps in any model training method when being executed by the processor or realizes the steps in any image processing method when being executed by the processor.
According to a sixth aspect of the present invention, there is also disclosed a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the model training methods described above, or which, when executed by a processor, implements the steps of any of the image processing methods described above.
In the embodiment of the invention, when the image processing model for realizing the specific purpose is trained based on the initial network model and the given limiting condition, the network structure superparameter in the network model can be automatically and iteratively updated based on the preset evolution strategy, so that the cost of the model training process is reduced, the model training efficiency is improved, the cost of the whole image processing process is further reduced, and the image processing efficiency is improved. In addition, the network structure super-parameters and the conventional parameters in the network model are updated alternately, so that the accuracy of the final model obtained through training can be ensured, and the image processing accuracy and efficiency are further improved.
Drawings
FIG. 1 is a flow chart of a model training method of one embodiment of the present invention;
FIG. 2 is a flow chart of a network fabric hyper-parameter update process of one embodiment of the invention;
FIG. 3 is a flow chart of an image processing method of one embodiment of the present invention;
FIG. 4 is a block diagram of a model training apparatus of one embodiment of the present invention;
fig. 5 is a block diagram of the image processing apparatus according to an embodiment of the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred embodiments, and that the acts are not necessarily required by the embodiments of the invention.
In some image processing scenarios, an image processing model for realizing a specific purpose is generally trained based on an initial network model, and an image to be processed is processed by means of the image processing model, wherein the model training process mainly involves updating network structure super-parameters and conventional parameters of the model.
In the prior art, when the network structure super parameters of the network model are updated, the network structure super parameters are mainly realized by manpower, and the method specifically comprises the following two modes: the first is to design network structure super parameters according to manual experience, and the second is to cut channels of each layer of the network by using a pruning algorithm, which also requires that the number of channels of each layer be determined manually. It can be seen that the model training process of the prior art has higher cost and lower efficiency, which results in higher cost and lower efficiency of the whole image processing process.
In order to solve the technical problems, the embodiment of the invention provides a model training method, an image processing device, electronic equipment and a storage medium.
The model training method provided by the embodiment of the invention is first described below.
It should be noted that, the model training method provided by the embodiment of the present invention is applicable to an electronic device, and in practical application, the electronic device may include: mobile terminals such as smartphones, tablet computers, personal digital assistants, and the like, may also include: the embodiments of the present invention are not limited to computer devices such as notebook/desktop computers, servers, and the like.
FIG. 1 is a flow chart of a model training method of one embodiment of the present invention, as shown in FIG. 1, which may include the steps of: step 101, and step 102, wherein,
in step 101, a preset initial network model and target definition conditions are obtained, wherein the preset initial network model comprises an initial network structure super parameter and an initial regular parameter.
In one example, the initial network model may be the network model that the user is using and the target constraints may be constraints given by the user.
In the embodiment of the invention, the conventional parameters are weight coefficients in a network model, and the network structure superparameter may include: the number of output channels per layer in the network model, the resolution of the input image of the network model, and the network depth of the network model. In addition, in view of the continuous improvement and the continuous proposal of the machine learning related algorithm, the network model constructed based on the related algorithm is endless, in this case, the network structure super-parameters are not limited to the three super-parameters, and may also include other super-parameters, which is not limited by the embodiment of the present invention.
In the embodiment of the invention, the target limiting conditions can comprise any one of the following: the total parameter number of the network model is lower than a preset number threshold, the total calculated amount of the network model is lower than a preset calculated amount threshold, and the running time of the network model on the specific equipment is lower than a preset time threshold. In addition, along with the upgrade of hardware configuration and software systems of the electronic device, the computing capability of the electronic device is stronger, and meanwhile, the performance requirement of a user on the model is also higher, in this case, the target limiting conditions are not limited to the three limiting conditions, but can be other limiting conditions, and the embodiment of the invention is not limited to this.
In step 102, under the condition of limiting a target, based on a preset alternate updating mode, an initial network structure super-parameter and an initial conventional parameter, performing iterative updating of the network structure super-parameter and the conventional parameter of the network model until the network structure super-parameter and the conventional parameter of the network model are converged to obtain the target network model; the preset alternate updating mode comprises the following steps: and each time of iterative updating of the conventional parameters is carried out, iterative updating of the super parameters of the network structure is carried out for T times based on a preset evolution strategy, T is smaller than S, and the target network model comprises the super parameters of the network structure and the conventional parameters of the target.
In the embodiment of the invention, considering that the number of network structure superparameters in the network model is smaller than that of the conventional parameters, convergence is faster when iterative updating is performed, so that more iterations can be used to update the conventional parameters in the network structure, preferably, for a given initial network model, after 2000 iterations of updating the conventional parameters, the network structure superparameters are updated for 10 iterations, and at this time, the value of S is 2000 and the value of t is 10.
In the embodiment of the invention, when updating the conventional parameters in the network model, any mode in the related technology can be adopted for updating, for example, a training sample can be obtained, the characteristic data of the training sample is extracted, the initial network model is trained based on the characteristic data of the training sample, and the conventional parameters are updated through a training process. It should be noted that, according to the purpose of the target network model, a training sample for model training may be selected, and corresponding feature data may be extracted, for example, when the target network model is used for category recognition, the feature data is a category label of the training sample.
Considering that the network structure super-parameter updating is manually realized, the cost is higher, the network model is an integral body, the number of output channels of each layer, the resolution of an input image and the network depth are considered cooperatively, the existing algorithm cannot be considered coordinately, so that the existing algorithm cannot obtain the optimal network structure super-parameter combination.
That is, for a given initial network model, the network structure is determined (i.e., the manner of connection is determined), the type of operation for each layer is also determined (e.g., the type of operation is 1x1 convolution or 3x3 convolution), its network structure superparameters (number of output channels per layer, resolution of input image and depth of network) are adjusted by the evolution strategy, and the optimal combination of network structure superparameters that meet the target constraints is found.
Accordingly, in one embodiment provided by the present invention, as shown in fig. 2, fig. 2 is a flowchart of a network structure super parameter updating process according to an embodiment of the present invention, and the method may include the following steps: step 201, step 202, step 203, step 204, step 205, step 206 and step 207, wherein,
In step 201, extracting an initial network structure superparameter a of a preset initial network model 1 、a 2 ,…,a N The method comprises the steps of carrying out a first treatment on the surface of the Wherein a is i For the i-th initial network structure superparameter in the preset initial network model, i is more than or equal to 1 and less than or equal to N, wherein N is the number of the initial network structure superparameters in the preset initial network model.
In the embodiment of the invention, when the network structure superparameter is updated, all network structure superparameters in a preset initial network model are firstly extracted, namely all initial network structure superparameters are extracted.
In one example, the initial network model is a network model constructed based on mobiletv 1, and if there are 13 convolution layers in mobiletv 1, the number of output channels of each layer of the 13 convolution layers, the resolution of the input image of the entire network model, and the total depth of the network model are extracted in this step, that is, 15 network structure superparameters are extracted.
In step 202, a superparameter a is super-parameterized based on the initial network structure 1 、a 2 ,…,a N Generating a netComplex super-parameter vector p, where p= (a) 1 ,a 2 ,…,a N )。
In the embodiment of the invention, after all network structure superparameters in a preset initial network model are extracted, the extracted network structure superparameters are constructed into a vector p.
In step 203, based on the network structure superparameter vector p, a directional derivative g for updating the network structure superparameter is generated P The method comprises the steps of carrying out a first treatment on the surface of the Wherein,loss is a preset loss function, and Δp is the disturbance quantity.
In the embodiment of the invention, the optimal values of the network structure superparameter can be obtained by a gradient descent algorithm in a derivation mode, but the derivative g of the network structure superparameter needs to be constructed in consideration of the fact that the network structure superparameter can not be directly derived like the conventional parameter value in a network model P In particular, the derivative g of the network structure super-parameter can be constructed by adopting the definition of the derivative P 。
In step 204, a disturbance variable set { Δp } is obtained 1 ,△p 2 ,…,△p M -a }; wherein Δp j And j is more than or equal to 1 and less than or equal to M, which is the number of the disturbance variables in the disturbance variable set.
In the embodiment of the invention, the derivative g of the network structure super-parameter is obtained P Afterwards, some slight disturbance quantity { DELTAp can be added to the initial network structure super-parameters 1 ,△p 2 ,…,△p M And deviating from the original value p to look at the influence of the disturbance, wherein the disturbance quantity Δp j As a vector, different perturbation amounts represent different update directions.
Preferably, in the embodiment of the present invention, in order to ensure the reliability of the update direction, 100 different disturbance amounts may be randomly selected, that is, the disturbance amount set is { Δp 1 ,△p 2 ,…,△p 100 }。
In step 205, disturbance basedQuantity set { Δp 1 ,△p 2 ,…,△p M Sum of directional derivatives g P Generating a set of directional derivatives { g P1 ,g P2 ,…,g PM }；
In an embodiment of the present invention,g Pj is the j-th directional derivative in the set of directional derivatives.
In the embodiment of the invention, a few slight disturbance { DELTAp is added to the initial network structure super-parameters 1 ,△p 2 ,…,△p M After each disturbance, the direction derivative corresponding to each disturbance can be obtained, and then the direction derivative is added to a direction derivative set { g } P1 ,g P2 ,…,g PM }。
In one example, if 100 perturbation amounts { Δp are included in the perturbation amount set 1 ,△p 2 ,…,△p 100 Then 100 directional derivatives { g } are also included in the directional derivative set P1 ,g P2 ,…,g P100 }。
In step 206, the set of directional derivatives { g } is based on a preset gradient descent algorithm P1 ,g P2 ,…,g PM Processing to obtain the network structure super-parameter vector p relative to { DELTAp 1 ,△p 2 ,…,△p M Update direction set { D } 1 ,D 2 ,…,D M -a }; wherein D is j P is relative to Δp j Is updated in the update direction.
In the embodiment of the invention, the direction derivative set { g) P1 ,g P2 ,…,g PM After } the gradient descent algorithm is adopted, the super-parameter vector p of the network structure relative to the { [ delta ] p can be obtained 1 ,△p 2 ,…,△p M Update direction set { D } 1 ,D 2 ,…,D M }。
In step 207, based on the update direction set { D ] 1 ,D 2 ,…,D M And determining the super-parameters of the target network structure.
In one embodiment of the present invention, the average value operation may be performed by selecting the update direction that partially satisfies the condition, so as to obtain a more reliable gradient descent direction, and further obtain a more reliable target network structure superparameter, where step 207 may include the following steps (not shown in the figure): step 2071 and step 2071, wherein,
In step 2071 { D } is determined 1 ,D 2 ,…,D M An update direction satisfying a preset condition in };
in step 2072, a mean value of the network structure superparameter corresponding to the update direction satisfying the preset condition is calculated, and the mean value is determined as the target network structure superparameter.
In one example, the update direction satisfying the preset condition includes: d (D) 1 ，D 2 And D 3 Due to D 1 The corresponding disturbance quantity is delta p 1 ，D 2 The corresponding disturbance quantity is delta p 2 ，D 3 The corresponding disturbance quantity is delta p 3 Thus D 1 The corresponding network structure super parameter is (p+ [ delta ] p 1 )，D 2 The corresponding network structure super parameter is (p+ [ delta ] p 2 )，D 3 The corresponding network structure super parameter is (p+ [ delta ] p 3 ) The superparameter of the target network structure is { p+ (DELTAp) 1 +△p 2 +△p 3 )/3}。
In another embodiment provided by the present invention, the step 207 may include the following steps (not shown in the figure): step 2073 and step 2074, wherein,
in step 2073 { D } is determined 1 ,D 2 ,…,D M A particular update direction in };
in step 2074, the average value of the network structure superparameters corresponding to the specific update directions is determined as the target network structure superparameters.
In one example, the specific update direction is satisfied as D4, since D 4 The corresponding disturbance quantity is delta p 4 Thus D 4 The corresponding network structure super parameter is (p+ [ delta ] p 4 ) The super parameter of the target network structure is (p+Deltap) 4 )。
Therefore, in the embodiment of the invention, the network structure super-parameters can be automatically learned, so that the proper network structure super-parameter combination can be found with low cost under the given network model and limitation conditions, and the precision is ensured.
In the embodiment of the invention, the target network model can be used for any of the following purposes: for determining the category to which the image to be processed belongs, for identifying a face in the image to be processed, for detecting a specific object in the image to be processed, for segmenting the specific object in the image to be processed, and for generating a new image, wherein the new image has similar features as the image to be processed.
As can be seen from the above embodiments, in this embodiment, when the image processing model for implementing a specific application is trained based on the initial network model and the given limiting condition, for the network structure superparameter in the network model, automatic iterative update can be performed based on the preset evolution policy, so as to reduce the cost of the model training process, improve the model training efficiency, further reduce the cost of the whole image processing process, and improve the image processing efficiency. In addition, the network structure super-parameters and the conventional parameters in the network model are updated alternately, so that the accuracy of the final model obtained through training can be ensured, and the image processing accuracy and efficiency are further improved.
Next, an image processing method provided by an embodiment of the present invention will be described.
It should be noted that, the image processing method provided by the embodiment of the present invention is applicable to an electronic device, and in practical application, the electronic device may include: mobile terminals such as smartphones, tablet computers, personal digital assistants, and the like, may also include: the embodiments of the present invention are not limited to computer devices such as notebook/desktop computers, servers, and the like.
Fig. 3 is a flowchart of an image processing method according to an embodiment of the present invention, which performs image processing based on the target network model in the embodiment shown in fig. 1, and as shown in fig. 3, the method may include the steps of: step 301, step 302, step 303 and step 304, wherein,
in step 301, an image to be processed is received.
In step 302, the image to be processed is converted into input data matching the target network model.
Considering that the format requirements of the network models for different purposes are different, for example, some network models require that the input data is an image with a specific resolution, some network models require that the input data is an image with a specific size, some network models require that the input data is a feature map of an original image, and the like, in this case, in the embodiment of the present invention, the image to be processed can be converted into the input data corresponding to the target network model so as to adapt to the target network model.
In step 303, the input data is input to the target network model for processing, so as to obtain an output result of the target network model.
In step 304, the output result of the target network model is determined as an image processing result of the image to be processed.
In the embodiment of the invention, when the target network model is a network model for determining the category to which the image to be processed belongs, the image processing result of the image to be processed is the category to which the image to be processed belongs; for example, it is recognized whether an image to be processed belongs to an animal or a plant, or it is recognized to which variety of animal the image to be processed belongs, or the like.
In the embodiment of the invention, when the target network model is a network model for identifying a face in an image to be processed, the image processing result of the image to be processed is the face identification result of the image to be processed; for example, it is recognized whether the face in the image to be processed is the face of a specific user or some feature of the face in the image to be processed.
In the embodiment of the invention, when the target network model is a network model for detecting a specific object in an image to be processed, the image processing result of the image to be processed is a detection result of the specific object in the image to be processed; for example, a defect of a product in an image to be processed is detected, or a specific user in an image to be processed is detected.
In the embodiment of the invention, when the target network model is a network model for dividing a specific object in an image to be processed, the image processing result of the image to be processed is a dividing result of the specific object in the image to be processed; for example, lane lines in the image to be processed are segmented.
In the embodiment of the invention, when the target network model is a network model for generating a new image, the image processing result of the image to be processed is the new image, wherein the new image has similar characteristics to the image to be processed; for example, when the image to be processed is a character image, a new character similar to the character style is generated.
As can be seen from the above embodiments, in this embodiment, when the image processing model for implementing a specific application is trained based on the initial network model and the given limiting condition, for the network structure superparameter in the network model, automatic iterative update can be performed based on the preset evolution policy, so as to reduce the cost of the model training process, improve the model training efficiency, further reduce the cost of the whole image processing process, and improve the image processing efficiency. In addition, the network structure super-parameters and the conventional parameters in the network model are updated alternately, so that the accuracy of the final model obtained through training can be ensured, and the image processing accuracy and efficiency are further improved.
Fig. 4 is a block diagram of a model training apparatus according to an embodiment of the present invention, and as shown in fig. 4, a model training apparatus 400 may include: an acquisition module 401 and a training module 402, wherein,
an obtaining module 401, configured to obtain a preset initial network model and a target defining condition, where the preset initial network model includes an initial network structure super parameter and an initial regular parameter;
the training module 402 is configured to perform iterative updating of the network structure super-parameter and the conventional parameter of the network model based on a preset alternate updating mode, the initial network structure super-parameter and the initial conventional parameter under the target limiting condition until the network structure super-parameter and the conventional parameter of the network model are converged, so as to obtain a target network model;
wherein, the preset alternate updating mode comprises the following steps: and each time of iterative updating of the conventional parameters is carried out, iterative updating of the super parameters of the network structure is carried out for T times based on a preset evolution strategy, T is smaller than S, and the target network model comprises the super parameters of the network structure and the conventional parameters of the target.
As can be seen from the above embodiments, in this embodiment, when the image processing model for implementing a specific application is trained based on the initial network model and the given limiting condition, for the network structure superparameter in the network model, automatic iterative update can be performed based on the preset evolution policy, so as to reduce the cost of the model training process, improve the model training efficiency, further reduce the cost of the whole image processing process, and improve the image processing efficiency. In addition, the network structure super-parameters and the conventional parameters in the network model are updated alternately, so that the accuracy of the final model obtained through training can be ensured, and the image processing accuracy and efficiency are further improved.
Alternatively, as an embodiment, the training module 402 may include: the network structure super-parameter updating sub-module may include:
an extracting unit, configured to extract an initial network structure superparameter a of the preset initial network model 1 、a 2 ,…,a N Wherein a is i The i-th initial network structure superparameter in the preset initial network model is equal to or more than 1 and equal to or less than N, wherein N is the number of the initial network structure superparameters in the preset initial network model;
a first generation unit for based on the a 1 、a 2 ,…,a N Generating a network structure super-parameter vector p, wherein p= (a) 1 ,a 2 ,…,a N )；
A second generation unit for generating a directional derivative g for updating the network structure super-parameters based on the p P Wherein, the method comprises the steps of, wherein,loss is a preset loss function, and Deltap is disturbance quantity;
an acquisition unit for acquiring a disturbance variable set { DELTAp 1 ,△p 2 ,…,△p M -wherein Δp j For the disturbance quantity setJ is not less than 1 and not more than M, M is the number of the disturbance variables in the disturbance variable set;
a third generation unit for generating a third generation result based on the { Δp 1 ,△p 2 ,…,△p M And said g P Generating a set of directional derivatives { g P1 ,g P2 ,…,g PM And } wherein,g Pj a j-th directional derivative of the set of directional derivatives;
A processing unit for performing a gradient descent algorithm on the { g }, based on a preset gradient descent algorithm P1 ,g P2 ,…,g PM Processing to obtain said p with respect to said { DELTAp 1 ,△p 2 ,…,△p M Update direction set { D } 1 ,D 2 ,…,D M }, wherein D j For said p relative to said Δp j Is updated in the update direction;
a determining unit for based on { D } 1 ,D 2 ,…,D M And determining the super-parameters of the target network structure.
Alternatively, as an embodiment, the determining unit may include:
a first determination subunit for determining the { D } 1 ,D 2 ,…,D M An update direction satisfying a preset condition in };
a calculating subunit, configured to calculate a mean value of network structure super parameters corresponding to the update direction that satisfies the preset condition;
and the second determination subunit is used for determining the average value as the target network structure super-parameter.
Alternatively, as an embodiment, the network structure super parameter may include: the number of output channels per layer in the network model, the resolution of the input image of the network model, and the network depth of the network model.
Alternatively, as an embodiment, the target network model may be used for any of the following purposes:
for determining a category to which the image to be processed belongs, for identifying a face in the image to be processed, for detecting a specific object in the image to be processed, for segmenting the specific object in the image to be processed, and for generating a new image, wherein the new image has similar features as the image to be processed.
Alternatively, as an embodiment, the target defining condition may include any one of the following:
the total parameter number of the network model is lower than a preset number threshold, the total calculated amount of the network model is lower than a preset calculated amount threshold, and the running time of the network model on the specific equipment is lower than a preset time threshold.
Fig. 5 is a block diagram of an image processing apparatus according to an embodiment of the present invention, and as shown in fig. 5, the image processing apparatus 500 may include: a receiving module 501, an input module 502, a processing module 503, and a determining module 504, wherein,
a receiving module 501, configured to receive an image to be processed;
an input module 502, configured to convert the image to be processed into input data matched with the target network model;
a processing module 503, configured to input the input data to the target network model for processing, so as to obtain an output result of the target network model;
and the determining module 504 is configured to determine an output result of the target network model as an image processing result of the image to be processed.
As can be seen from the above embodiments, in this embodiment, when the image processing model for implementing a specific application is trained based on the initial network model and the given limiting condition, for the network structure superparameter in the network model, automatic iterative update can be performed based on the preset evolution policy, so as to reduce the cost of the model training process, improve the model training efficiency, further reduce the cost of the whole image processing process, and improve the image processing efficiency. In addition, the network structure super-parameters and the conventional parameters in the network model are updated alternately, so that the accuracy of the final model obtained through training can be ensured, and the image processing accuracy and efficiency are further improved.
Alternatively, as an embodiment, the image processing result may include any one of the following:
the method comprises the steps of determining the category of an image to be processed, the face recognition result of the image to be processed, the detection result of a specific object in the image to be processed, the segmentation result of the specific object in the image to be processed and a new image, wherein the new image has similar characteristics to the image to be processed.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
According to still another embodiment of the present invention, there is provided an electronic apparatus including: a memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed by the processor, performs the steps of the model training method as described in any one of the embodiments above.
According to still another embodiment of the present invention, there is provided an electronic apparatus including: a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor performs the steps of the image processing method according to any one of the embodiments described above.
According to yet another embodiment of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the model training method according to any of the embodiments described above.
According to still another embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image processing method according to any one of the embodiments described above.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The foregoing has described in detail the model training and image processing method, apparatus, electronic device and storage medium provided by the present invention, and specific examples have been applied to illustrate the principles and embodiments of the present invention, and the above examples are only used to help understand the method and core idea of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (7)
1. An image processing method, the method comprising:
receiving an image to be processed;
converting the image to be processed into input data matched with a target network model;
inputting the input data into the target network model for processing to obtain an output result of the target network model;
determining an output result of the target network model as an image processing result of the image to be processed;
the method further comprises the steps of:
acquiring a preset initial network model and a target limiting condition, wherein the preset initial network model comprises an initial network structure super-parameter and an initial conventional parameter;
Under the target limiting condition, based on a preset alternate updating mode, the initial network structure super-parameters and the initial conventional parameters, carrying out iterative updating on the network structure super-parameters and the conventional parameters of the network model until the network structure super-parameters and the conventional parameters of the network model are converged to obtain the target network model;
wherein, the preset alternate updating mode comprises the following steps: performing iterative updating of the super-parameters of the T times of network structures based on a preset evolution strategy every time of iterative updating of the S times of conventional parameters, wherein T is less than S, and the target network model comprises the super-parameters of the target network structures and the target conventional parameters;
the network structure super-parameters include: the number of output channels per layer in the network model, the resolution of the input image of the network model, and the network depth of the network model.
2. The method of claim 1, wherein the updating of the network fabric superparameter comprises:
extracting an initial network structure superparameter a of the preset initial network model 1 、a 2 ,…,a N Wherein a is i For the preset initialThe i-th initial network structure superparameter in the network model is more than or equal to 1 and less than or equal to N, wherein N is the number of the initial network structure superparameters in the preset initial network model;
Based on the a 1 、a 2 ,…,a N Generating a network structure super-parameter vector p, wherein p= (a) 1 ,a 2 ,…,a N )；
Based on the p, generating a directional derivative g for updating the network structure super-parameters P Wherein, the method comprises the steps of, wherein,loss is a preset loss function, and Deltap is disturbance quantity;
obtaining disturbance quantity set { DELTAp 1 ,△p 2 ,…,△p M -wherein Δp j J is more than or equal to 1 and less than or equal to M, which is the number of disturbance variables in the disturbance variable set;
based on the { DELTAp 1 ,△p 2 ,…,△p M And said g P Generating a set of directional derivatives { g P1 ,g P2 ,…,g PM And } wherein,g Pj a j-th directional derivative of the set of directional derivatives;
based on a preset gradient descent algorithm, the { g } P1 ,g P2 ,…,g PM Processing to obtain said p with respect to said { DELTAp 1 ,△p 2 ,…,△p M Update direction set { D } 1 ,D 2 ,…,D M }, wherein D j For said p relative to said Δp j Is updated in the update direction;
based on { D } 1 ,D 2 ,…,D M And determining the super-parameters of the target network structure.
3. The method according to claim 2, wherein the { D } is based on 1 ,D 2 ,…,D M -determining target network structure superparameters, comprising:
determining the { D } 1 ,D 2 ,…,D M An update direction satisfying a preset condition in };
and calculating the average value of the network structure super-parameters corresponding to the updating direction meeting the preset conditions, and determining the average value as the target network structure super-parameters.
4. The method of claim 1, wherein the target defining condition comprises any one of:
the total parameter number of the network model is lower than a preset number threshold, the total calculated amount of the network model is lower than a preset calculated amount threshold, and the running time of the network model on the specific equipment is lower than a preset time threshold.
5. An image processing apparatus, characterized in that the apparatus comprises:
the receiving module is used for receiving the image to be processed;
the input module is used for converting the image to be processed into input data matched with a target network model;
the processing module is used for inputting the input data into the target network model for processing to obtain an output result of the target network model;
the determining module is used for determining the output result of the target network model as an image processing result of the image to be processed;
the device further comprises:
the acquisition module is used for acquiring a preset initial network model and target limiting conditions, wherein the preset initial network model comprises an initial network structure super-parameter and an initial conventional parameter;
the training module is used for carrying out iterative updating on the network structure super-parameters and the conventional parameters of the network model based on a preset alternate updating mode, the initial network structure super-parameters and the initial conventional parameters under the target limiting condition until the network structure super-parameters and the conventional parameters of the network model are converged to obtain a target network model;
Wherein, the preset alternate updating mode comprises the following steps: performing iterative updating of the super-parameters of the T times of network structures based on a preset evolution strategy every time of iterative updating of the S times of conventional parameters, wherein T is less than S, and the target network model comprises the super-parameters of the target network structures and the target conventional parameters;
the network structure super-parameters include: the number of output channels per layer in the network model, the resolution of the input image of the network model, and the network depth of the network model.
6. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the image processing method according to any of claims 1-4.
7. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, implements the steps in the image processing method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911097179.6A CN110942090B (en) | 2019-11-11 | 2019-11-11 | Model training method, image processing device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911097179.6A CN110942090B (en) | 2019-11-11 | 2019-11-11 | Model training method, image processing device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110942090A CN110942090A (en) | 2020-03-31 |
| CN110942090B true CN110942090B (en) | 2024-03-29 |
Family
ID=69907658
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911097179.6A Active CN110942090B (en) | 2019-11-11 | 2019-11-11 | Model training method, image processing device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110942090B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113555008A (en) * | 2020-04-17 | 2021-10-26 | 阿里巴巴集团控股有限公司 | Parameter adjusting method and device for model |
| CN112257561B (en) * | 2020-10-20 | 2021-07-30 | 广州云从凯风科技有限公司 | Human face living body detection method and device, machine readable medium and equipment |
| CN113240107A (en) * | 2021-05-08 | 2021-08-10 | 北京字跳网络技术有限公司 | Image processing method and device and electronic equipment |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103499835A (en) * | 2013-10-13 | 2014-01-08 | 中国石油集团西北地质研究所 | Method for inverting near-surface velocity model by utilizing preliminary waveforms |
| CN108256555A (en) * | 2017-12-21 | 2018-07-06 | 北京达佳互联信息技术有限公司 | Picture material recognition methods, device and terminal |
| AU2018101317A4 (en) * | 2018-09-07 | 2018-10-11 | Chen, Guoyi Mr | A Deep Learning Based System for Animal Species Classification |
| CN109740657A (en) * | 2018-12-27 | 2019-05-10 | 郑州云海信息技术有限公司 | A kind of training method and equipment of the neural network model for image data classification |
| CN109767759A (en) * | 2019-02-14 | 2019-05-17 | 重庆邮电大学 | End-to-end speech recognition methods based on modified CLDNN structure |
| DE102017128082A1 (en) * | 2017-11-28 | 2019-05-29 | Connaught Electronics Ltd. | Meta-architecture design for a CNN network |
| CN110188862A (en) * | 2019-04-12 | 2019-08-30 | 北京迈格威科技有限公司 | Searching method, the device, system of model hyper parameter for data processing |
| CN110399917A (en) * | 2019-07-24 | 2019-11-01 | 东北大学 | A kind of image classification method based on hyperparameter optimization CNN |
| CN110415233A (en) * | 2019-07-26 | 2019-11-05 | 东南大学 | Pavement crack rapid extracting method based on two step convolutional neural networks |
-
2019
- 2019-11-11 CN CN201911097179.6A patent/CN110942090B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103499835A (en) * | 2013-10-13 | 2014-01-08 | 中国石油集团西北地质研究所 | Method for inverting near-surface velocity model by utilizing preliminary waveforms |
| DE102017128082A1 (en) * | 2017-11-28 | 2019-05-29 | Connaught Electronics Ltd. | Meta-architecture design for a CNN network |
| CN108256555A (en) * | 2017-12-21 | 2018-07-06 | 北京达佳互联信息技术有限公司 | Picture material recognition methods, device and terminal |
| AU2018101317A4 (en) * | 2018-09-07 | 2018-10-11 | Chen, Guoyi Mr | A Deep Learning Based System for Animal Species Classification |
| CN109740657A (en) * | 2018-12-27 | 2019-05-10 | 郑州云海信息技术有限公司 | A kind of training method and equipment of the neural network model for image data classification |
| CN109767759A (en) * | 2019-02-14 | 2019-05-17 | 重庆邮电大学 | End-to-end speech recognition methods based on modified CLDNN structure |
| CN110188862A (en) * | 2019-04-12 | 2019-08-30 | 北京迈格威科技有限公司 | Searching method, the device, system of model hyper parameter for data processing |
| CN110399917A (en) * | 2019-07-24 | 2019-11-01 | 东北大学 | A kind of image classification method based on hyperparameter optimization CNN |
| CN110415233A (en) * | 2019-07-26 | 2019-11-05 | 东南大学 | Pavement crack rapid extracting method based on two step convolutional neural networks |
Non-Patent Citations (3)
| Title |
|---|
| Population Based Training of Neural Networks;Max Jaderberg;《arXiv》;全文 * |
| 基于人群的神经网络超参数优化的研究;朱汇龙;刘晓燕;刘瑶;;信息技术(11);全文 * |
| 遗传算法优化的神经网络频谱预测模型训练;杨健;《解放军理工大学学报（自然科学版）》;第17卷(第6期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110942090A (en) | 2020-03-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110942090B (en) | Model training method, image processing device, electronic equipment and storage medium | |
| KR102170105B1 (en) | Method and apparatus for generating neural network structure, electronic device, storage medium | |
| Bi et al. | Automatic liver lesion detection using cascaded deep residual networks | |
| US20230237088A1 (en) | Automatically detecting user-requested objects in digital images | |
| JP5506785B2 (en) | Fingerprint representation using gradient histogram | |
| CN106897746B (en) | Data classification model training method and device | |
| JP5766620B2 (en) | Object region detection apparatus, method, and program | |
| CN110506274B (en) | Object detection and representation in images | |
| KR102347496B1 (en) | Automatic multi-organ and tumor contouring system based on artificial intelligence for radiation treatment planning | |
| CN113095333B (en) | Unsupervised feature point detection method and unsupervised feature point detection device | |
| WO2010043954A1 (en) | Method, apparatus and computer program product for providing pattern detection with unknown noise levels | |
| Miclea et al. | Real-time semantic segmentation-based stereo reconstruction | |
| US20220245510A1 (en) | Multi-dimensional model shape transfer | |
| CN109271957B (en) | Face gender identification method and device | |
| Xu et al. | Robust surface segmentation and edge feature lines extraction from fractured fragments of relics | |
| CN109993026A (en) | The training method and device of relatives' identification network model | |
| CN108765445B (en) | Lung trachea segmentation method and device | |
| CN111178200B (en) | Method for identifying instrument panel indicator lamp and computing equipment | |
| CN112508966A (en) | Interactive image segmentation method and system | |
| CN112529027A (en) | Data processing method, client, device and computer readable storage medium | |
| CN114241411B (en) | Counting model processing method and device based on target detection and computer equipment | |
| CN115240843A (en) | Fairness prediction system based on structure causal model | |
| CN114240770A (en) | Image processing method, device, server and storage medium | |
| CN113282781A (en) | Image retrieval method and device | |
| CN113361530A (en) | Image semantic accurate segmentation and optimization method using interaction means |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |