CN111784787B - Image generation method and device - Google Patents
Image generation method and device Download PDFInfo
- Publication number
- CN111784787B CN111784787B CN201910646743.9A CN201910646743A CN111784787B CN 111784787 B CN111784787 B CN 111784787B CN 201910646743 A CN201910646743 A CN 201910646743A CN 111784787 B CN111784787 B CN 111784787B
- Authority
- CN
- China
- Prior art keywords
- attribute
- attribute information
- image
- sub
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000001131 transforming effect Effects 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims description 94
- 230000009466 transformation Effects 0.000 claims description 37
- 239000013598 vector Substances 0.000 claims description 23
- 238000000605 extraction Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 230000000386 athletic effect Effects 0.000 description 5
- 239000000284 extract Substances 0.000 description 5
- 230000008676 import Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application discloses an image generation method and device. One embodiment of the method comprises the following steps: extracting first attribute information of an original image; extracting second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type, and the second attribute type comprises at least two sub second attributes; transforming the second attribute information according to the ranking of the sub second attribute aiming at the second attribute type, and generating derivative attribute information of the indicating sub second attribute; and generating a derivative image of the original image according to the first attribute information and the derivative attribute information. This embodiment provides a new way of image generation.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to an image generation method and device.
Background
With the development of computer technology, images can be automatically generated by a computer. The computer automatically generated images may be applied to multiple scenes. For example, images may be generated in batches as a sample set for machine learning training; the images of all the articles of the same type can be generated through the images of part of the articles of the same type, so that the link of shooting the images for the articles of the same type one by one can be omitted.
Disclosure of Invention
The embodiment of the application provides an image generation method and device.
In a first aspect, an embodiment of the present application provides an image generating method, including: extracting first attribute information of an original image, wherein a first attribute indicated by the first attribute information belongs to a first attribute type which is predefined; extracting second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type, and the second attribute type comprises at least two sub second attributes; transforming the second attribute information according to the ranking of the sub second attribute aiming at the second attribute type, and generating derivative attribute information of the sub second attribute; and generating a derivative image of the original image according to the first attribute information and the derivative attribute information.
In a second aspect, an embodiment of the present application provides an image generating apparatus, including: a first extraction unit configured to extract first attribute information of an original image, wherein a first attribute indicated by the first attribute information belongs to a first attribute type defined in advance; a second extraction unit configured to extract second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type including at least two sub-second attributes; a transformation unit configured to transform the second attribute information according to a ranking of sub-second attributes for the second attribute type, generating derived attribute information indicating the sub-second attributes; and a generation unit configured to generate a derivative image of the original image based on the first attribute information and the derivative attribute information.
In a third aspect, an embodiment of the present application provides an image generating electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement a method as in any of the embodiments of the image generation method described above.
In a fourth aspect, embodiments of the present application provide an image generation computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method as in any of the embodiments of the image generation method described above.
The image generation method and device provided by the embodiment of the application are used for extracting the first attribute information and the second attribute information of the original image; then, according to the ranking of the sub second attribute aiming at the second attribute type, the second attribute information is transformed to generate derivative attribute information; in generating a derivative image of the original image according to the first attribute information and the derivative attribute information, the technical effects may include at least: a new way of generating an image is provided.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings, in which:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;
FIG. 2 is a flow chart of one embodiment of an image generation method according to the present application;
FIG. 3 is a schematic illustration of an application scenario of an image generation method according to the present application;
FIG. 4 is a schematic diagram of a generation network according to the present application;
FIG. 5 is a schematic diagram of a training method to generate a network according to the present application;
FIG. 6 is a schematic diagram of one implementation of step 503 in a training method according to the present application;
FIG. 7 is a flow chart of yet another embodiment of an image generation method according to the present application;
FIG. 8 is a schematic illustration of an application scenario of an image generation method according to the present application;
FIG. 9 is a schematic structural view of one embodiment of an image generation apparatus according to the present application;
fig. 10 is a schematic diagram of a computer system suitable for use in implementing embodiments of the present application.
Detailed Description
The present application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the image generation method or image generation apparatus of the present application may be applied.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 may be a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The terminal devices 101, 102, 103 may interact with the server 105 via the network 104 to receive or send messages or the like. Various communication client applications, such as shopping applications, image processing applications, instant messaging tools, etc., may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be hardware, may be electronic devices with display screens, including but not limited to smartphones, tablet computers, electronic book readers, MP3 players (Moving Picture Experts Group Audio Layer III, moving picture experts compression standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture experts compression standard audio layer 4) players, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for image processing class applications on the terminal devices 101, 102, 103. The background server can analyze and the like the received data such as the original image, generate a derivative image of the original image, and return the generated derivative image to the terminal equipment.
It should be noted that, the image generating method provided in the embodiment of the present application may be executed by the server 105, and accordingly, the image generating apparatus may be disposed in the server 105.
The server may be hardware or software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. When the electronic device on which the image generation method is operating does not require data transmission with other electronic devices, the system architecture may include only the electronic device on which the image generation method is operating.
With continued reference to FIG. 2, a flow 200 of one embodiment of an image generation method according to the present application is shown. The image generation method comprises the following steps:
in step 201, first attribute information of an original image is extracted.
In this embodiment, a first execution subject (e.g., a server shown in fig. 1) of the image generation method may extract first attribute information of an original image.
In this embodiment, the first attribute indicated by the first attribute information belongs to a first attribute type defined in advance.
In the present embodiment, the original image may have various attribute types, for example, shape, color, whether a target object is included, or the like.
Here, the first attribute type may be defined from among attribute types of the original image. As an example, a shape may be selected as the first attribute type. The first attribute information may indicate a square; the first attribute indicated by the first attribute information, which is "square", belongs to a predefined first attribute type "shape".
In this embodiment, the first attribute information may indicate a first attribute. As an example, the first attribute information may indicate a moon shape, a shoe shape, a clothing shape, and the like.
Step 202, extracting second attribute information of the original image.
In this embodiment, the first execution body may extract the second attribute information of the original image.
In this embodiment, the second attribute indicated by the second attribute information belongs to a predefined second attribute type, and the second attribute type may include at least two sub second attributes.
Here, the second attribute type may be defined from among attribute types of the original image. As an example, a color may be defined as the second attribute type.
Here, the second attribute information may indicate a second attribute. The second attribute type may be at least two sub second attributes. As an example, if the second attribute type is color, the sub-second attribute may be red, green, purple, and the like.
The execution sequence of step 201 and step 202 may be executed simultaneously, or step 201 may be executed first and then step 202 may be executed, or step 202 may be executed first and then step 201 may be executed.
Step 203, transforming the second attribute information according to the ranking of the sub second attribute for the second attribute type, and generating derivative attribute information indicating the sub second attribute.
In this embodiment, the first execution body may transform the second attribute information according to the ranking of the sub-second attribute for the second attribute type, and generate the derivative attribute information indicating the sub-second attribute.
In this embodiment, since the derived attribute information is obtained by transforming the second attribute information, the derived attribute information and the second attribute information may be different. The second attribute information and the derived attribute information may indicate different sub-second attributes in the second attribute type.
As an example, the second attribute information may indicate a sub-second attribute of red. The derived attribute information may indicate a sub-second attribute of green. The sub-second attribute indicated by the second attribute information may be different from the sub-second attribute indicated by the derivative attribute information.
Here, a ranking for sub-second attributes of the second attribute type may be generated in advance, and then derived attribute information of the above-described second attribute information may be generated according to the ranking.
As an example, determining whether the sub-second attribute indicated by the second attribute information is located in the first bit of the rank; if not, transforming the second attribute information to generate information indicating sub second attributes positioned in the first ranking bit so as to obtain derivative attribute information; if so, the second attribute information is transformed to generate information of sub second attributes of the second bit of the rank to obtain derivative attribute information.
In some embodiments, the original image is an image of a first item. The ranking described above may be generated by: acquiring a second object image set of a second object, wherein the first object and the second object belong to the same object type, the second object image set comprises at least two kinds of second object images, and different kinds of second object images have different sub second attributes; and generating a ranking of the sub second attribute of the second attribute type according to the pre-generated scores of the second object images of the various types.
As an example, the first item is a men's slipper. The second article is a men's athletic shoe. The second article image set includes a red men's athletic shoe image and a green men's athletic shoe image. The second attribute type may be a color and red and green may be different sub-second attributes. A first score may be obtained for a red men's athletic shoe image and a second score may be obtained for a green men's athletic shoe image. Then, the two sub-second attributes of red and green are ranked according to the first score and the second score, resulting in a ranking for the sub-second attributes.
As an example, the acquisition of the first score and the second score may be achieved in various ways. By utilizing the method, the questionnaire can be developed for the user to obtain scores; the sales of red men's shoes may also be given as a first score and the sales of green men's shoes as a second score.
In some embodiments, the ranking includes sub-second attribute feature information and a rank corresponding to the sub-second attribute feature information. The step 203 may include: and transforming the second attribute information according to the sub second attribute characteristic information corresponding to the bit number positioned in the preset number of bits in the ranking, so as to generate the derivative attribute information.
Step 204, generating a derivative image of the original image according to the first attribute information and the derivative attribute information.
In this embodiment, the first execution body may generate the derivative image of the original image based on the first attribute information and the derivative attribute information.
Here, the generated derivative image may have the first attribute indicated by the first attribute information, or may have the sub-second attribute indicated by the derivative attribute information. It will be appreciated that the first attribute information and the derived attribute information may be combined to generate a derived image. As an example, the first attribute information indicates a triangle, the derivative attribute information indicates red, and the triangle and red may be combined to generate a red triangle as a derivative image.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the image generating method according to the present embodiment. As shown in fig. 3:
first, the server may acquire an original image 301.
Then, the server may extract the first attribute information 302 of the original image. And, second attribute information 303 of the original image is extracted.
The server may then transform the second attribute information according to the ranking 304 for the sub-second attribute of the second attribute type, generating derived attribute information 305 indicating the sub-second attribute as described above.
Finally, the server may generate a derived image 306 of the original image based on the first attribute information 302 and the derived attribute information 305.
The method shown in this embodiment is performed by extracting first attribute information and second attribute information of an original image; then, according to the ranking of the sub second attribute aiming at the second attribute type, the second attribute information is transformed to generate derivative attribute information; in generating a derivative image of the original image according to the first attribute information and the derivative attribute information, the technical effects may include at least:
first, a new way of generating images is provided.
Secondly, converting the second attribute information according to the ranking to generate the derivative attribute information; and generating a derivative image based on the derivative attribute information. Thus, a derivative image may be generated based on the ranking; ranking may be considered as a guideline to the transformation direction of the original image, whereby the generation of a derived image according to the desired transformation direction may be achieved.
In some embodiments, the above method may further comprise: training to generate a network.
In some embodiments, the generation network may include at least one of, but is not limited to, a first encoding model, a second encoding model, an attribute transformation model, and a generation model. Referring to fig. 4, a schematic structure of a generation network is shown. Wherein the output of the first coding model is in communication with the input of the generative model. The output of the second encoding model is in communication with the input of the attribute transformation model. The output of the attribute transformation model is in communication with the input of the generative model. Here, the input of the first coding model and the input of the second coding model are the same image, the input of the attribute conversion model is the output of the second coding model, the input of the generation model is the output of the first coding model and the attribute conversion model, and the output of the generation model is the derivative image.
It should be noted that, each model in the present application may be a model based on a neural network. The neural network may refer to an artificial neural network (Artificial Neural Network, ANN). A neural network is typically an operational model, consisting of a large number of nodes (or neurons) interconnected. Each node may represent a specific output function, called an excitation function (activation function). The connection between each two nodes represents a weight, called a weight, for the signal passing through the connection, which corresponds to the memory of the artificial neural network. Herein, the neural network may refer to an artificial neural network. Common neural networks include, for example, deep neural networks (Deep Neural Network, DNN), convolutional neural networks (Convolutional Neural Network, CNN), recurrent neural networks (Recurrent Neural Network, RNN), and the like.
Here, it may be preset which layers the neural network includes (e.g., convolutional layer, pooled layer, fully connected layer, classifier, etc.), the connection order relationship between layers, and which network parameters each layer includes (e.g., weights, bias terms, step sizes of convolutions), etc. Wherein a convolution layer may be used to extract image features. How many convolution kernels are can be determined for each convolution layer, the size of each convolution kernel, the weight of the individual neurons in each convolution kernel, the bias term corresponding to each convolution kernel, the step size between two adjacent convolutions, and so forth. The network parameters of the neural network described above may be adjusted using a back propagation algorithm (Back Propagation Algorithm, BP algorithm) and a gradient descent method (e.g., a random gradient descent algorithm) to train the model.
It should be noted that, the training model in the present application may include updating network parameters of the model.
In some embodiments, the constructed generation network may be trained by:
referring to fig. 5, a flow 500 of training a generating network is shown, wherein the flow 500 may include:
step 501, importing at least one training derivative image generated by a generation model in a generation network into a pre-established discrimination model to generate a discrimination result.
Here, the second execution body of the flow 500 may be the same as or different from the first execution body of the flow 200.
Here, the second execution body may introduce at least one training derivative image generated by the generation network into a previously constructed discrimination model to generate a discrimination result.
Here, the above-described discrimination result may be used to indicate whether the training derived image is a true image.
Here, generating the attribute transformation model in the network may output at least one training derived attribute information. Each training derivative image of the at least one training derivative image may correspond to training derivative attribute information of training derivative attribute information.
In some embodiments, the discrimination results generated by the discrimination model may be indicative of at least one of, but not limited to: the derived image is a probability of whether or not the real image is a real image and whether or not the derived image is a real image.
Step 502, importing at least one training derivative image and rank into a pre-established ranking model to generate a rank recognition result for the training derivative image.
Here, the second execution body may import at least one training derivative image into a pre-trained ranking model to generate the rank recognition result. Here, the rank recognition result generated by the ranking model may indicate the ranks of the sub-second attributes of the respective training derivative images in the ranking.
Here, the ranking may include sub-second attribute feature information and corresponding rank. The sub-second attribute feature information may indicate a sub-second attribute (e.g., red, green, or purple, etc.). As an example, ranking may include: red characteristic information, 1; green characteristic information, 2; purple characteristic information, 3. The ranking in this example may represent: the sub-second attribute of red is located at the first position of the rank, the sub-second attribute of green is located at the second position of the rank, and the sub-second attribute of purple is located at the third position of the rank.
In some embodiments, the ranking model may determine sub-second attributes that each training derived image has based on the sub-second attribute feature information in the ranking; then, the rank recognition result is generated according to the rank of the sub-second attribute of each training derivative image in the rank. As an example, the ranking order of the sub-second attribute in the ranking of the target training derivative image, which is imported into the ranking model, may be used as the ranking recognition result.
Step 503, generating network parameters of the update generation network based on the discrimination result and the rank identification result.
Here, the second execution body may generate and update the network parameter of the generation network based on the discrimination result and the bit rate identification result.
In some embodiments, the above-described discriminant model may be trained by: the training real images in the pre-established training real image set can be imported into the discrimination model to generate a training discrimination result; and updating the network parameters of the discrimination model according to the discrimination result for training. Training false images in a pre-established training false image set can be imported into the discrimination model to generate a training discrimination result; and updating the network parameters of the discrimination model according to the discrimination result for training.
It should be noted that, training the generated network by using the ranking model and the discrimination model may at least include: firstly, ensuring that the generated image is true and reasonable; second, ensure that the derived image generated has a particular attribute, which may be the top ranking sub-second attribute.
In some embodiments, the above-described process 500 may further include: importing the training image into a second coding model in a generating network to generate training second attribute information; importing the generated training second attribute information into an attribute transformation model to generate at least one training derivative attribute information; acquiring random vectors from a pre-established image vector set; and importing the obtained random vector and training derivative attribute information into a generation model to generate a training derivative image.
It should be noted that, for the generation model, the vector input to the generation model may be a random vector in a random vector space.
In some embodiments, the above set of image vectors is established by: the training image is imported into a first coding model, training first attribute information is generated, and the generated training first attribute information is taken as a set element to establish an image vector set.
The method includes the steps that the first coding model is utilized, an image vector is generated based on a training image, an image vector set is established, then the image vector (or first attribute information) in the image vector set is imported into the generation model, random vectors acquired by the generation model can be trained based on the training image, the similarity between a derivative image which can be generated by the trained generation model and an original image is high, and the derivative image is generated based on the original image.
Referring to fig. 6, which illustrates an exemplary flow 503 of one implementation of step 503, flow 503 may include:
step 5031, updating network parameters of at least one of the following according to the bit number identification result: attribute transformation models and generative models.
In some embodiments, determining whether the sub-second attribute of the training derived image is located in the first preset number of bits in the rank according to the bit recognition result; and adjusting network parameters of the attribute transformation model and/or the generation model in response to the training derived image having sub-second attributes not located in the top preset number of digits in the rank. In other words, if the rank recognition result indicates that the sub-second attribute corresponding to the derived image is not the sub-second attribute of the first preset number of bits (e.g., the first one or the first three bits) in the rank, the attribute transformation model and/or the network parameters of the generation model are adjusted.
As an example, the training derivative image generated by the generating model is one, the training derivative image can be imported into the ranking model, and the rank of the sub second attribute of the training derivative image in the ranking is determined; if the determined rank is not the first rank, adjusting the attribute transformation model and/or generating network parameters of the model; if the determined rank is the first rank, the generation network is not adjusted based on the rank identification result (it should be noted that even if the generation network is not adjusted based on the rank identification result, the generation network may be adjusted based on the discrimination result).
It should be noted that, by using the ranking model, selection of the sub-second attribute may be achieved, and the generating network may be updated based on the rank recognition result to enable the generating network to generate the derivative image according to the ranking, with respect to whether the generated image matches the selected sub-second attribute.
Step 5032, updating network parameters of at least one of the following according to the discrimination result: a first encoding model, a second encoding model, and a generation model.
Here, the step 5032 may include: and (3) indicating the training derivative image to be a false image by the judging result, and adjusting network parameters of at least one of the following: the first coding model, the second coding model and the generating model; if the discrimination result indicates that the training derived image is a true image, the generation network is not adjusted based on the discrimination result (it is to be noted that even if the generation network is not adjusted based on the discrimination model, the generation network may be adjusted based on the rank recognition result).
It should be noted that, by using the discrimination result of the discrimination model, the generated network is trained, so that the generated image is as true and reasonable as possible.
With further reference to fig. 7, a flow 700 of yet another embodiment of an image generation method is shown. The flow 700 of the image generation method includes the steps of:
step 701, importing an original image into a first coding model, and extracting first attribute information of the original image.
In this embodiment, the first execution subject (e.g., the server shown in fig. 1) of the image generation method may import the original image into the first coding model, and extract the first attribute information of the original image.
Here, the first coding model is used to characterize a correspondence between the image and the attribute information of the first attribute type.
Alternatively, the first coding model may be the first coding model in the generation network, or may be a model that is independently trained with respect to the generation network.
Step 702, importing the original image into a second coding model to generate second attribute information of the original image.
In this embodiment, the first execution body may import the original image into a second encoding model to generate second attribute information of the original image.
Here, the second coding model is used to characterize a correspondence between the image and the attribute information of the second attribute type.
Alternatively, the second coding model may be the first coding model in the generation network, or may be a model that is independently trained with respect to the generation network.
Step 703, importing the second attribute information into an attribute transformation model to generate derivative attribute information of the second attribute information.
In this embodiment, the first executing body may import the second attribute information into the attribute transformation model, and generate derivative attribute information of the second attribute information.
Here, the above-described attribute transformation model is used to characterize correspondence between attribute information.
Alternatively, the attribute transformation model may be an attribute transformation model in the generation network, or may be a model independently trained with respect to the generation network.
Step 704, importing the first attribute information and the derived attribute information into a generation model to generate a derived image.
In this embodiment, the first execution body may import the first attribute information and the charity attribute information into a generation model to generate the derivative image.
Here, the above-described generation model is used to characterize the correspondence between the attribute information and the image.
Alternatively, the generated model may be a generated model in the generated network, or may be a model independently trained with respect to the generated network.
As can be seen from fig. 7, compared with the embodiment corresponding to fig. 2, the flow 700 of the image generating method in this embodiment highlights the step of generating the derivative image of the original image using the trained generation network, and thus, the technical effects of the solution described in this embodiment may at least include:
first, a new way of generating images is provided.
Secondly, the derivative images can be generated in batches through a trained generation network, so that the image generation efficiency can be improved.
Third, by ranking as a goal, a high quality derivative image is automatically generated.
Please refer to fig. 8, which is a schematic diagram illustrating the generation of a derivative image using a generating network. In fig. 8, the original image 801 may be imported into a first encoding model and a second encoding model, where the original image 801 may be a red shoe image. The first coding model outputs first attribute information z, and the second coding model outputs second attribute information y. The second attribute information may be imported into an attribute transformation model that generates the first derived attribute information y 1 Second derived attribute information y 2 And third derived attribute information y 3 . Will be z and y 1 、y 2 、y 3 The first derivative image 802, the second derivative image 803, and the third derivative image 804 are generated by importing the first derivative image and the third derivative image into the generation model. Here, the first derivative image 801 may be a green shoe image, the second derivative image 803 may be a purple shoe image, and the third derivative image 804 may be a white shoe image.
With further reference to fig. 9, as an implementation of the method shown in the foregoing figures, the present application provides an embodiment of an image generating apparatus, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the embodiment of the apparatus may further include the same or corresponding features as the embodiment of the method shown in fig. 2, except for the features described below. The device can be applied to various electronic equipment.
As shown in fig. 9, the image generating apparatus 900 of the present embodiment includes: a first extraction unit 901, a second extraction unit 902, a transformation unit 903, and a generation unit 904. A first extraction unit configured to extract first attribute information of an original image, wherein a first attribute indicated by the first attribute information belongs to a first attribute type defined in advance; a second extraction unit configured to extract second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type including at least two sub-second attributes; a transformation unit configured to transform the second attribute information according to a ranking of sub-second attributes for the second attribute type, generating derived attribute information indicating the sub-second attributes; and a generation unit configured to generate a derivative image of the original image based on the first attribute information and the derivative attribute information.
In this embodiment, the specific processes and the technical effects of the first extraction unit 901, the second extraction unit 902, the transformation unit 903, and the generation unit 904 of the image generating apparatus 900 may refer to the relevant descriptions of the steps 201, 202, 203, and 204 in the corresponding embodiment of fig. 2, and are not repeated herein.
In some embodiments, the first extraction unit is further configured to: importing the original image into the first coding model, and extracting first attribute information of the original image; and the second extraction unit is further configured to: and importing the original image into the second coding model, and extracting second attribute information of the original image.
In some embodiments, the above-mentioned transforming unit is further configured to: importing second attribute information into the attribute transformation model to generate derivative attribute information of the second attribute information; and the generation unit is further configured to: and importing the first attribute information and the derivative attribute information into the generation model to generate a derivative image of the original image.
In some embodiments, the apparatus further comprises: a training unit (not shown) configured to: training a generating network, wherein the generating network comprises a first coding model, a second coding model, an attribute transformation model and a generating model, wherein the input of the first coding model and the input of the second coding model are images, the input of the attribute transformation model is the output of the second coding model, the input of the generating model is the output of the first coding model and the attribute transformation model, and the output of the generating model is a derivative image of the images.
In some embodiments, the training unit is further configured to: importing at least one training derivative image generated by a generation model in the generation network into a pre-established discrimination model to generate a discrimination result, wherein the discrimination result is used for indicating whether the training derivative image is a real image or not; importing at least one training derivative image and the ranking into a pre-established ranking model to generate a rank recognition result for the training derivative image, wherein the rank recognition result is used for indicating the rank of the sub second attribute of the training derivative image in the ranking; and updating the network parameters of the generated network based on the discrimination result and the bit rate recognition result.
In some embodiments, the training unit is further configured to: importing the training image into a second coding model in a generating network to generate training second attribute information; importing the generated training second attribute information into an attribute transformation model to generate at least one training derivative attribute information; acquiring random vectors from a pre-established image vector set; and importing the obtained random vector and training derivative attribute information into a generation model to generate a training derivative image.
In some embodiments, the above set of image vectors is established by: the training image is imported into a first coding model, training first attribute information is generated, and the generated training first attribute information is taken as a set element to establish an image vector set.
In some embodiments, the ranking model described above is used to: determining sub-second attributes of each training derivative image according to the sub-second attribute characteristic information in the ranking; and generating a rank recognition result according to the ranks of the sub second attributes of the training derivative images in the ranks.
In some embodiments, the training unit is further configured to: updating network parameters of at least one of the following according to the bit rate identification result: an attribute transformation model and the generation model; updating network parameters of at least one of the following according to the discrimination result: the first coding model, the second coding model, and the generation model.
In some embodiments, the training unit is further configured to: determining whether the sub second attribute of the training derivative image is positioned at the preset number of bits before ranking according to the bit recognition result; and adjusting network parameters of the attribute transformation model and/or the generation model in response to the training derived image having sub-second attributes not located in the top preset number of digits in the rank.
In some embodiments, the original image is an image of a first item; the above apparatus further comprises: the acquisition unit (not shown) is configured to: acquiring a second object image set of a second object, wherein the first object and the second object belong to the same object type, the second object image set comprises at least two kinds of second object images, and different kinds of second object images have different sub second attributes; a ranking unit (not shown) configured to: and generating a ranking of the sub second attribute of the second attribute type according to the pre-generated scores of the second object images of the various types.
In some embodiments, the ranking includes sub-second attribute feature information and a rank corresponding to the sub-second attribute feature information; and the above-mentioned conversion unit, is configured to: and transforming the second attribute information according to the sub second attribute characteristic information corresponding to the bit number positioned in the preset number of bits in the ranking, so as to generate the derivative attribute information.
Referring now to FIG. 10, there is illustrated a schematic diagram of a computer system 1000 suitable for use in implementing an electronic device of an embodiment of the present application. The electronic device shown in fig. 10 is only an example, and should not impose any limitation on the functionality and scope of use of the embodiments of the present application.
As shown in fig. 10, the computer system 1000 includes a central processing unit (CPU, central Processing Unit) 1001 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage section 1006 into a random access Memory (RAM, random Access Memory) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other by a bus 1004. An Input/Output (I/O) interface 1005 is also connected to bus 1004.
The following components are connected to the I/O interface 1005: a storage portion 1006 including a hard disk or the like; and a communication section 1007 including a network interface card such as a LAN (local area network ) card, a modem, or the like. The communication section 1007 performs communication processing via a network such as the internet. The drive 1008 is also connected to the I/O interface 1005 as required. A removable medium 1009 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed on the drive 1008 as needed, so that a computer program read therefrom is installed into the storage portion 1006 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network through the communication portion 1007 and/or installed from the removable medium 1009. The above-described functions defined in the method of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 1001. It should be noted that the computer readable medium described in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented by software, or may be implemented by hardware. The described units may also be provided in a processor, for example, described as: a processor includes a first extraction unit, a second extraction unit, a transformation unit, and a generation unit. The names of these units do not constitute a limitation of the unit itself in some cases, and for example, the first extraction unit may also be described as "a unit that extracts the first attribute information of the original image".
As another aspect, the present application also provides a computer-readable medium that may be contained in the electronic device described in the above embodiment; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: extracting first attribute information of an original image, wherein a first attribute indicated by the first attribute information belongs to a first attribute type which is predefined; extracting second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type, and the second attribute type comprises at least two sub second attributes; transforming the second attribute information according to the ranking of the sub second attribute aiming at the second attribute type, and generating derivative attribute information of the sub second attribute; and generating a derivative image of the original image according to the first attribute information and the derivative attribute information.
The foregoing description is only of the preferred embodiments of the present application and is presented as a description of the principles of the technology being utilized. It will be appreciated by persons skilled in the art that the scope of the invention referred to in this application is not limited to the specific combinations of features described above, but it is intended to cover other embodiments in which any combination of features described above or equivalents thereof is possible without departing from the spirit of the invention. Such as the above-described features and technical features having similar functions (but not limited to) disclosed in the present application are replaced with each other.
Claims (13)
1. An image generation method, comprising:
extracting first attribute information of an original image, wherein a first attribute indicated by the first attribute information belongs to a first attribute type which is predefined, and the original image is an image of a first object;
extracting second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type, and the second attribute type comprises at least two sub second attributes;
transforming the second attribute information according to the ranking of the sub second attribute aiming at the second attribute type, and generating derivative attribute information indicating the sub second attribute;
generating a derivative image of the original image according to the first attribute information and the derivative attribute information;
the method further comprises the steps of: acquiring a second article image set of a second article, wherein the first article and the second article belong to the same article type, the second article image set comprises at least two kinds of second article images, and different kinds of second article images have different sub-second attributes; and generating a ranking of sub second attributes for the second attribute type according to the pre-generated scores of the second object images for the various categories.
2. The method of claim 1, wherein the transforming the second attribute information according to the ranking of sub-second attributes for the second attribute type to generate derived attribute information indicative of sub-second attributes comprises:
importing second attribute information into a pre-trained attribute transformation model to generate derivative attribute information of the second attribute information, wherein the attribute transformation model is used for representing the corresponding relation between the attribute information; and
the generating a derived image of the original image according to the first attribute information and the derived attribute information includes:
and importing the first attribute information and the derivative attribute information into a pre-trained generation model to generate a derivative image of the original image, wherein the generation model is used for representing the corresponding relation between the attribute information and the image.
3. The method of claim 2, wherein the extracting the first attribute information of the original image comprises:
importing the original image into a pre-trained first coding model, and extracting first attribute information of the original image, wherein the first coding model is used for representing a corresponding relation between the image and attribute information of a first attribute type; and
The extracting the second attribute information of the original image includes:
and importing the original image into a pre-trained second coding model, and extracting second attribute information of the original image, wherein the second coding model is used for representing the corresponding relation between the image and the attribute information of the second attribute type.
4. A method according to claim 3, wherein the method further comprises: training a generating network, wherein the generating network comprises a second coding model, an attribute transformation model and a generating model; wherein the method comprises the steps of
The training generation network comprises:
at least one training derivative image generated by a generation model in the generation network is imported into a pre-established discrimination model to generate a discrimination result, wherein the discrimination result is used for indicating whether the training derivative image is a real image or not;
importing at least one training derivative image and the ranking into a pre-established ranking model, and generating a rank recognition result for the training derivative image, wherein the rank recognition result is used for indicating the rank of a sub second attribute of the training derivative image in the ranking;
and updating network parameters of the generated network based on the discrimination result and the rank identification result.
5. The method of claim 4, wherein the training generation network comprises:
importing the training image into a second coding model to generate training second attribute information;
importing the generated training second attribute information into an attribute transformation model to generate at least one training derivative attribute information;
acquiring random vectors from a pre-established image vector set;
and importing the obtained random vector and training derivative attribute information into a generation model to generate a training derivative image.
6. The method of claim 5, wherein the generation network comprises a first coding model; and
the set of image vectors is established by:
the training image is imported into a first coding model, training first attribute information is generated, and the generated training first attribute information is taken as a set element to establish an image vector set.
7. The method of claim 6, wherein the ranking model is to: determining sub-second attributes of each training derivative image according to the sub-second attribute characteristic information in the ranking; and generating a rank recognition result according to the ranks of the sub second attributes of the training derivative images in the ranks.
8. The method of claim 7, wherein the training the generated network based on the discrimination result and the rank identification result comprises at least one of:
updating network parameters of at least one of the following according to the bit recognition result: an attribute transformation model and the generation model;
updating network parameters of at least one of the following according to the discrimination result: the first encoding model, the second encoding model, and the generation model.
9. The method of claim 8, wherein the updating network parameters according to the bit-rate recognition result comprises updating network parameters of at least one of: an attribute transformation model and the generation model, comprising:
determining whether the sub second attribute of the training derivative image is positioned at the preset number of bits before ranking according to the rank recognition result;
and adjusting network parameters of the attribute transformation model and/or the generation model in response to the training derived image having sub-second attributes not located in the top preset number of digits in the rank.
10. The method of claim 2, wherein the ranking comprises sub-second attribute feature information and a rank corresponding to the sub-second attribute feature information; and
The transforming the second attribute information according to the ranking of the sub second attribute for the second attribute type, generating the derivative attribute information indicating the sub second attribute, includes:
and transforming the second attribute information according to the sub second attribute characteristic information corresponding to the number of bits positioned in the preset number of bits in the ranking, and generating the derivative attribute information.
11. An image generating apparatus comprising:
a first extraction unit configured to extract first attribute information of an original image, wherein a first attribute indicated by the first attribute information belongs to a first attribute type defined in advance, the original image being an image of a first article;
a second extraction unit configured to extract second attribute information of the original image, wherein a second attribute indicated by the second attribute information belongs to a predefined second attribute type including at least two sub-second attributes;
a transformation unit configured to transform the second attribute information according to a ranking of sub-second attributes for the second attribute type, generating derived attribute information indicating sub-second attributes;
a generation unit configured to generate a derivative image of the original image based on the first attribute information and the derivative attribute information;
The apparatus further comprises: an acquisition unit configured to: acquiring a second article image set of a second article, wherein the first article and the second article belong to the same article type, the second article image set comprises at least two kinds of second article images, and different kinds of second article images have different sub-second attributes; a ranking unit configured to: and generating a ranking of sub second attributes for the second attribute type according to the pre-generated scores of the second object images for the various categories.
12. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-10.
13. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910646743.9A CN111784787B (en) | 2019-07-17 | 2019-07-17 | Image generation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910646743.9A CN111784787B (en) | 2019-07-17 | 2019-07-17 | Image generation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111784787A CN111784787A (en) | 2020-10-16 |
| CN111784787B true CN111784787B (en) | 2024-04-09 |
Family
ID=72755158
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910646743.9A Active CN111784787B (en) | 2019-07-17 | 2019-07-17 | Image generation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111784787B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20230146676A1 (en) * | 2021-11-05 | 2023-05-11 | Lemon Inc. | Portrait stylization framework to control the similarity between stylized portraits and original photo |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653142A (en) * | 2016-01-07 | 2016-06-08 | 厦门美图之家科技有限公司 | Image special effect ordering method and system based on areas and user attributes |
| CN106560809A (en) * | 2015-10-02 | 2017-04-12 | 奥多比公司 | Modifying At Least One Attribute Of Image With At Least One Attribute Extracted From Another Image |
| CN107248181A (en) * | 2017-06-16 | 2017-10-13 | 北京三快在线科技有限公司 | Image generating method and device, electronic equipment |
| CN107924577A (en) * | 2015-10-26 | 2018-04-17 | 松下知识产权经营株式会社 | Position generating means of making up and makeup position generation method |
| CN108197180A (en) * | 2017-12-25 | 2018-06-22 | 中山大学 | A kind of method of the editable image of clothing retrieval of clothes attribute |
| CN108334821A (en) * | 2018-01-18 | 2018-07-27 | 联想(北京)有限公司 | A kind of image processing method and electronic equipment |
| CN108460739A (en) * | 2018-03-02 | 2018-08-28 | 北京航空航天大学 | A kind of thin cloud in remote sensing image minimizing technology based on generation confrontation network |
| CN109345612A (en) * | 2018-09-13 | 2019-02-15 | 腾讯数码（天津）有限公司 | A kind of image generating method, device, equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10163042B2 (en) * | 2016-08-02 | 2018-12-25 | International Business Machines Corporation | Finding missing persons by learning features for person attribute classification based on deep learning |
| EP3516598A4 (en) * | 2016-09-21 | 2019-11-20 | Equifax Inc. | Transforming attributes for training automated modeling systems |
-
2019
- 2019-07-17 CN CN201910646743.9A patent/CN111784787B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106560809A (en) * | 2015-10-02 | 2017-04-12 | 奥多比公司 | Modifying At Least One Attribute Of Image With At Least One Attribute Extracted From Another Image |
| CN107924577A (en) * | 2015-10-26 | 2018-04-17 | 松下知识产权经营株式会社 | Position generating means of making up and makeup position generation method |
| CN105653142A (en) * | 2016-01-07 | 2016-06-08 | 厦门美图之家科技有限公司 | Image special effect ordering method and system based on areas and user attributes |
| CN107248181A (en) * | 2017-06-16 | 2017-10-13 | 北京三快在线科技有限公司 | Image generating method and device, electronic equipment |
| CN108197180A (en) * | 2017-12-25 | 2018-06-22 | 中山大学 | A kind of method of the editable image of clothing retrieval of clothes attribute |
| CN108334821A (en) * | 2018-01-18 | 2018-07-27 | 联想(北京)有限公司 | A kind of image processing method and electronic equipment |
| CN108460739A (en) * | 2018-03-02 | 2018-08-28 | 北京航空航天大学 | A kind of thin cloud in remote sensing image minimizing technology based on generation confrontation network |
| CN109345612A (en) * | 2018-09-13 | 2019-02-15 | 腾讯数码（天津）有限公司 | A kind of image generating method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于BRISK的图像快速匹配与变换算法;卫星;樊绍胜;高山;;电子测量技术(03);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111784787A (en) | 2020-10-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108830235B (en) | Method and apparatus for generating information | |
| CN111324774B (en) | Video duplicate removal method and device | |
| CN108427708B (en) | Data processing method, data processing apparatus, storage medium, and electronic apparatus | |
| CN109492772B (en) | Method and device for generating information | |
| US10719693B2 (en) | Method and apparatus for outputting information of object relationship | |
| CN113139628B (en) | Sample image identification method, device and equipment and readable storage medium | |
| CN110555714A (en) | method and apparatus for outputting information | |
| CN112418292B (en) | Image quality evaluation method, device, computer equipment and storage medium | |
| US11640634B2 (en) | Deep learning based visual compatibility prediction for bundle recommendations | |
| CN113434716B (en) | Cross-modal information retrieval method and device | |
| CN113379449B (en) | Multimedia resource recall method and device, electronic equipment and storage medium | |
| CN111581926A (en) | Method, device and equipment for generating file and computer readable storage medium | |
| CN113240127A (en) | Federal learning-based training method and device, electronic equipment and storage medium | |
| CN114298122A (en) | Data classification method, device, equipment, storage medium and computer program product | |
| CN114282059A (en) | Video retrieval method, device, equipment and storage medium | |
| CN107968743A (en) | The method and apparatus of pushed information | |
| CN114299304A (en) | Image processing method and related equipment | |
| CN111784787B (en) | Image generation method and device | |
| CN112307243A (en) | Method and apparatus for retrieving image | |
| US11962817B2 (en) | Machine learning techniques for advanced frequency management | |
| CN110727705B (en) | Information recommendation method and device, electronic equipment and computer-readable storage medium | |
| CN114186039A (en) | Visual question answering method and device and electronic equipment | |
| CN113821676A (en) | Video retrieval method, device, equipment and storage medium | |
| CN113780324A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112819078B (en) | Iteration method and device for picture identification model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |