CN108765278B - Image processing method, mobile terminal and computer readable storage medium - Google Patents
Image processing method, mobile terminal and computer readable storage medium Download PDFInfo
- Publication number
- CN108765278B CN108765278B CN201810570632.XA CN201810570632A CN108765278B CN 108765278 B CN108765278 B CN 108765278B CN 201810570632 A CN201810570632 A CN 201810570632A CN 108765278 B CN108765278 B CN 108765278B
- Authority
- CN
- China
- Prior art keywords
- image
- processed
- foreground
- label
- style
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 51
- 238000013508 migration Methods 0.000 claims abstract description 51
- 230000005012 migration Effects 0.000 claims abstract description 51
- 238000012545 processing Methods 0.000 claims abstract description 45
- 238000013527 convolutional neural network Methods 0.000 claims description 47
- 230000008569 process Effects 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 21
- 238000001514 detection method Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 14
- 239000003086 colorant Substances 0.000 claims description 4
- 230000006870 function Effects 0.000 description 11
- 239000011159 matrix material Substances 0.000 description 8
- 230000007704 transition Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013178 mathematical model Methods 0.000 description 3
- 230000003796 beauty Effects 0.000 description 2
- 238000005282 brightening Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 241000251468 Actinopterygii Species 0.000 description 1
- 244000025254 Cannabis sativa Species 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002087 whitening effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The application is applicable to the technical field of image processing, and provides an image processing method, a mobile terminal and a computer readable storage medium, wherein the image processing method comprises the following steps: the method comprises the steps of obtaining an image to be processed, identifying the image to be processed to obtain a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed, carrying out local processing on the foreground target in the image to be processed based on the foreground label, carrying out style migration on the image to be processed based on the background label to obtain a style migration image, and obtaining diversified images through the method.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing method, a mobile terminal, and a computer-readable storage medium.
Background
With the development of intelligent mobile terminals, people use mobile terminals such as mobile phones and the like to take pictures more and more frequently. Most of the existing photographing functions of mobile terminals support image processing, such as a filter function for a human face, a peeling function, a whitening function, and the like.
However, the current image processing method is single, for example, the processing for human face is the processing related to beauty. Therefore, the effect of the shot photos is single at present, and the user experience is poor.
Disclosure of Invention
In view of this, embodiments of the present application provide an image processing method, a mobile terminal, and a computer-readable storage medium, so as to solve the problems of single effect and poor user experience of a currently-captured photo.
A first aspect of an embodiment of the present application provides an image processing method, including:
acquiring an image to be processed, and identifying the image to be processed to obtain a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed;
performing local processing on a foreground target in the image to be processed based on the foreground label;
and performing style migration on the image to be processed based on the background label to obtain a style migration image.
A second aspect of an embodiment of the present application provides a mobile terminal, including:
the first identification result acquisition module is used for acquiring an image to be processed and identifying the image to be processed to acquire a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed;
the foreground processing module is used for carrying out local processing on a foreground target in the image to be processed based on the foreground label;
and the style migration module is used for carrying out style migration on the image to be processed based on the background label to obtain a style migration image.
A third aspect of an embodiment of the present application provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method provided in the first aspect of the embodiment of the present application when executing the computer program.
A fourth aspect of embodiments of the present application provides a computer-readable storage medium storing a computer program which, when executed by one or more processors, performs the steps of the method provided by the first aspect of embodiments of the present application.
A fifth aspect of embodiments of the present application provides a computer program product comprising a computer program that, when executed by one or more processors, performs the steps of the method provided by the first aspect of embodiments of the present application.
The embodiment of the application acquires a to-be-processed image, and is right the to-be-processed image is identified to obtain a first identification result, the first identification result comprises a foreground label of a foreground target in the to-be-processed image and a background label of the to-be-processed image, based on the foreground label is right the foreground target in the to-be-processed image is locally processed, based on the background label is right the to-be-processed image is subjected to style migration to obtain a style migration image, because in the application, the foreground label of the foreground target in the to-be-processed image can be subjected to the foreground target and the foreground label is locally processed in a relevant manner, the background label of the to-be-processed image is right the to-be-processed image is subjected to the style migration, and the finally obtained style migration image can present diversified effects.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic implementation flowchart of an image processing method according to an embodiment of the present application;
fig. 2 is a schematic implementation flowchart of another image processing method provided in an embodiment of the present application;
FIG. 3 is a schematic flow chart illustrating another implementation of an image processing method according to an embodiment of the present disclosure;
fig. 4 is a schematic block diagram of a mobile terminal according to an embodiment of the present application;
fig. 5 is a schematic block diagram of another mobile terminal provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
In the embodiment of the application, the style of an image A is converted into another style b according to a preset condition, namely the style b is transferred into the image A, and an image A is subjected to style transfer to obtain a style transfer image. In fact, each image has artistic styles, such as sketch, chinese wind, stereology, impression pie, modern, sanskrit style, etc. Taking the Sanskrit style as an example, the style of a piece of Sanskrit-high works (images) is analyzed, a mathematical model of the style of the image can be constructed, the style corresponding to the mathematical model is defined as the Sanskrit style, the style of an image to be processed can be converted into the image with the Sanskrit style based on the mathematical model corresponding to the Sanskrit style, namely the image with the Sanskrit style is obtained after the style of the image to be processed is migrated. It should be noted that, although the style is changed between the image to be processed and the style transition image obtained after the style transition, the content is not changed, for example, the image to be processed is a face image in a face photograph taken by a camera and still is a face image in the style transition image obtained after the conversion into the sketch style, but the face image becomes a sketch style face image. In order to further illustrate the technical solutions described in the present application, the following description is given by way of specific examples.
Fig. 1 is a schematic flow chart of an implementation of an image processing method provided in an embodiment of the present application, and as shown in the figure, the method may include the following steps:
step S101, obtaining an image to be processed, and identifying the image to be processed to obtain a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed.
In this embodiment of the application, the image to be processed may be any image selected by a user, may also be a preview image acquired by a camera on the mobile terminal, and may also be a photo taken by the camera on the mobile terminal, which is not limited herein.
When the image to be processed is identified, whether a foreground target exists in the image to be processed can be identified through the convolutional neural network model, if the identification result shows that the foreground target exists, the identification image with the detection frame can be output, the foreground target in the detection frame is identified, and meanwhile, a foreground label of the foreground target is output. If the foreground object exists in the image to be processed, the background of the image to be processed can be continuously identified to obtain the background label. In the process of obtaining the foreground tag and the background tag, the to-be-processed image may be respectively identified through two independent convolutional neural network models, and after the foreground target is identified, the foreground target may be removed from the to-be-processed image to obtain the background image, and the background tag may be obtained according to the background image, and specifically, what kind of method is used to obtain the foreground tag of the foreground target and the background tag of the to-be-processed image is not limited herein. If the foreground object is not identified in the image to be processed, the original image can be output, namely, the foreground object does not exist, and the background label does not need to be continuously identified.
As another embodiment of the present application, before performing local processing on a foreground target in the image to be processed based on the foreground tag, the method further includes:
judging whether the combination of the foreground label and the background label is a preset label combination or not;
and if the combination of the foreground label and the background label is a preset label combination, locally processing a foreground target in the image to be processed based on the foreground label.
In this embodiment of the present application, a combination of a foreground tag and a background tag may be preset, and only when the combination of the foreground tag and the background tag is a preset tag combination, relevant image processing may be further performed on the image to be processed, for example, the preset tag combination is a face tag + a landscape tag, the identified foreground object of the image to be processed is a face, and the background image is a landscape, that is, the combination of the foreground tag and the background tag is the face tag + the landscape tag, at this time, beauty processing may be performed on the face part.
And S102, locally processing a foreground target in the image to be processed based on the foreground label.
In the embodiment of the present application, the local processing performed on the foreground object is actually related processing performed on the foreground object, for example, focusing processing is performed on the detected foreground object, and may also be area brightening processing, and when the foreground object is a human face, the local processing may be skin color brightening processing, and the like. In practical application, the foreground object can be subjected to an image processing mode with the foreground label based on different foreground labels. When a foreground target is processed, not only position information of a detection frame corresponding to the foreground target but also the position of the foreground target needs to be accurately determined, so that the foreground target needs to be segmented from an image to be processed.
As another embodiment of the present application, a method for segmenting the foreground object from the image to be processed includes:
acquiring an image in a detection frame corresponding to the foreground target;
based on the gray gradient of the image in the detection frame, identifying the boundary of a foreground target in the image in the detection frame to obtain a foreground target contour line;
acquiring a gray threshold sequence, and performing binarization processing on the image in the detection frame through each gray threshold in the gray threshold sequence to obtain a gray foreground target image sequence;
and determining a foreground target area based on the gray level foreground target image sequence and the foreground target contour line, wherein the image in the foreground target area in the image to be processed is a foreground target.
In the embodiment of the present application, both the binarization method and the gray scale gradient method have certain defects, and the obtained result is not a very accurate result. In order to obtain an accurate result, the embodiment of the application combines a binarization method and a gray gradient method to obtain a foreground target.
Determining a foreground target area based on the gray level foreground target image sequence and the foreground target contour line, wherein the step of determining an image in the foreground target area in the image to be processed as a foreground target comprises the following steps:
acquiring a gray level foreground target image with the highest matching degree with the foreground target contour line from the gray level foreground target image sequence;
and fusing the gray level foreground target image with the highest matching degree with the foreground target contour line to generate a continuous foreground target area.
In the embodiment of the present application, no matter what the target area in the obtained grayscale foreground target image with the highest matching degree or what the target area represented by the foreground target contour line obtained by the grayscale gradient method cannot accurately describe the foreground target area, however, the grayscale foreground target image with the highest matching degree with the foreground target contour line may be fused with the foreground target contour line to generate a continuous foreground target area, an inaccurate part in the foreground target contour line obtained by the grayscale gradient is discarded by the binarized image, an inaccurate part in the grayscale foreground target image is discarded by the foreground target contour line obtained by the grayscale gradient method, a continuous foreground target area is obtained after fusion, and the foreground target area is not a real foreground target image because the grayscale image and the contour line are fused after binarization, so the obtained foreground target area represents the coordinates of the foreground target in the image to be processed, and the image in the foreground target area in the image to be processed is the foreground target.
And step S103, performing style migration on the image to be processed based on the background label to obtain a style migration image.
In this embodiment of the present application, performing style migration on the image to be processed based on the background tag to obtain a style migration image, for example:
when the detected foreground label is a castle and the detected background label is sunset, the style of the image to be processed can be converted into a cartoon style;
when the detected foreground label is a human face and the detected background label is a blue sky, the style of the image to be processed can be converted into a sketch style;
when the detected foreground label is a human face and the detected background label is night space, the style of the image to be processed can be converted into the Sanskrit style;
……。
as another embodiment of the present application, performing style migration on the image to be processed based on the background tag to obtain a style migration image includes:
acquiring a Gram matrix corresponding to the background label based on the background label;
and carrying out style migration on the image to be processed based on the Gram matrix to obtain a style migration image.
In the embodiment of the application, the Gram matrix is a Gram matrix, the Gram matrix of an image can be used as the style characteristic of the image, and a convolution neural network model can be constructed; and training the convolutional neural network model, and performing style migration on the image to be processed to obtain a style migration image by using the image obtained by the trained convolutional neural network model.
The training process is divided into a forward propagation process and a backward propagation process, wherein the forward propagation process comprises the following steps: processing the image to be processed through the constructed convolutional neural network model to obtain an output image; and (3) a back propagation process: performing back propagation on a difference value between the Gram matrix of the output image and the Gram matrix corresponding to the background label, and updating parameters of the convolutional neural network model; and obtaining a trained convolutional neural network model after the convolutional neural network model converges, and processing the image to be processed through the trained convolutional neural network model to obtain a style transition image.
The embodiment of the application acquires a to-be-processed image, and is right the to-be-processed image is identified to obtain a first identification result, the first identification result comprises a foreground label of a foreground target in the to-be-processed image and a background label of the to-be-processed image, based on the foreground label is right the foreground target in the to-be-processed image is locally processed, based on the background label is right the to-be-processed image is subjected to style migration to obtain a style migration image, because in the application, the foreground label of the foreground target in the to-be-processed image can be subjected to the foreground target and the foreground label is locally processed in a relevant manner, the background label of the to-be-processed image is right the to-be-processed image is subjected to the style migration, and the finally obtained style migration image can present diversified effects.
Fig. 2 is a schematic flowchart of another image processing method provided in an embodiment of the present application, and as shown in the drawing, the method describes how to perform style migration on the image to be processed based on the background label to obtain a style migration image, which specifically includes the following steps:
step S201, obtaining a style image corresponding to the background label from a preset database based on the background label.
In this embodiment of the application, a database may be preset, and style images corresponding to a background label or style images corresponding to a combination of a foreground label and a background label are stored in the database, for example, a sanskrit style corresponding to night sky of the background label, a sketch style corresponding to blue sky of the background label, a sanskrit style corresponding to night sky of a foreground label face and background label, and a sketch style corresponding to blue sky of a foreground label face and background label may be set.
After obtaining the style image, the style in the style image can be migrated to the image to be processed to obtain a style migrated image based on the style image. Step S202 to step S204 are another method of obtaining a style transition image.
Step S202, acquiring the content characteristics of the image to be processed and the style characteristics of the style image.
In the embodiment of the application, as described above, the style feature of an image may be represented by a Gram matrix, the content feature of an image may be represented by a SIFT feature, and it is actually intended to obtain an image by migrating the style image to the image to be processed, where the image has both the style feature and the content feature of the image to be processed. Therefore, it is necessary to acquire the content features of the image to be processed and the style features of the style image.
Step S203, training the constructed convolutional neural network model based on the content characteristics of the image to be processed and the style characteristics of the style image, and obtaining the trained convolutional neural network model.
In the embodiment of the present application, we can construct a convolutional neural network model, which can generate an image from a noise signal, and therefore, the convolutional neural network model can be a generating network model capable of generating an image.
The training of the constructed convolutional neural network model based on the content features of the image to be processed and the style features of the style image to obtain the trained convolutional neural network model comprises the following steps:
and (3) forward propagation process: generating an initial image through the constructed convolutional neural network model, and obtaining style characteristics and content characteristics of the initial image;
and (3) a back propagation process: reversely training the convolutional neural network model based on the difference between the style features of the initial image and the style features of the style image and the difference between the content features of the initial image and the content features of the image to be processed;
obtaining a trained convolutional neural network model after the convolutional neural network model converges.
In this embodiment, the training process may be understood as that, after an initial image is generated through a constructed convolutional neural network model, differences between style features of the generated initial image and style features of a wind image may be obtained, and differences between content features of the generated initial image and content features of an image to be processed may also be obtained, where the two differences may construct a loss function, and the reverse training process is actually a process of deriving the loss function and updating parameters of layers in the convolutional neural network model in a reverse manner, and when the convolutional neural network model converges, that is, when differences between the style features of the generated initial image and the style features of the wind image are smaller and smaller, and differences between the content features of the generated initial image and the content features of the image to be processed are smaller and smaller, it indicates that an image generated through the convolutional neural network model has both content features of the image to be processed and style features of the wind image. Then the image generated by the trained convolutional neural network model is the style migration image obtained by performing style migration on the image to be processed based on the style image.
It should be noted that the condition for ending the training of the convolutional neural network model may be: and if the difference between the style characteristic of the generated initial image and the style characteristic of the stylized image is smaller than a first preset value and the difference between the content characteristic of the generated initial image and the content characteristic of the image to be processed is smaller than a second preset value, the corresponding initial image can be used as the style transition image of the image to be processed.
And step S204, generating a style transition image through the trained convolutional neural network model.
According to the method and the device, a convolutional neural network model can be constructed in advance, then the constructed convolutional neural network model is trained on the basis of the content characteristics of the image to be processed and the style characteristics of the obtained style image, and the trained convolutional neural network model can generate the style migration image.
Fig. 3 is a schematic flow chart of another image processing method provided in the embodiment of the present application, and as shown in the figure, the method is based on the embodiment shown in fig. 1, and describes how to identify the image to be processed to obtain a first identification result, which specifically includes the following steps:
step S301, identifying the image to be processed to obtain a second identification result.
Step S302, if the second recognition result includes at least two candidate targets, classifying the candidate targets based on the position information of each candidate target in the second recognition result in the image to be processed to obtain a foreground target and a background target.
In this embodiment of the application, the image to be processed may be identified by a target identification model, where the target identification model may be a convolutional neural network model, and identify whether a candidate target exists in the image to be processed by the target identification model, and if the candidate target exists, output position information of the candidate target in the image to be processed, where the position information of the candidate target in the image to be processed may be position information of a detection frame corresponding to the candidate target; if no candidate target exists, the original image is output. It should be noted that the candidate target refers to a target that can be recognized by the target recognition model in the image to be processed.
In practical applications, the object recognition model may be capable of recognizing not only a foreground object but also a small object in a background area, for example, when a person is self-portrait in a landscape, a human face may be recognized, and a flower may also be recognized, however, the flower appears as a background image, so the object recognized by the object recognition model is marked as a candidate object, and a classification according to the candidate object is also needed to obtain the foreground object and the background object. In a specific distinguishing manner, the foreground target and the candidate targets may be distinguished based on the size, the position, and the like of the identified candidate targets, for example, a candidate target whose position is in the most central region of the image to be processed in the candidate targets is taken as the foreground target, and other candidate targets are taken as the background target; the candidate target with the largest area in the candidate targets can be used as the foreground target, and other candidate targets can be used as the background target. Of course, it is also possible to provide: calculating the distance from each candidate target to the central point of the image to be processed, and taking the candidate targets with the distance from the central point of the image to be processed within a preset distance as foreground targets; also, it is possible to provide: and calculating the area occupied by each candidate target, and taking the candidate targets with the occupied areas within a preset value as the foreground targets.
And step S303, obtaining a foreground label according to the foreground target.
In the embodiment of the application, after the foreground target is determined, the foreground label of the foreground target can be correspondingly obtained. Since the background object is only a part of the background image, the label of the background object cannot be used as the background label, and the background label needs to be generated according to the background object. The step of generating the background label according to the background object may refer to step S304 to step S306.
Step S304, a background image in the image to be processed is obtained, wherein the background image is an image except the candidate target in the image to be processed.
Step S305, based on the color coordinates of the pixel points in the background image in the color gamut space, clustering the colors of the pixel points to obtain the color class corresponding to the background image.
In the embodiment of the present application, in order to obtain the background tag, a background image in the image to be processed needs to be determined, and the background image may be an image after the candidate object is removed. In order to reduce the memory usage rate, a background label of a background image may be estimated according to the background image, for example, color coordinates of a pixel point in the background image in an RGB color gamut space are obtained; and clustering the colors of the pixel points to obtain the color class corresponding to the background image, and calculating the coordinate mean value of all the pixel points in the RGB color gamut space without adopting a clustering mode to determine the color class corresponding to the coordinate mean value in the color gamut space.
The color class corresponding to the background image obtained by the coordinate mean may have a deviation, because when the background image includes a grass, the background image may also include a blue sky, and the color class of the background image obtained by the coordinate mean may not be green nor blue. The color class corresponding to the background image obtained by the clustering method can display two color classes, one blue and one green, so that two background labels may exist: blue sky and grassland.
In practical applications, different regions may be divided according to color ranges in the color gamut space, and different color classes may be set for the different regions.
Step S306, generating a background label based on the background object and the color class corresponding to the background image.
In the embodiment of the present application, the background label may be generated according to the color class corresponding to the background image, for example,
if the color class obtained after the clustering of the pixel points in the background image is green, the background label of the background image can be estimated to be a grassland; if the color class obtained after the clustering of the pixel points in the background image is black, the background label of the background image can be estimated to be a night scene; if the color class obtained after the clustering of the pixels in the background image is blue, the background label of the background image can be estimated to be a blue sky.
However, there is a disadvantage in such a manner that the background label is generated only by the color class corresponding to the background image, for example, if the color corresponding to the blue sky is blue, then the color corresponding to the sea is also blue, and when the color class obtained after clustering the pixels in the background image is blue, errors may occur regardless of whether the background label is estimated to be the blue sky or the sea, so the background label is obtained by combining the background object and the color class corresponding to the background image in the embodiment of the present application.
For example, when the color class obtained after clustering the pixels in the background image is blue and the background target is a medium bird class, it may be determined that the background label is a blue sky, and when the color class obtained after clustering the pixels in the background image is blue and the background target is a medium fish class, it may be determined that the background label is a sea.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by functions and internal logic of the process, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 4 is a schematic block diagram of a mobile terminal according to an embodiment of the present application, and only a portion related to the embodiment of the present application is shown for convenience of description.
The mobile terminal 4 may be a software unit, a hardware unit or a combination of software and hardware unit built in a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, etc., or may be integrated into a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, etc., as an independent pendant.
The mobile terminal 4 includes:
a first identification result obtaining module 41, configured to obtain an image to be processed, and identify the image to be processed to obtain a first identification result, where the first identification result includes a foreground tag of a foreground object in the image to be processed and a background tag of the image to be processed;
a foreground processing module 42, configured to perform local processing on a foreground target in the image to be processed based on the foreground tag;
and a style migration module 43, configured to perform style migration on the image to be processed based on the background tag to obtain a style migration image.
Optionally, the style migration module 43 includes:
a style image obtaining unit 431, configured to obtain a style image corresponding to the background tag from a preset database based on the background tag;
a style migration unit 432, configured to migrate a style in the style image into the image to be processed to obtain a style migration image based on the style image.
Optionally, the style migration unit 432 includes:
a feature obtaining subunit 4321, configured to obtain a content feature of the image to be processed and a style feature of the style image;
a training subunit 4322, configured to train the constructed convolutional neural network model based on the content features of the image to be processed and the style features of the style image, to obtain a trained convolutional neural network model;
and the style migration subunit 4323 is configured to generate a style migration image through the trained convolutional neural network model.
Optionally, the training subunit 4322 is further configured to:
and (3) forward propagation process: generating an initial image through the constructed convolutional neural network model, and obtaining style characteristics and content characteristics of the initial image;
and (3) a back propagation process: reversely training the convolutional neural network model based on the difference between the style features of the initial image and the style features of the style image and the difference between the content features of the initial image and the content features of the image to be processed;
and after the convolutional neural network model is converged, obtaining a trained convolutional neural network model.
Optionally, the first recognition result obtaining module 41 includes:
a second recognition result obtaining unit 411, configured to recognize the image to be processed to obtain a second recognition result;
a classifying unit 412, configured to, if the second recognition result includes at least two candidate targets, classify the candidate targets based on position information of each candidate target in the second recognition result in the image to be processed to obtain a foreground target and a background target;
a label obtaining unit 413, configured to obtain a foreground label according to the foreground object, and generate a background label according to the background object.
Optionally, the tag obtaining unit 413 includes:
a background image obtaining subunit 4131, configured to obtain a background image in the image to be processed, where the background image is an image of the image to be processed other than the candidate target;
a clustering subunit 4132, configured to cluster colors of the pixels in the background image based on color coordinates of the pixels in a color gamut space, to obtain a color class corresponding to the background image;
a label obtaining subunit 4133, configured to generate a background label based on the background object and the color class corresponding to the background image.
Optionally, the mobile terminal 4 further includes:
the judging module is used for judging whether the combination of the foreground label and the background label is a preset label combination or not before the foreground label is used for carrying out local processing on the foreground target in the image to be processed;
the foreground processing module 42 is further configured to:
and if the combination of the foreground label and the background label is a preset label combination, locally processing a foreground target in the image to be processed based on the foreground label.
It will be apparent to those skilled in the art that, for convenience and simplicity of description, the foregoing functional units and modules are merely illustrated in terms of division, and in practical applications, the foregoing functional allocation may be performed by different functional units and modules as needed, that is, the internal structure of the mobile terminal is divided into different functional units or modules to perform all or part of the above described functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the above-mentioned apparatus may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 5 is a schematic block diagram of a mobile terminal according to another embodiment of the present application. As shown in fig. 5, the mobile terminal 5 of this embodiment includes: one or more processors 50, a memory 51 and a computer program 52 stored in said memory 51 and executable on said processors 50. The processor 50, when executing the computer program 52, implements the steps in the various image processing method embodiments described above, such as steps S101 to S103 shown in fig. 1. Alternatively, the processor 50, when executing the computer program 52, implements the functions of the modules/units in the above-described mobile terminal embodiments, such as the functions of the modules 41 to 43 shown in fig. 4.
Illustratively, the computer program 52 may be partitioned into one or more modules/units, which are stored in the memory 51 and executed by the processor 50 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing certain functions, which are used to describe the execution of the computer program 52 in the mobile terminal 5. For example, the computer program 52 may be divided into a first recognition result obtaining module, a foreground processing module, and a style migration module.
The first identification result acquisition module is used for acquiring an image to be processed and identifying the image to be processed to acquire a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed;
the foreground processing module is used for carrying out local processing on a foreground target in the image to be processed based on the foreground label;
and the style migration module is used for carrying out style migration on the image to be processed based on the background label to obtain a style migration image.
Other modules or units may refer to the description of the embodiment shown in fig. 4, and are not described herein again.
The mobile terminal includes, but is not limited to, a processor 50, a memory 51. Those skilled in the art will appreciate that fig. 5 is only one example of a mobile terminal 5 and is not intended to limit the mobile terminal 5 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the mobile terminal may also include input devices, output devices, network access devices, buses, etc.
The Processor 50 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 51 may be an internal storage unit of the mobile terminal 5, such as a hard disk or a memory of the mobile terminal 5. The memory 51 may also be an external storage device of the mobile terminal 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the mobile terminal 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the mobile terminal 5. The memory 51 is used for storing the computer program and other programs and data required by the mobile terminal. The memory 51 may also be used to temporarily store data that has been output or is to be output.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed mobile terminal and method may be implemented in other ways. For example, the above-described embodiments of the mobile terminal are merely illustrative, and for example, the division of the modules or units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a computer-readable storage medium and can realize the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media which may not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (6)
1. An image processing method, characterized by comprising:
acquiring an image to be processed, and identifying the image to be processed to obtain a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed; wherein, segmenting the foreground target from the image to be processed comprises: acquiring an image in a detection frame corresponding to the foreground target; based on the gray gradient of the image in the detection frame, identifying the boundary of a foreground target in the image in the detection frame, and acquiring a foreground target contour line; acquiring a gray threshold sequence, and performing binarization processing on the image in the detection frame through each gray threshold in the gray threshold sequence to obtain a gray foreground target image sequence; determining a foreground target area based on the gray level foreground target image sequence and the foreground target contour line, wherein an image in the foreground target area in the image to be processed is a foreground target;
locally processing a foreground target in the image to be processed based on the foreground label;
performing style migration on the image to be processed based on the background label to obtain a style migration image, including: acquiring a style image corresponding to the background label from a preset database based on the background label;
acquiring the content characteristics of the image to be processed and the style characteristics of the style image;
and (3) forward propagation process: generating an initial image through the constructed convolutional neural network model, and obtaining style characteristics and content characteristics of the initial image;
and (3) a back propagation process: reversely training the convolutional neural network model based on the difference between the style features of the initial image and the style features of the style image and the difference between the content features of the initial image and the content features of the image to be processed;
after the convolutional neural network model is converged, obtaining a trained convolutional neural network model;
generating a style migration image through the trained convolutional neural network model;
before locally processing a foreground object in the image to be processed based on the foreground label, the method further comprises:
judging whether the combination of the foreground label and the background label is a preset label combination or not;
and if the combination of the foreground label and the background label is a preset label combination, locally processing a foreground target in the image to be processed based on the foreground label.
2. The image processing method of claim 1, wherein the recognizing the image to be processed to obtain a first recognition result comprises:
identifying the image to be processed to obtain a second identification result;
if the second recognition result comprises at least two candidate targets, classifying the candidate targets based on the position information of each candidate target in the second recognition result in the image to be processed to obtain a foreground target and a background target;
and obtaining a foreground label according to the foreground target, and generating a background label according to the background target.
3. The image processing method of claim 2, wherein the generating a background label from the background object comprises:
acquiring a background image in the image to be processed, wherein the background image is an image out of the candidate target in the image to be processed;
clustering the colors of the pixel points based on the color coordinates of the pixel points in the background image in the color gamut space to obtain a color class corresponding to the background image;
and generating a background label based on the background target and the color class corresponding to the background image.
4. A mobile terminal, comprising:
the first identification result acquisition module is used for acquiring an image to be processed and identifying the image to be processed to acquire a first identification result, wherein the first identification result comprises a foreground label of a foreground target in the image to be processed and a background label of the image to be processed; wherein, segmenting the foreground target from the image to be processed comprises: acquiring an image in a detection frame corresponding to the foreground target; based on the gray gradient of the image in the detection frame, identifying the boundary of a foreground target in the image in the detection frame, and acquiring a foreground target contour line; acquiring a gray threshold sequence, and performing binarization processing on the image in the detection frame through each gray threshold in the gray threshold sequence to obtain a gray foreground target image sequence; determining a foreground target area based on the gray level foreground target image sequence and the foreground target contour line, wherein an image in the foreground target area in the image to be processed is a foreground target;
the foreground processing module is used for carrying out local processing on a foreground target in the image to be processed based on the foreground label;
the style migration module is used for carrying out style migration on the image to be processed based on the background label to obtain a style migration image;
the style migration module comprises:
the style image acquisition unit is used for acquiring a style image corresponding to the background label from a preset database based on the background label;
the characteristic obtaining subunit is used for obtaining the content characteristic of the image to be processed and the style characteristic of the style image;
the training subunit is used for:
and (3) forward propagation process: generating an initial image through the constructed convolutional neural network model, and obtaining style characteristics and content characteristics of the initial image;
and (3) a back propagation process: reversely training the convolutional neural network model based on the difference between the style features of the initial image and the style features of the style image and the difference between the content features of the initial image and the content features of the image to be processed;
after the convolutional neural network model is converged, obtaining a trained convolutional neural network model;
generating a style migration image through the trained convolutional neural network model;
the mobile terminal further includes:
the judging module is used for judging whether the combination of the foreground label and the background label is a preset label combination or not before the foreground label is used for carrying out local processing on the foreground target in the image to be processed;
the foreground processing module is further to:
and if the combination of the foreground label and the background label is a preset label combination, locally processing a foreground target in the image to be processed based on the foreground label.
5. A mobile terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 3 when executing the computer program.
6. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by one or more processors, implements the steps of the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810570632.XA CN108765278B (en) | 2018-06-05 | 2018-06-05 | Image processing method, mobile terminal and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810570632.XA CN108765278B (en) | 2018-06-05 | 2018-06-05 | Image processing method, mobile terminal and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108765278A CN108765278A (en) | 2018-11-06 |
| CN108765278B true CN108765278B (en) | 2023-04-07 |
Family
ID=63999786
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810570632.XA Active CN108765278B (en) | 2018-06-05 | 2018-06-05 | Image processing method, mobile terminal and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108765278B (en) |
Families Citing this family (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109510943A (en) * | 2018-12-17 | 2019-03-22 | 三星电子（中国）研发中心 | Method and apparatus for shooting image |
| CN109859096A (en) * | 2018-12-28 | 2019-06-07 | 北京达佳互联信息技术有限公司 | Image Style Transfer method, apparatus, electronic equipment and storage medium |
| CN111383289A (en) * | 2018-12-29 | 2020-07-07 | Tcl集团股份有限公司 | Image processing method, image processing device, terminal equipment and computer readable storage medium |
| CN110083430B (en) * | 2019-04-30 | 2022-03-29 | 成都映潮科技股份有限公司 | System theme color changing method, device and medium |
| CN112400186B (en) * | 2019-05-15 | 2023-08-01 | 微软技术许可有限责任公司 | Image generation |
| CN110598781A (en) * | 2019-09-05 | 2019-12-20 | Oppo广东移动通信有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN110781416A (en) * | 2019-09-16 | 2020-02-11 | 上海掌门科技有限公司 | Method and device for providing landscape information |
| CN110991516A (en) * | 2019-11-28 | 2020-04-10 | 哈尔滨工程大学 | Side-scan sonar image target classification method based on style migration |
| CN111127309B (en) * | 2019-12-12 | 2023-08-11 | 杭州格像科技有限公司 | Portrait style migration model training method, portrait style migration method and device |
| CN112995539B (en) * | 2019-12-16 | 2022-07-01 | 青岛海信移动通信技术股份有限公司 | Mobile terminal and image processing method |
| CN111161193B (en) * | 2019-12-31 | 2023-09-05 | 深圳度影医疗科技有限公司 | Ultrasonic image quality optimization method, storage medium and terminal equipment |
| CN111223039A (en) * | 2020-01-08 | 2020-06-02 | 广东博智林机器人有限公司 | Image style conversion method and device, electronic equipment and storage medium |
| CN111325664B (en) * | 2020-02-27 | 2023-08-29 | Oppo广东移动通信有限公司 | Style migration method and device, storage medium and electronic equipment |
| CN111986075B (en) * | 2020-08-12 | 2022-08-09 | 兰州交通大学 | Style migration method for target edge clarification |
| CN112232425B (en) * | 2020-10-21 | 2023-11-28 | 腾讯科技（深圳）有限公司 | Image processing method, device, storage medium and electronic equipment |
| CN112561782B (en) * | 2020-12-15 | 2023-01-03 | 哈尔滨工程大学 | Method for improving reality degree of simulation picture of offshore scene |
| CN112764845B (en) | 2020-12-30 | 2022-09-16 | 北京字跳网络技术有限公司 | Video processing method and device, electronic equipment and computer readable storage medium |
| CN112712138B (en) * | 2021-01-19 | 2022-05-20 | 腾讯科技（深圳）有限公司 | Image processing method, device, equipment and storage medium |
| CN115049888A (en) * | 2021-02-25 | 2022-09-13 | 华为技术有限公司 | Data processing method, device and system |
| CN112990390B (en) * | 2021-05-19 | 2021-08-06 | 腾讯科技（深圳）有限公司 | Training method of image recognition model, and image recognition method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933613A (en) * | 2016-06-28 | 2016-09-07 | 广东欧珀移动通信有限公司 | Image processing method and apparatus and mobile terminal |
| WO2016180224A1 (en) * | 2015-05-11 | 2016-11-17 | 北京锤子数码科技有限公司 | Method and device for processing image of person |
| CN106651766A (en) * | 2016-12-30 | 2017-05-10 | 深圳市唯特视科技有限公司 | Image style migration method based on deep convolutional neural network |
| CN107277391A (en) * | 2017-06-30 | 2017-10-20 | 北京奇虎科技有限公司 | Image switching network processing method, server, computing device and storage medium |
| CN107705242A (en) * | 2017-07-20 | 2018-02-16 | 广东工业大学 | A kind of image stylization moving method of combination deep learning and depth perception |
| WO2018075927A1 (en) * | 2016-10-21 | 2018-04-26 | Google Llc | Stylizing input images |
| CN107993191A (en) * | 2017-11-30 | 2018-05-04 | 腾讯科技（深圳）有限公司 | A kind of image processing method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101777180B (en) * | 2009-12-23 | 2012-07-04 | 中国科学院自动化研究所 | Complex background real-time alternating method based on background modeling and energy minimization |
| HUE061085T2 (en) * | 2010-09-20 | 2023-05-28 | Fraunhofer Ges Forschung | Method for differentiating background and foreground of a setting and method for replacing a background in images of a setting |
| US9576351B1 (en) * | 2015-11-19 | 2017-02-21 | Adobe Systems Incorporated | Style transfer for headshot portraits |
| US9773196B2 (en) * | 2016-01-25 | 2017-09-26 | Adobe Systems Incorporated | Utilizing deep learning for automatic digital image segmentation and stylization |
-
2018
- 2018-06-05 CN CN201810570632.XA patent/CN108765278B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016180224A1 (en) * | 2015-05-11 | 2016-11-17 | 北京锤子数码科技有限公司 | Method and device for processing image of person |
| CN105933613A (en) * | 2016-06-28 | 2016-09-07 | 广东欧珀移动通信有限公司 | Image processing method and apparatus and mobile terminal |
| WO2018075927A1 (en) * | 2016-10-21 | 2018-04-26 | Google Llc | Stylizing input images |
| CN106651766A (en) * | 2016-12-30 | 2017-05-10 | 深圳市唯特视科技有限公司 | Image style migration method based on deep convolutional neural network |
| CN107277391A (en) * | 2017-06-30 | 2017-10-20 | 北京奇虎科技有限公司 | Image switching network processing method, server, computing device and storage medium |
| CN107705242A (en) * | 2017-07-20 | 2018-02-16 | 广东工业大学 | A kind of image stylization moving method of combination deep learning and depth perception |
| CN107993191A (en) * | 2017-11-30 | 2018-05-04 | 腾讯科技（深圳）有限公司 | A kind of image processing method and device |
Non-Patent Citations (2)
| Title |
|---|
| Liqian Ma 等.Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency.《Computer Science》.2018,1-10. * |
| 杨汉宏.露天矿交通运输安全预警预控原理及实践.中国矿业大学出版社,2017,159-160. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108765278A (en) | 2018-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108765278B (en) | Image processing method, mobile terminal and computer readable storage medium | |
| CN108764091B (en) | Living body detection method and apparatus, electronic device, and storage medium | |
| US10872420B2 (en) | Electronic device and method for automatic human segmentation in image | |
| US8792722B2 (en) | Hand gesture detection | |
| WO2017190646A1 (en) | Facial image processing method and apparatus and storage medium | |
| US8750573B2 (en) | Hand gesture detection | |
| CN108428214B (en) | Image processing method and device | |
| US20170032222A1 (en) | Cross-trained convolutional neural networks using multimodal images | |
| CN109117773B (en) | Image feature point detection method, terminal device and storage medium | |
| CN107633237B (en) | Image background segmentation method, device, equipment and medium | |
| US10824910B2 (en) | Image processing method, non-transitory computer readable storage medium and image processing system | |
| CN110335216B (en) | Image processing method, image processing apparatus, terminal device, and readable storage medium | |
| CN111008935B (en) | Face image enhancement method, device, system and storage medium | |
| CN103617432A (en) | Method and device for recognizing scenes | |
| US9401027B2 (en) | Method and apparatus for scene segmentation from focal stack images | |
| CN111739027B (en) | Image processing method, device, equipment and readable storage medium | |
| CN110781770B (en) | Living body detection method, device and equipment based on face recognition | |
| CN108764139B (en) | Face detection method, mobile terminal and computer readable storage medium | |
| CN111784675A (en) | Method and device for processing article texture information, storage medium and electronic equipment | |
| CN113627428A (en) | Document image correction method and device, storage medium and intelligent terminal device | |
| CN108805838B (en) | Image processing method, mobile terminal and computer readable storage medium | |
| Yoo et al. | Red-eye detection and correction using inpainting in digital photographs | |
| CN113947613B (en) | Target area detection method, device, equipment and storage medium | |
| CN108769521B (en) | Photographing method, mobile terminal and computer readable storage medium | |
| CN108810407B (en) | Image processing method, mobile terminal and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |