CN112307983B - Method and system for enhancing plant color in image - Google Patents
Method and system for enhancing plant color in image Download PDFInfo
- Publication number
- CN112307983B CN112307983B CN202011202232.7A CN202011202232A CN112307983B CN 112307983 B CN112307983 B CN 112307983B CN 202011202232 A CN202011202232 A CN 202011202232A CN 112307983 B CN112307983 B CN 112307983B
- Authority
- CN
- China
- Prior art keywords
- plants
- plant
- green
- color
- original image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000002708 enhancing effect Effects 0.000 title claims abstract description 27
- 241001464837 Viridiplantae Species 0.000 claims abstract description 68
- 239000003086 colorant Substances 0.000 claims abstract description 23
- 241000196324 Embryophyta Species 0.000 claims description 331
- 241000124033 Salix Species 0.000 claims description 20
- 238000001514 detection method Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 14
- ZZUFCTLCJUWOSV-UHFFFAOYSA-N furosemide Chemical compound C1=C(Cl)C(S(=O)(=O)N)=CC(C(O)=O)=C1NCC1=CC=CO1 ZZUFCTLCJUWOSV-UHFFFAOYSA-N 0.000 claims description 10
- 241000218631 Coniferophyta Species 0.000 claims description 6
- 102000002322 Egg Proteins Human genes 0.000 claims description 6
- 108010000912 Egg Proteins Proteins 0.000 claims description 6
- 238000012790 confirmation Methods 0.000 claims description 6
- 210000004681 ovum Anatomy 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 230000001629 suppression Effects 0.000 claims description 6
- 241000894007 species Species 0.000 claims description 4
- 230000006870 function Effects 0.000 description 9
- 235000017166 Bambusa arundinacea Nutrition 0.000 description 4
- 235000017491 Bambusa tulda Nutrition 0.000 description 4
- 244000252337 Epipremnum pinnatum Species 0.000 description 4
- 235000015334 Phyllostachys viridis Nutrition 0.000 description 4
- 235000008331 Pinus X rigitaeda Nutrition 0.000 description 4
- 241000018646 Pinus brutia Species 0.000 description 4
- 235000011613 Pinus brutia Nutrition 0.000 description 4
- 240000003243 Thuja occidentalis Species 0.000 description 4
- 235000008109 Thuja occidentalis Nutrition 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- HXCILVUBKWANLN-UHFFFAOYSA-N brilliant green cation Chemical compound C1=CC(N(CC)CC)=CC=C1C(C=1C=CC=CC=1)=C1C=CC(=[N+](CC)CC)C=C1 HXCILVUBKWANLN-UHFFFAOYSA-N 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- DSSYKIVIOFKYAU-XCBNKYQSSA-N (R)-camphor Chemical compound C1C[C@@]2(C)C(=O)C[C@@H]1C2(C)C DSSYKIVIOFKYAU-XCBNKYQSSA-N 0.000 description 2
- 240000004144 Acer rubrum Species 0.000 description 2
- 235000011772 Acer rubrum var tomentosum Nutrition 0.000 description 2
- 235000009057 Acer rubrum var tridens Nutrition 0.000 description 2
- 241000209128 Bambusa Species 0.000 description 2
- 241001330002 Bambuseae Species 0.000 description 2
- 241000218993 Begonia Species 0.000 description 2
- 244000025254 Cannabis sativa Species 0.000 description 2
- 244000281594 Cassia siamea Species 0.000 description 2
- 235000009024 Ceanothus sanguineus Nutrition 0.000 description 2
- 241000218645 Cedrus Species 0.000 description 2
- 241000755729 Clivia Species 0.000 description 2
- 240000005636 Dryobalanops aromatica Species 0.000 description 2
- 240000003553 Leptospermum scoparium Species 0.000 description 2
- 241000830535 Ligustrum lucidum Species 0.000 description 2
- 235000015459 Lycium barbarum Nutrition 0.000 description 2
- 241001092500 Photinia x fraseri Species 0.000 description 2
- 241000218657 Picea Species 0.000 description 2
- 241000646858 Salix arbusculoides Species 0.000 description 2
- 241000219784 Sophora Species 0.000 description 2
- 239000011425 bamboo Substances 0.000 description 2
- 229960001506 brilliant green Drugs 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011282 treatment Methods 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
-
- G06T5/94—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Abstract
The invention provides a method and a system for enhancing plant colors in an image, wherein the method comprises the following steps: acquiring an original image of a color to be enhanced, and detecting and identifying the types of plants in the original image by using an SSD model; further judging the color of the plant according to the detected G/R, G/B values in RGB three channels in the plant area in the original image; and judging whether the plant is the subject of the original image; if the plant is the subject of the original image; and carrying out differential color enhancement on the plants in the original image according to the color characteristics of the plants. The invention solves the problems of insufficient bright colors of plants in the image and the like, and is particularly suitable for enhancing green of green plants in the image.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a method and a system for enhancing plant colors in an image.
Background
With the continuous improvement and development of digital camera technology, the requirements of people on image effects are also higher and higher, and the plants such as leaves or large-area grasslands which are expected to be shot look more vivid and discuss happiness; however, due to the inherent hardware limitations of the CMOS image sensor and the drawbacks of the ISP algorithm, the plants in the image are yellow and not bright enough.
In the experience and demand of users, the green of the enhanced plants accounts for most of the color enhancement demand, at present, most of the existing green enhancement algorithms judge green according to the difference between the G channel and the R, B channel in the live condition, and then perform green enhancement, or rotate or map the pixels of the yellow-green region on the YCbCr space to make the pixels more emerald green. These methods do not consider whether the plant photographed by the user is green, whether the green plant is the subject of photographing, and whether the green enhancement is performed to different degrees according to the kind of the green plant, and cannot meet the higher requirements of people on the green enhancement.
Disclosure of Invention
In view of the above-mentioned shortcomings of the prior art, the present invention proposes a method and system for enhancing plant color in an image. Aims to solve the problems that the plant color in the image is yellow, bright enough and the differential color enhancement can not be carried out.
In order to achieve the above purpose, the invention adopts the following technical scheme:
a first aspect of the present invention provides a method of enhancing plant colour in an image, the method comprising:
acquiring an original image of a color to be enhanced, and detecting and identifying the types of plants in the original image by using an SSD model;
further judging the color of the plant according to the detected G/R, G/B values in RGB three channels in the plant area in the original image; and judging whether the plant is the subject of the original image;
if the plant is the subject of the original image;
and carrying out differential color enhancement on the plants in the original image according to the color characteristics of the plants.
In a preferred aspect of the present invention, before the detection and the type recognition of the plant in the original image by using the SSD model, the method further includes training the SSD model, including:
collecting a plurality of sample images containing plants, and classifying according to the leaf shapes of the plants;
inputting a sample image into an SSD model, setting priori frames with different sizes, matching plant types with different shapes, and generating a corresponding feature map;
converting the feature map through a prediction convolution layer to obtain a prediction frame for distinguishing plant categories and corresponding category confidence;
and updating SSD model parameters through the category loss function and the category loss function to obtain an optimal SSD model.
In a preferred form provided by the first aspect of the invention, detecting and identifying species of plants in the original image comprises
Inputting an original image into a trained SSD model, arranging the boundary frames in a descending order through category confidence, and if the confidence is higher than a preset value, calculating the intersection ratio of the boundary frames and the prior frame;
and obtaining the optimal prediction frame and the category to which each category belongs by using a non-maximum suppression method, and outputting a recognition result.
In a preferred embodiment of the first aspect of the present invention, the determining the color of the plant includes obtaining edges of leaves of the plant through an SSD model, tracing a boundary area of the plant, and counting G/R of plant pixel points in the boundary area ave And G/B ave Judging the true color of the plant.
In a preferred embodiment of the first aspect of the present invention, the determining whether the plant is the subject of the original image is determined according to a position of the plant in the original image or/and a pixel ratio of the plant in the original image.
In a preferred aspect of the present invention, the determining whether the plant is a subject of the original image according to the position of the plant in the original image includes:
setting a certain point in the original image as an image center coordinate C (x, y), and setting a certain point of the plant area as a plant center coordinate G (x 1 ，y 1 ) The length and width of the original image are W, H respectively, and the distance between the center point of the plant and the center point of the image is
Judging the plant as the main body of the original image if the following conditions are met;
in a preferred aspect of the present invention, the determining whether a plant is a subject of an original image with a pixel ratio of the plant in the original image includes:
calculating the number n of pixels of the original image and calculating the number n of pixels of the plant area in the original image 1 ，
If n 1 And (4) not less than 0.4 x n, judging that the plant is the main body of the original image.
In a preferred embodiment provided in the first aspect of the present invention, the performing differential color enhancement on the plant in the original image according to the color feature of the plant includes:
acquiring plant input pixel values (R in ，G in ，B in ) The output value after the differential color enhancement is (R out ，G out ，B out ) The process of enhancing the color is:
R out ＝k 1 *R in +b 1
G out ＝k 2 *G in +b 2
B out ＝k 3 *B in +b 3
wherein k is 1 ，k 2 ，k 3 Representing the intensity coefficients of the RGB three channels, b 1 ，b 2 ，b 3 Representing the offset of the RGB three channels.
A second aspect of the present invention provides a system for enhancing plant colour in an image, the system comprising: an acquisition module, an SSD model, a color confirmation module, a main body judgment module and a color enhancement module, wherein,
the acquisition module is used for acquiring an original image with the color to be enhanced;
the SSD model is used for detecting and identifying the types of plants in the original image;
the color confirmation module is used for further judging the color of the plant according to the G/R, G/B values in the RGB three channels in the plant area detected in the original image;
the main body judging module is used for judging whether the plant is the main body of the original image;
if the plant is the main body of the original image, the color enhancement module is used for carrying out differential color enhancement on the plant in the original image according to the color characteristics of the plant.
In a preferred version provided by the second aspect of the present invention, training the SSD model is further included before detecting and identifying plants in the original image using the SSD model; training the SSD model includes:
collecting a plurality of sample images containing plants, and classifying according to the leaf shapes of the plants;
inputting a sample image into an SSD model, setting priori frames with different sizes, matching plant types with different shapes, and generating a corresponding feature map;
converting the feature map through a prediction convolution layer to obtain a prediction frame for distinguishing plant categories and corresponding category confidence;
and updating SSD model parameters through the category loss function and the category loss function to obtain an optimal SSD model.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a method and a system for enhancing plant colors in an image, wherein the method comprises the following steps: acquiring an original image of a color to be enhanced, and detecting and identifying the types of plants in the original image by using an SSD model; further judging the color of the plant according to the detected G/R, G/B values in RGB three channels in the plant area in the original image; and judging whether the plant is the subject of the original image; if the plant is the subject of the original image; and carrying out differential color enhancement on the plants in the original image according to the color characteristics of the plants. The invention solves the problems of insufficient bright colors of plants in the image and the like, and is particularly suitable for enhancing green of green plants in the image.
Drawings
FIG. 1 is a schematic block diagram of a method for enhancing plant color in an image according to the present invention.
FIG. 2 is a schematic block diagram of a process for training SSD models in accordance with the present invention.
FIG. 3 is a schematic block diagram of a system for enhancing plant color in an image in accordance with the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and more specific, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It should be noted that, the scheme of the invention is suitable for terminals with shooting functions, such as digital cameras, smart phones, tablet computers, PC devices and the like; by utilizing the scheme, the colors of the picture to be shot and the shot picture can be enhanced; in addition, in view of receiving information fed back from a user, various demands are often made on photographing green plants, and therefore, the following embodiments will describe color enhancement in detail taking an image to be photographed and green enhancement of a plant as an example.
It can be understood that the scheme is not only used for enhancing the green color of the plant, but also suitable for enhancing other colors such as the red color, the yellow color and the like of the plant; of course, it is understood that the present solution is not limited to enhancement of colors of plants, but is equally applicable to enhancement of colors of other subjects such as animals, apparel, and the like.
Referring to fig. 1, a first aspect of the present invention provides a method of enhancing plant colour in an image, the method comprising the steps of:
s10, acquiring an original image of a color to be enhanced, and detecting and identifying the types of plants in the original image by using an SSD model;
s20, further judging the color of the plant according to the G/R, G/B values in the RGB three channels in the plant area detected in the original image; and judging whether the plant is the subject of the original image;
s30, if the plant is the main body of the original image;
s40, carrying out differentiated color enhancement on the plants in the original image according to the color characteristics of the plants.
The SSD (Single Shot MultiBox Detector) model is a single-stage object detector. Different from the two-stage detection method, the single-stage target detection does not carry out region recommendation, but directly regresses the bounding box and the classification probability of the target from the feature map; the SSD uses the idea of this single-stage detection and improves it: targets of corresponding scales are detected on feature maps of different scales.
And detecting and classifying green plants by using an SSD model detection algorithm, and judging whether the categories of the green plants in the shot pictures are salix leaf plants, needle leaf plants, round or elliptic tip leaf plants and wide-oval or narrow-oval leaf plants according to the outline shapes of the plant leaves.
After the plant type is judged according to the outline shape of the plant leaf, the color of the plant in the original image is further judged according to the G/R, G/B value in RGB three channels in the plant area detected in the original image.
Judging whether the plant is a main shooting object in shooting according to the area ratio of the plant in the original image or/and the position relation of the plant in the original image; if so, the first and second data are not identical,
carrying out differential color enhancement according to leaf color characteristics of different leaf plants, such as salix leaf plants (such as grasslands, bamboos and the like), namely enhancing the leaf color of the plants to be yellow-green; for example, needle-leaf plants (e.g., pine, arborvitae, etc.) should have leaf color enhanced to blue-green, while oval, oval leaf plants (e.g., glossy privet, scindapsus aureus, etc.) should have leaf color enhanced to brilliant-green.
In conclusion, the invention solves the problems of insufficient bright colors of plants in the image and the like, and is particularly suitable for enhancing green of green plants in the image.
Referring to fig. 2, in a preferred embodiment provided in the first aspect of the present invention, before detecting and identifying the plant in the original image by using the SSD model, training the SSD model is further included, and the training method includes the following steps:
s11, collecting a plurality of sample images containing plants, and classifying according to the leaf shapes of the plants;
s12, inputting a sample image into an SSD model, setting priori frames with different sizes, matching plant types with different shapes, and generating a corresponding feature map;
s13, converting the feature map through a prediction convolution layer to obtain a prediction frame for distinguishing plant categories and corresponding category confidence degrees;
and S14, updating SSD model parameters through the category loss function and the category loss function to obtain an optimal SSD model.
The preparation of data samples, collecting a plurality of sample images containing common plants, wherein the larger the collected cardinality is, the better the plant types are; plants in these sample images are then labeled and can be divided into four categories: willow leaf shape plants (such as grass, bamboo leaf, clivia etc.); needle leaf plants (such as pine, spruce, cedar, arborvitae, etc.); round (including oval, oval tip) leaf plants (such as fructus Ligustri Lucidi, fructus Gardeniae, sophora tree, cinnamomum camphora, etc.); oval (including wide oval and narrow oval) leaf plants (such as scindapsus aureus, cassia tree, begonia, tea tree, etc.);
thus, the type of the plant in the sample image can be primarily judged, and the conventional color of the plant can be judged according to the type of the plant; of course, most plants are grown in a growing period and in a wilting period, and leaves of some plants are green in the growing period, but leaves of these plants may be yellow in the wilting period, so that further detection of the color of the plants in the original image is required.
In a preferred form provided by the first aspect of the invention, detecting and identifying species of plants in the original image comprises
Inputting an original image into a trained SSD model, arranging the boundary frames in a descending order through category confidence, and if the confidence is higher than a preset value, calculating the intersection ratio of the boundary frames and the prior frame;
and obtaining the optimal prediction frame and the category to which each category belongs by using a non-maximum suppression method, and outputting a recognition result.
Inputting plant pictures (original images) to be identified into a trained SSD model, arranging the boundary frames in a descending order through category confidence, calculating the intersection ratio of the boundary frames and the prior frames, obtaining the optimal prediction frames and the categories of each category by using a non-maximum suppression method, and outputting an identification result, wherein the boundary frames with the opposite confidence higher than 80% (the numerical value of the percentage can be freely set); namely four categories of plants: a willow leaf plant; a needle leaf plant; elliptic or elliptic pointed plants; wide-egg or narrow-egg plants.
In a preferred embodiment of the first aspect of the present invention, the determining the color of the plant includes obtaining edges of leaves of the plant through an SSD model, tracing a boundary area of the plant, and counting G/R of plant pixel points in the boundary area ave And G/B ave Judging the true color of the plant.
It can be understood that, because the intelligent recognition algorithm has a certain misjudgment and the growing period or withering period of the plants, in order to further improve the accuracy of green plant recognition, when multiple types of plants exist in the image and plants with other colors exist, such as red maple leaves or photinia fraseri, etc., the true color expression of the green plants or other plants with non-green colors needs to be further judged, and whether the green plants are the green plants of the type needs to be further confirmed according to the values of G/R, G/B in the RGB three channels in the detection area in the picture. The edge of the green leaf is obtained through the SSD model after training, the boundary area of the detected green plant is drawn, and G/Rave and G/Bave of green plant pixel points in the boundary area are counted, so that the following conditions are required to be met:
the Thrg/r and Thrg/b values should not be set too large, otherwise green plants would be misjudged as non-green plants. Since the green expression of leaf colors of different kinds of green plants is different, the enhancement is performed according to the true green expression of different plants; the specific implementation method comprises the following steps: the green color of different kinds of green plants in the image is different, and the proportion of RGB three channels in the corresponding areas is also different, for example, the leaf color of the salix leaf plant is yellow green, the g/r is relatively big, the leaf color of the conifer plant is blue green, and the g/b is relatively small. Therefore, to perform different treatments on different kinds of green plants, different threshold criteria need to be set according to the actual green appearance of the different kinds of green plants:
such as
Willow leaf shape plant: thrg/r=1.2 and Thrg/b=1.3;
needle leaf plant: thrg/r=1.3 and Thrg/b=1.2;
elliptic or circular plants: thrg/r=1.3 and Thrg/b=1.3;
wide-egg or narrow-egg plants: thrg/r=1.3 and Thrg/b=1.3.
It should be noted that, the green enhancement in the conventional sense is to enhance the green by adjusting the values of three channels R, G, B, and since the green enhancement is global and brings about a certain side effect, in order to weaken the influence of the part, the invention proposes an algorithm for judging whether a plant is the main body of the image, and mainly judges whether the plant is the main body of the image by two conditions, and can judge that the plant is the main body of the image by meeting any one of the following conditions:
firstly, calculating the position of the plant in the image;
secondly, calculating the area ratio of the plant in the image;
first, assuming that the center coordinates of the image are C (x, y), the center coordinates of the plant are G (x 1 ，y 1 ) The length and width of the image are W, H respectively, and the distance between the center point of the plant and the center point of the image is
Judging the plant as the main body of the original image if the following conditions are met;
secondly, when the central coordinates of the plant and the central coordinates of the image do not meet the conditions of the first step, but the pixel point ratio of the plant is higher, the plant can be judged to be the main body of the original image; because the plant information in the image is more, the image is observed or attracted by a large number of plants, so that the color enhancement can be performed for the scene;
assume that the number of pixel points in the plant area is n 1 The number of the whole pixel points of the image is n, if n 1 More than or equal to 0.4 x n, judging the plant as the main body of the original image;
the plant area accounts for more than forty percent of the whole image area, and of course, the plant area can be thirty percent or twenty percent, and the duty ratio value can be adjusted according to the preference of a user.
It will be appreciated that the above scheme has determined the plant in the original image, as well as the plant type, the color enhancement direction and whether the plant is the subject of the image, in a particular color enhancement, assuming that green enhancement is required for the plant in the image, the image input pixel value is (R in ，G in ，B in ) The output value after the differential color enhancement is (R out ，G out ，B out ) The green enhancement process can be implemented by the following formula:
R out ＝k 1 *R in +b 1
G out ＝k 2 *G in +b 2
B out ＝k 3 *B in +b 3
wherein k is 1 ，k 2 ，k 3 Representing the intensity coefficients of the RGB three channels, b 1 ，b 2 ，b 3 Representing the offset of the RGB three channels.
Empirically, the direction of enhancement is:
for example, salix plants are enhanced toward yellowish green:
k1＝1.1；k2＝1.1；k3＝1.0；b1＝5；b2＝5；b3＝0；
the conifer plants are enhanced towards bluish green:
k1＝1.0；k2＝1.1；k3＝1.1；b1＝0；b2＝5；b3＝5；
oval or round plants are enhanced towards a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5；
broad-oval or narrow-oval plants are enhanced toward a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5；
of course, the user can perform color enhancement for different kinds of green plants according to his own preference.
In summary, according to the method of the present invention, the plants in the image are identified, the colors of the plants are further confirmed, whether the plants are the main body of the image is confirmed, and finally, the plants in the image are subjected to differential color enhancement according to the above-mentioned points. The invention solves the problems of insufficient bright colors of plants in the image and the like, and is particularly suitable for enhancing green of green plants in the image.
Referring to fig. 3, a second aspect of the present invention provides a system for enhancing plant color in an image, the system comprising: an acquisition module 100, an SSD model 101, a color confirmation module 102, a body judgment module 103, and a color enhancement module 104, wherein,
the acquiring module 100 is configured to acquire an original image of a color to be enhanced;
the SSD model 101 is used for detecting and identifying the types of plants in the original image;
the color confirmation module 102 is configured to further determine a color of the plant according to the detected G/R, G/B values in the RGB three channels in the plant area in the original image;
the main body judging module 103 is used for judging whether the plant is the main body of the original image;
if the plant is the subject of the original image, the color enhancement module 104 performs differential color enhancement on the plant in the original image according to the color characteristics of the plant.
The SSD (Single Shot MultiBox Detector) model is a single-stage object detector. Different from the two-stage detection method, the single-stage target detection does not carry out region recommendation, but directly regresses the bounding box and the classification probability of the target from the feature map; the SSD uses the idea of this single-stage detection and improves it: targets of corresponding scales are detected on feature maps of different scales.
And detecting and classifying the green plants by using an SSD model detection algorithm, and judging whether the categories of the green plants in the shot pictures are salix leaf plants, needle leaf plants, round or elliptic tip green plants and wide ovum or narrow ovum leaf plants according to the outline shape of the plant leaves.
After the plant type is judged according to the outline shape of the plant leaf, the color of the plant in the original image is further judged according to the G/R, G/B value in RGB three channels in the plant area detected in the original image.
Judging whether the plant is a main shooting object in shooting according to the area ratio of the plant in the original image or/and the position relation of the plant in the original image; if so, the first and second data are not identical,
carrying out differential color enhancement according to leaf color characteristics of different leaf plants, such as salix leaf plants (such as grasslands, bamboos and the like), namely enhancing the leaf color of the plants to be yellow-green; for example, needle-leaf plants (e.g., pine, arborvitae, etc.) should have leaf color enhanced to blue-green, while oval, oval leaf plants (e.g., glossy privet, scindapsus aureus, etc.) should have leaf color enhanced to brilliant-green.
In conclusion, the invention solves the problems of insufficient bright colors of plants in the image and the like, and is particularly suitable for enhancing green of green plants in the image.
In a preferred version provided by the second aspect of the present invention, training the SSD model is further included before detecting and identifying plants in the original image using the SSD model; training the SSD model includes:
collecting a plurality of sample images containing plants, and classifying according to the leaf shapes of the plants;
inputting a sample image into an SSD model, setting priori frames with different sizes, matching plant types with different shapes, and generating a corresponding feature map;
converting the feature map through a prediction convolution layer to obtain a prediction frame for distinguishing plant categories and corresponding category confidence;
and updating SSD model parameters through the category loss function and the category loss function to obtain an optimal SSD model.
The preparation of data samples, collecting a plurality of sample images containing common plants, wherein the larger the collected cardinality is, the better the plant types are; plants in these sample images are then labeled and can be divided into four categories: willow leaf shape plants (such as grass, bamboo leaf, clivia etc.); needle leaf plants (such as pine, spruce, cedar, arborvitae, etc.); round (including oval, oval tip) leaf plants (such as fructus Ligustri Lucidi, fructus Gardeniae, sophora tree, cinnamomum camphora, etc.); oval (including wide oval and narrow oval) leaf plants (such as scindapsus aureus, cassia tree, begonia, tea tree, etc.);
thus, the type of the plant in the sample image can be primarily judged, and the conventional color of the plant can be judged according to the type of the plant; of course, most plants are grown in a growing period and in a wilting period, and leaves of some plants are green in the growing period, but leaves of these plants may be yellow in the wilting period, so that further detection of the color of the plants in the original image is required.
In a preferred embodiment provided in the second aspect of the present invention, detecting and identifying the kind of the plant in the original image includes:
inputting an original image into a trained SSD model, arranging the boundary frames in a descending order through category confidence, and if the confidence is higher than a preset value, calculating the intersection ratio of the boundary frames and the prior frame;
and obtaining the optimal prediction frame and the category to which each category belongs by using a non-maximum suppression method, and outputting a recognition result.
Inputting plant pictures to be identified into a trained SSD model, arranging the boundary frames in a descending order through category confidence, comparing the boundary frames with confidence higher than 80%, calculating the intersection ratio of the boundary frames and the prior frames, obtaining the optimal prediction frames and the categories to which each category belongs by using a non-maximum suppression method, and outputting an identification result; namely four categories of plants: a willow leaf plant; a needle leaf plant; elliptic or elliptic pointed plants; wide-egg or narrow-egg plants.
The judging of the plant color comprises obtaining the edge of the plant leaf through SSD model, drawing out the boundary area of the plant, counting the G/R of plant pixel points in the boundary area ave And G/B ave Judging the true color of the plant.
It can be understood that, because of a certain misjudgment of the intelligent recognition algorithm and the growing period or withering period of the plants, in order to further improve the accuracy of green plant recognition, when multiple types of plants exist in the image and plants with special colors such as red maple leaves or photinia fraseri exist in the image, the true color expression of the green plants or other plants with non-green colors needs to be further judged, and whether the green plants are the green plants of the type needs to be further confirmed according to the values of G/R, G/B in the RGB three channels in the detection area in the picture. The edge of the green leaf is obtained through the SSD model after training, the boundary area of the detected green plant is drawn, and G/Rave and G/Bave of green plant pixel points in the boundary area are counted, so that the following conditions are required to be met:
the Thrg/r and Thrg/b values should not be set too large, otherwise green plants would be misjudged as non-green plants. Since the green expression of leaf colors of different kinds of green plants is different, the enhancement is performed according to the true green expression of different plants; the specific implementation method comprises the following steps: the green color of different kinds of green plants in the image is different, and the proportion of RGB three channels in the corresponding areas is also different, for example, the leaf color of the salix leaf plant is yellow green, the g/r is relatively big, the leaf color of the conifer plant is blue green, and the g/b is relatively small. Therefore, to perform different treatments on different kinds of green plants, different threshold criteria need to be set according to the actual green appearance of the different kinds of green plants:
such as
Willow leaf shape plant: thrg/r=1.2 and Thrg/b=1.3;
needle leaf plant: thrg/r=1.3 and Thrg/b=1.2;
elliptic or circular plants: thrg/r=1.3 and Thrg/b=1.3;
wide-egg or narrow-egg plants: thrg/r=1.3 and Thrg/b=1.3.
It should be noted that, the green enhancement in the conventional sense is to enhance the green by adjusting the values of three channels R, G, B, and since the green enhancement is global and brings about a certain side effect, in order to weaken the influence of the part, the invention proposes an algorithm for judging whether a plant is the main body of the image, and mainly judges whether the plant is the main body of the image by two conditions, and can judge that the plant is the main body of the image by meeting any one of the following conditions:
firstly, calculating the position of the plant in the image;
secondly, calculating the area ratio of the plant in the image;
first, assuming that the center coordinates of the image are C (x, y), the center coordinates of the plant are G (x 1 ，y 1 ) The length and width of the image are W, H respectively, and the distance between the center point of the plant and the center point of the image is
Judging the plant as the main body of the original image if the following conditions are met;
secondly, when the central coordinates of the plant and the central coordinates of the image do not meet the conditions of the first step, but the pixel point ratio of the plant is higher, the plant can be judged to be the main body of the original image; because the plant information in the image is more, the image is observed or attracted by a large number of plants, so that the color enhancement can be performed for the scene;
assume that the number of pixel points in the plant area is n 1 The number of the whole pixel points of the image is n, if n 1 More than or equal to 0.4 x n, judging the plant as the main body of the original image;
the plant area accounts for more than forty percent of the whole image area, and of course, the plant area can be thirty percent or twenty percent, and the duty ratio value can be adjusted according to the preference of a user.
It will be appreciated that the above scheme has determined the plant in the original image, as well as the plant type, the color enhancement direction and whether the plant is the subject of the image, in a particular color enhancement, assuming that green enhancement is required for the plant in the image, the image input pixel value is (R in ，G in ，B in ) The output value after the differential color enhancement is (R out ，G out ，B out ) The green enhancement process can be implemented by the following formula:
R out ＝k 1 *R in +b 1
G out ＝k 2 *G in +b 2
B out ＝k 3 *B in +b 3
wherein k is 1 ，k 2 ，k 3 Representing the intensity coefficients of the RGB three channels, b 1 ，b 2 ，b 3 Representing the offset of the RGB three channels.
Empirically, the direction of enhancement is:
for example, salix plants are enhanced toward yellowish green:
k1＝1.1；k2＝1.1；k3＝1.0；b1＝5；b2＝5；b3＝0；
the conifer plants are enhanced towards bluish green:
k1＝1.0；k2＝1.1；k3＝1.1；b1＝0；b2＝5；b3＝5；
oval or round plants are enhanced towards a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5；
broad-oval or narrow-oval plants are enhanced toward a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5；
of course, the user can perform color enhancement for different kinds of green plants according to his own preference.
In summary, according to the system disclosed by the invention, the plants in the image are identified, the colors of the plants are further confirmed, whether the plants are the main body of the image or not is confirmed, and finally, the plants in the image are subjected to differentiated color enhancement according to the key points. The invention solves the problems of insufficient bright colors of plants in the image and the like, and is particularly suitable for enhancing green of green plants in the image.
It will be understood that all or part of the steps in the methods according to the above embodiments may be implemented by a program for instructing relevant hardware, and the program may be stored in a storage medium readable by a computer device, for performing all or part of the steps in the methods according to the above embodiments. The computer device includes, but is not limited to: personal computers, servers, general purpose computers, special purpose computers, network devices, embedded devices, programmable devices, intelligent mobile terminals, intelligent home devices, wearable intelligent devices, vehicle-mounted intelligent devices and the like; the storage medium includes, but is not limited to: RAM, ROM, magnetic disk, magnetic tape, optical disk, flash memory, usb disk, removable hard disk, memory card, memory stick, web server storage, web cloud storage, etc.
It will be understood that equivalents and modifications will occur to those skilled in the art in light of the present invention and their spirit, and all such modifications and substitutions are intended to be included within the scope of the present invention as defined in the following claims.
Claims (9)
1. A method of enhancing plant color in an image, the method comprising:
acquiring an original image of a color to be enhanced, and detecting and identifying the types of plants in the original image by using an SSD model; the detection and the type identification of the plants in the original image are to judge whether the categories of the green plants in the shot pictures are salix leaf plants, needle leaf plants, round or elliptic sharp green plants and wide ovum or narrow ovum leaf plants according to the outline shape of the plant leaves;
further judging the color of the plant according to the detected G/R, G/B values in RGB three channels in the plant area in the original image; and judging whether the plant is the subject of the original image; further judging the true color expression of green plants or other non-green plants, further confirming whether the green plants are the green plants according to the values of G/R, G/B in RGB three channels in a detection area in a picture, acquiring the edges of green leaves through the SSD model, drawing out the boundary area of the detected green plants, and counting the G/Rave and G/Bave of green plant pixel points in the boundary area, wherein the following conditions are required to be satisfied:
the Thrg/r and the Thrg/b are not suitable to be set too large, otherwise, green plants are misjudged as non-green plants, and the green expression of the leaf colors of different types of green plants is enhanced according to the actual green expression of the different plants; the specific implementation method comprises the following steps: the green color of different kinds of green plants in the image is different, the proportion of RGB three channels in the corresponding areas is also different, for example, the leaf color of the salix leaf plant is yellow green, the g/r is relatively large, the leaf color of the coniform plant is blue green, and the g/b is relatively small, so that different threshold standards are set according to the actual green color of different kinds of green plants in order to treat different kinds of green plants:
willow leaf shape plant: thrg/r=1.2 and Thrg/b=1.3;
needle leaf plant: thrg/r=1.3 and Thrg/b=1.2;
elliptic or circular plants: thrg/r=1.3 and Thrg/b=1.3;
wide-egg or narrow-egg plants: thrg/r=1.3 and Thrg/b=1.3;
if the plant is the subject of the original image;
differential color enhancement of plants in an original image based on the color characteristics of the plants is embodied by obtaining input pixel values (R in ，G in ，B in ) The output value after the differential color enhancement is (R out ，G out ，B out ) The process of enhancing the color is:
R out ＝k 1 *R in +b 1
G out ＝k 2 *G in +b 2
B out ＝k 3 *B in +b 3
wherein k is 1 ，k 2 ，k 3 Representing the intensity coefficients of the RGB three channels, b 1 ，b 2 ，b 3 Representing the offset of RGB three channels;
the enhancement direction is:
the salix plants are enhanced towards yellow-green:
k1＝1.1；k2＝1.1；k3＝1.0；b1＝5；b2＝5；b3＝0；
the conifer plants are enhanced towards bluish green:
k1＝1.0；k2＝1.1；k3＝1.1；b1＝0；b2＝5；b3＝5；
oval or round plants are enhanced towards a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5；
broad-oval or narrow-oval plants are enhanced toward a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5。
2. the method of claim 1, further comprising training the SSD model prior to detecting and identifying the species of the plant in the original image using the SSD model, the training method comprising:
collecting a plurality of sample images containing plants, and classifying according to the leaf shapes of the plants;
inputting a sample image into an SSD model, setting priori frames with different sizes, matching plant types with different shapes, and generating a corresponding feature map;
converting the feature map through a prediction convolution layer to obtain a prediction frame for distinguishing plant categories and corresponding category confidence;
and updating SSD model parameters through the category loss function and the category loss function to obtain an optimal SSD model.
3. The method of claim 2, wherein detecting and identifying species of plants in the original image comprises
Inputting an original image into a trained SSD model, arranging the boundary frames in a descending order through category confidence, and if the confidence is higher than a preset value, calculating the intersection ratio of the boundary frames and the prior frame;
and obtaining the optimal prediction frame and the category to which each category belongs by using a non-maximum suppression method, and outputting a recognition result.
4. The method of claim 1, wherein determining the color of the plant comprises obtaining edges of leaves of the plant by an SSD model, tracing a border area of the plant, and counting G/R of plant pixels in the border area ave And G/B ave Judging the true color of the plant.
5. The method of claim 1, wherein the determining whether the plant is the subject of the original image is determined based on a position of the plant in the original image or/and a pixel ratio of the plant in the original image.
6. The method of claim 5, wherein determining whether the plant is the subject of the original image based on the position of the plant in the original image comprises:
setting a certain point in the original image as an image center coordinate C (x, y), and setting a certain point of the plant area as a plant center coordinate C (x 1 ，y 1 ) The length and width of the original image are W, H respectively, and the distance between the center point of the plant and the center point of the image is
Judging the plant as the main body of the original image if the following conditions are met;
7. the method of claim 5, wherein determining whether a plant is the subject of an original image at a pixel fraction of the plant in the original image comprises:
calculating the number of pixels of the original imageQuantity n, and calculating the number n of pixel points of the plant area in the original image 1 ，
If n 1 And (4) not less than 0.4 x n, judging that the plant is the main body of the original image.
8. A system for enhancing plant color in an image, the system comprising:
an acquisition module, an SSD model, a color confirmation module, a main body judgment module and a color enhancement module, wherein,
the acquisition module is used for acquiring an original image with the color to be enhanced;
detecting and identifying the types of plants in the original image by using an SSD model; the detection and the type identification of the plants in the original image are to judge whether the categories of the green plants in the shot pictures are salix leaf plants, needle leaf plants, round or elliptic sharp green plants and wide ovum or narrow ovum leaf plants according to the outline shape of the plant leaves;
the color confirmation module is used for further judging the color of the plant according to the G/R, G/B values in the RGB three channels in the plant area detected in the original image; further judging the true color expression of green plants or other non-green plants, further confirming whether the green plants are the green plants according to the values of G/R, G/B in RGB three channels in a detection area in a picture, acquiring the edges of green leaves through the SSD model, drawing out the boundary area of the detected green plants, and counting the G/Rave and G/Bave of green plant pixel points in the boundary area, wherein the following conditions are required to be satisfied:
the Thrg/r and the Thrg/b are not suitable to be set too large, otherwise, green plants are misjudged as non-green plants, and the green expression of the leaf colors of different types of green plants is enhanced according to the actual green expression of the different plants; the specific implementation method comprises the following steps: the green color of different kinds of green plants in the image is different, the proportion of RGB three channels in the corresponding areas is also different, for example, the leaf color of the salix leaf plant is yellow green, the g/r is relatively large, the leaf color of the coniform plant is blue green, and the g/b is relatively small, so that different threshold standards are set according to the actual green color of different kinds of green plants in order to treat different kinds of green plants:
willow leaf shape plant: thrg/r=1.2 and Thrg/b=1.3;
needle leaf plant: thrg/r=1.3 and Thrg/b=1.2;
elliptic or circular plants: thrg/r=1.3 and Thrg/b=1.3;
wide-egg or narrow-egg plants: thrg/r=1.3 and Thrg/b=1.3;
the main body judging module is used for judging whether the plant is the main body of the original image;
if the plant is the subject of the original image, differentiating the plant in the original image by the color enhancement module according to the color characteristics of the plant, which is embodied by obtaining the input pixel value (R in ，G in ，B in ) The output value after the differential color enhancement is (R out ，G out ，B out ) The process of enhancing the color is:
R out ＝k 1 *R in +b 1
G o u t ＝k 2 *G in +b 2
B out ＝k 3 *B in +b 3
wherein k is 1 ，k 2 ，k 3 Representing the intensity coefficients of the RGB three channels, b 1 ，b 2 ，b 3 Representing the offset of RGB three channels;
the enhancement direction is:
the salix plants are enhanced towards yellow-green:
k1＝1.1；k2＝1.1；k3＝1.0；b1＝5；b2＝5；b3＝0；
the conifer plants are enhanced towards bluish green:
k1＝1.0；k2＝1.1；k3＝1.1；b1＝0；b2＝5；b3＝5；
oval or round plants are enhanced towards a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5；
broad-oval or narrow-oval plants are enhanced toward a bright green color:
k1＝1.1；k2＝1.1；k3＝1.1；b1＝5；b2＝5；b3＝5。
9. the system of claim 8, further comprising training the SSD model prior to detecting plants in the identified original image with the SSD model; training the SSD model includes:
collecting a plurality of sample images containing plants, and classifying according to the leaf shapes of the plants;
inputting a sample image into an SSD model, setting priori frames with different sizes, matching plant types with different shapes, and generating a corresponding feature map;
converting the feature map through a prediction convolution layer to obtain a prediction frame for distinguishing plant categories and corresponding category confidence;
and updating SSD model parameters through the category loss function and the category loss function to obtain an optimal SSD model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011202232.7A CN112307983B (en) | 2020-11-02 | 2020-11-02 | Method and system for enhancing plant color in image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011202232.7A CN112307983B (en) | 2020-11-02 | 2020-11-02 | Method and system for enhancing plant color in image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112307983A CN112307983A (en) | 2021-02-02 |
| CN112307983B true CN112307983B (en) | 2024-03-26 |
Family
ID=74333337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011202232.7A Active CN112307983B (en) | 2020-11-02 | 2020-11-02 | Method and system for enhancing plant color in image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112307983B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101193316A (en) * | 2006-11-21 | 2008-06-04 | 安凯（广州）软件技术有限公司 | A self-adapted white balance correction method |
| CN106709922A (en) * | 2016-12-29 | 2017-05-24 | 江苏省无线电科学研究所有限公司 | Image based forage grass coverage and biomass automatic detection method |
| CN107958470A (en) * | 2017-12-18 | 2018-04-24 | 维沃移动通信有限公司 | A kind of color correcting method, mobile terminal |
| CN110991454A (en) * | 2019-12-23 | 2020-04-10 | 云南大学 | Blade image recognition method and device, electronic equipment and storage medium |
| CN111163628A (en) * | 2017-05-09 | 2020-05-15 | 蓝河技术有限公司 | Automatic plant detection using image data |
| CN111476280A (en) * | 2020-03-27 | 2020-07-31 | 海南医学院 | Plant leaf identification method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016111376A1 (en) * | 2015-01-09 | 2016-07-14 | 日立マクセル株式会社 | Plant information acquisition system, plant information acquisition device, plant information acquisition method, crop management system, and crop management method |
-
2020
- 2020-11-02 CN CN202011202232.7A patent/CN112307983B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101193316A (en) * | 2006-11-21 | 2008-06-04 | 安凯（广州）软件技术有限公司 | A self-adapted white balance correction method |
| CN106709922A (en) * | 2016-12-29 | 2017-05-24 | 江苏省无线电科学研究所有限公司 | Image based forage grass coverage and biomass automatic detection method |
| CN111163628A (en) * | 2017-05-09 | 2020-05-15 | 蓝河技术有限公司 | Automatic plant detection using image data |
| CN107958470A (en) * | 2017-12-18 | 2018-04-24 | 维沃移动通信有限公司 | A kind of color correcting method, mobile terminal |
| CN110991454A (en) * | 2019-12-23 | 2020-04-10 | 云南大学 | Blade image recognition method and device, electronic equipment and storage medium |
| CN111476280A (en) * | 2020-03-27 | 2020-07-31 | 海南医学院 | Plant leaf identification method and system |
Non-Patent Citations (4)
| Title |
|---|
| A Review of Visual Descriptors and Classification Techniques Used in Leaf Species Identification;K.K.Thyagharajan 等;《Archives of Computational Methods in Engineering》;933-960 * |
| On Solving leaf classification using linear regression;Neha Goyal 等;《Multimedia Tools and Applications》;4533-4551 * |
| 景深差分提取绿色植物的真彩色夜视图像颜色校正;赵振芬;陈远金;张猛蛟;陈东启;王岭雪;蔡毅;;《红外技术》;第42卷(第09期);886-892 * |
| 用于低照度图像增强的自适应颜色保持算法;朱德利;杨德刚;万辉;杨雨浓;;《计算机工程与应用》;第55卷(第24期);190-195+213 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112307983A (en) | 2021-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106874871B (en) | Living body face double-camera identification method and identification device | |
| CN104850850B (en) | A kind of binocular stereo vision image characteristic extracting method of combination shape and color | |
| CN107862698B (en) | Light field foreground segmentation method and device based on K mean cluster | |
| CN104364798B (en) | System and method for face verification | |
| US7218759B1 (en) | Face detection in digital images | |
| US8660342B2 (en) | Method to assess aesthetic quality of photographs | |
| CN105184787B (en) | A kind of license camera and method for portrait scratch figure automatically | |
| US8385638B2 (en) | Detecting skin tone in images | |
| CN108717524B (en) | Gesture recognition system based on double-camera mobile phone and artificial intelligence system | |
| CN103634680B (en) | The control method for playing back and device of a kind of intelligent television | |
| CN103714181B (en) | A kind of hierarchical particular persons search method | |
| CN107742274A (en) | Image processing method, device, computer-readable recording medium and electronic equipment | |
| CN106960182B (en) | A kind of pedestrian's recognition methods again integrated based on multiple features | |
| CN103902958A (en) | Method for face recognition | |
| CN102881160B (en) | Outdoor traffic sign identification method under low-illumination scene | |
| Obrador et al. | Towards category-based aesthetic models of photographs | |
| CN104361357B (en) | Photo album categorizing system and sorting technique based on image content analysis | |
| CN104636759A (en) | Method for obtaining picture recommending filter information and picture filter information recommending system | |
| CN103440035A (en) | Gesture recognition system in three-dimensional space and recognition method thereof | |
| Čuljak et al. | Classification of art paintings by genre | |
| CN109919224B (en) | Artificial intelligence-based identification method for true and false mixed traditional Chinese medicinal materials | |
| CN112434647A (en) | Human face living body detection method | |
| CN109587394A (en) | A kind of intelligence patterning process, electronic equipment and storage medium | |
| CN112307983B (en) | Method and system for enhancing plant color in image | |
| CN110503051A (en) | A kind of precious timber identifying system and method based on image recognition technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |