CN113837316A - Method, device, equipment and medium for detecting abnormal area based on agricultural products - Google Patents
Method, device, equipment and medium for detecting abnormal area based on agricultural products Download PDFInfo
- Publication number
- CN113837316A CN113837316A CN202111193335.6A CN202111193335A CN113837316A CN 113837316 A CN113837316 A CN 113837316A CN 202111193335 A CN202111193335 A CN 202111193335A CN 113837316 A CN113837316 A CN 113837316A
- Authority
- CN
- China
- Prior art keywords
- abnormal
- agricultural product
- region detection
- training
- detection model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 228
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000001514 detection method Methods 0.000 claims abstract description 230
- 238000012549 training Methods 0.000 claims abstract description 193
- 238000006243 chemical reaction Methods 0.000 claims abstract description 13
- 238000003860 storage Methods 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 claims description 8
- 239000000126 substance Substances 0.000 claims description 7
- 201000010099 disease Diseases 0.000 claims description 4
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 235000014787 Vitis vinifera Nutrition 0.000 description 19
- 241000219095 Vitis Species 0.000 description 17
- 235000009754 Vitis X bourquina Nutrition 0.000 description 17
- 235000012333 Vitis X labruscana Nutrition 0.000 description 17
- 239000002420 orchard Substances 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 208000024891 symptom Diseases 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 241000700605 Viruses Species 0.000 description 6
- 208000015181 infectious disease Diseases 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 239000000843 powder Substances 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 241000221785 Erysiphales Species 0.000 description 2
- 241000219094 Vitaceae Species 0.000 description 2
- 240000006365 Vitis vinifera Species 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000001580 bacterial effect Effects 0.000 description 2
- 244000052616 bacterial pathogen Species 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 235000021021 grapes Nutrition 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 101001018064 Homo sapiens Lysosomal-trafficking regulator Proteins 0.000 description 1
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 102100033472 Lysosomal-trafficking regulator Human genes 0.000 description 1
- 244000038561 Modiola caroliniana Species 0.000 description 1
- 235000010703 Modiola caroliniana Nutrition 0.000 description 1
- 241000233679 Peronosporaceae Species 0.000 description 1
- 238000012271 agricultural production Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000017074 necrotic cell death Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 241000894007 species Species 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
Abstract
The invention relates to the field of artificial intelligence, and discloses an abnormal region detection method and device based on agricultural products, electronic equipment and a storage medium, wherein the method comprises the following steps: receiving an original training atlas of agricultural products, dividing the original training atlas to obtain a normal agricultural product training set and an abnormal agricultural product training set, constructing an abnormal region detection model to be trained, training the abnormal region detection model to be trained by using the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model, receiving a picture of the agricultural products to be detected, performing picture conversion on the picture of the agricultural products to be detected according to the input requirement of the abnormal region detection model to obtain a converted picture to be detected, inputting the converted picture to be detected to the abnormal region detection model to perform abnormal region detection, and obtaining an abnormal region detection result. The invention can solve the problems of low detection efficiency and low detection intelligent degree of abnormal region detection of agricultural products.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to an abnormal region detection method and device based on agricultural products, electronic equipment and a computer readable storage medium.
Background
Along with the development of science and technology, how to efficiently improve the upgrading of industrial structures is the current technical direction of fire and heat, for example, in the agricultural field, abnormal areas such as necrosis and virus invasion of agricultural products are intelligently detected, the structure upgrading in the agricultural field can be effectively improved, and the production efficiency of agricultural production is improved.
At present, the abnormal region detection technology based on agricultural products mainly relies on experienced experts to observe a shot picture, and computer calibration software is utilized to calibrate abnormal regions in the shot picture. In order to find out which grape trees infected by viruses exist in the grape orchard in time, an aerial photography graph of the grape orchard is firstly aerial photographed, and then an area of the grape trees infected by the viruses is marked in the aerial photography graph by means of experienced agriculture and forestry experts and computer calibration software, so that abnormal area detection is completed.
Although the method can realize the detection of the abnormal area, the detection efficiency of the detection of the abnormal area is low and the detection intellectualization needs to be further improved due to excessive manual intervention.
Disclosure of Invention
In order to solve the technical problems, the invention provides an abnormal region detection method and device based on agricultural products, an electronic device and a computer readable storage medium, which can solve the problems of low detection efficiency and low detection intelligence degree of abnormal region detection based on agricultural products.
In a first aspect, the present invention provides a method for detecting an abnormal area based on an agricultural product, including:
receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set;
constructing an abnormal region detection model of an agricultural product to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
receiving an image of an agricultural product to be detected, and performing image conversion on the image of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain a converted image to be detected;
and inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection to obtain an abnormal area detection result.
It can be seen that, in the embodiment of the present invention, an original training atlas of an agricultural product is received, according to whether each original training atlas in the original training atlas includes an abnormal region, the original training atlas is divided to obtain a normal agricultural product training set and an abnormal agricultural product training set, and abnormal region detection of the agricultural product can be performed only after an abnormal region detection model to be trained needs to be trained, so that training of the abnormal region detection model to be trained can be realized through the normal agricultural product training set and the abnormal agricultural product training set obtained by dividing the abnormal region, thereby providing necessary conditions for subsequent abnormal region detection, in addition, according to the input requirement of the abnormal region detection model, image transformation is performed on the image of the agricultural product to be detected to obtain a transformed image, and further, the abnormal region detection model is utilized to complete final abnormal region detection, compared with the background technology, the whole detection process is an automatic detection process without human intervention. Therefore, the abnormal region detection method based on the agricultural products, provided by the embodiment of the invention, can solve the problems of low detection efficiency and low detection intelligence degree of the abnormal region detection based on the agricultural products.
In a possible implementation manner of the first aspect, the constructing an abnormal region detection model to be trained of an agricultural product includes:
acquiring a trained 16-layer VGG network;
replacing the fully-connected layers in the 16-layer VGG network with a first preset number of convolutional layers to obtain an improved VGG network;
and adding a second preset number of convolutional layers at the tail part of the improved VGG network to obtain the abnormal region detection model to be trained.
In a possible implementation manner of the first aspect, the training the abnormal area detection model to be trained by using the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal area detection model includes:
constructing a detection frame generation algorithm, and generating a corresponding region detection regression formula according to the detection frame generation algorithm;
receiving training learning rate and batch size input by a user for training the abnormal region detection model to be trained;
taking the normal agricultural product training set as a positive sample, taking the abnormal agricultural product training set as a negative sample, and inputting the negative sample into the abnormal area detection model to be trained;
according to the training learning rate and the batch size, performing first feature extraction on the positive sample and the negative sample by using the improved VGG network to obtain a first positive sample feature set and a first negative sample feature set;
performing second feature extraction on the first positive sample feature set and the first negative sample feature set by using a second preset number of the convolutional layers to obtain a second positive sample feature set and a second negative sample feature set;
performing region detection frame prediction on the second positive sample feature set and the second negative sample feature set by using the region detection regression formula to obtain a region detection frame set;
adjusting the training learning rate, the batch size and the internal parameters of the abnormal region detection model to be trained by using the region detection frame set, and returning to the first feature extraction step;
and determining the abnormal region detection model to be trained as the abnormal region detection model until the training learning rate, the batch size and the adjustment times of the internal parameters of the abnormal region detection model to be trained are adjusted to be larger than a specified adjustment threshold.
In a possible implementation manner of the first aspect, the building a detection box generation algorithm includes:
the following detection box generation algorithm is adopted:
wherein the content of the first and second substances,represents the maximum and minimum proportion of the preset detection frame to each original training image in the original training images,representing the number of detection boxes in each original training image,the first to represent each original training diagramAnd predicting the position of each detection frame.
In a possible implementation manner of the first aspect, the generating a corresponding region detection regression formula according to the detection box generation algorithm includes:
generating the region detection regression formula by adopting the following method:
wherein the content of the first and second substances,representing the region detection regression formula,is the number of positive samples included within the generated detection box,to indicate a function, take the value (0,1),to detect the predicted values of positive or negative samples within the box,in order to detect the position prediction value of the frame,position parameters of real frames of agricultural products in the inner circles of the positive sample and the negative sample,in order to adjust the parameters of the device,represents a regression function of the position of the detection frame,representing a check box class regression function.
In a possible implementation manner of the first aspect, the receiving a picture of an agricultural product to be detected includes:
receiving the geographical position of an agricultural product to be detected;
executing shooting at the geographic position by means of shooting equipment to generate a panoramic photo of the agricultural product to be detected;
and receiving the picture interception of the panoramic picture by the user to obtain the picture of the agricultural product to be detected.
In a possible implementation manner of the first aspect, the dividing the original training image set according to whether each original training image in the original training image set includes an abnormal region or not to obtain a normal agricultural product training set and an abnormal agricultural product training set includes:
transmitting each of the original training images to a researcher studying the agricultural product;
receiving a returned result of whether the researcher includes diseases or not for each original training image;
and dividing the original training atlas according to the returned result to obtain the normal agricultural product training set and the abnormal agricultural product training set.
In a second aspect, the present invention provides an abnormal area detection apparatus based on agricultural products, the apparatus comprising:
the training image set classification module is used for receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set;
the model training module is used for constructing an abnormal region detection model of agricultural products to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
the image conversion module is used for receiving an image of the agricultural product to be detected, and executing image conversion on the image of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain a converted image to be detected;
and the abnormal area detection module is used for inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection so as to obtain an abnormal area detection result.
In a third aspect, the present invention provides an electronic device comprising:
at least one processor; and a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the agricultural product-based abnormal area detection method according to any one of the first aspects.
In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the agricultural product-based abnormal area detection method according to any one of the first aspects.
It is understood that the beneficial effects of the second to fourth aspects can be seen from the description of the first aspect, and are not described herein again.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a detailed flowchart of an abnormal area detection method based on agricultural products according to an embodiment of the present invention;
fig. 2 is a model structure diagram of an abnormal region detection method based on agricultural products according to an embodiment of the present invention, which is provided in fig. 1;
FIG. 3 is a schematic flow chart illustrating another step of the method for detecting abnormal regions based on agricultural products of FIG. 1 according to an embodiment of the present invention;
FIG. 4 is a block diagram of an abnormal area detecting device for agricultural products according to an embodiment of the present invention;
fig. 5 is a schematic internal structural diagram of an electronic device for implementing an abnormal region detection method based on agricultural products according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
A method for detecting an abnormal area based on agricultural products according to an embodiment of the present invention is described with reference to a flowchart shown in fig. 1. The method for detecting the abnormal region based on the agricultural product, which is described in the figure 1, comprises the following steps:
s1, receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set.
It should be explained that the present invention can obtain the original training atlas from the network, each agricultural product APP, etc. Further, division may be manually performed to sort and screen out a normal agricultural product training set and an abnormal agricultural product training set, and in detail, according to whether each original training image in the original training image set includes an abnormal region, division is performed on the original training image set to obtain a normal agricultural product training set and an abnormal agricultural product training set, including:
transmitting each of the original training images to a researcher studying the agricultural product;
receiving a returned result of whether the researcher includes diseases or not for each original training image;
and dividing the original training atlas according to the returned result to obtain the normal agricultural product training set and the abnormal agricultural product training set.
In an embodiment of the present invention, taking grapes as agricultural products as an example, the original training atlas includes grape pictures, and therefore, according to whether each original training atlas in the original training atlas includes an abnormal region, the original training atlas is divided to obtain a normal agricultural product training set and an abnormal agricultural product training set, which may be replaced with:
receiving a list of infection types infected with a virus, wherein the list of infection types includes a description of infection types and symptoms;
sending the original training atlas to a picture classification person;
and receiving the normal agricultural product training set and the abnormal agricultural product training set which are obtained by classifying the original training atlas by a classifier according to the infection type and symptom description.
Illustratively, the grapes infected by the virus mainly comprise six infection species, respectively, anthracnose, downy mildew, bacterial wilt, brown spot and powdery mildew, and the description of each symptom is shown in table 1, such as the description of the symptom of bacterial wilt: the whole leaves of the grapevine tree can generate round black brown specks, the middle part of the grape tree becomes grey white, and small black spots (which are conidiophores and ascochy shells of pathogenic bacteria) can be densely generated; description of symptoms of powdery mildew: the whole leaf surface of the grapevine tree is covered with off-white powder (wherein the off-white powder is mycelium and conidium of pathogenic bacteria), and when the disease is serious, the whole leaf surface is covered with the white powder; description of symptoms of anthracnose: the center of the page is gray, slightly concave, and the edge is dark brown or mauve, and the leaf may have perforation, shrinkage deformity, etc. Therefore, the grape pictures can be effectively divided into a normal grape training set and an abnormal grape training set according to the description of each symptom.
Table 1 agricultural product symptom description table based on abnormal region detection method of agricultural product
S2, constructing an abnormal region detection model of the agricultural product to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain the abnormal region detection model.
Referring to fig. 2, a model structure of the abnormal region detection model to be trained, which is described in detail with reference to fig. 3, the constructing of the abnormal region detection model to be trained of agricultural products includes:
s21, acquiring a trained 16-layer VGG network;
s22, replacing the full connection layers in the 16-layer VGG network with a first preset number of convolutional layers to obtain an improved VGG network;
and S23, adding a second preset number of convolution layers at the tail part of the improved VGG network to obtain the abnormal region detection model to be trained.
It should be noted that the VGG is a neural network model proposed by Visual Geometry Group of Oxford university, and the VGG mainly has two structures, namely, a 16-layer VGG network and a 19-layer VGG network, and the numbers 16 and 19 indicate the difference of the network depths.
It should be further explained that, the 16-layer VGG network includes a plurality of convolution kernels of 3x3 and fully connected layers, and compared with the original neural network using a plurality of convolution kernels of 7x7 or convolution kernels of 5 x 5, the convolution kernels of 3x3 can ensure that the original neural network has a more detailed sensing field and has a certain effect of detecting abnormal regions of agricultural products.
In detail, the first preset number includes 2 or 4, and the second preset number includes 4 or 6.
After the abnormal region detection model to be trained is constructed, the abnormal region detection model to be trained needs to be trained further, and in detail, the abnormal region detection model to be trained is trained by using the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model, including:
constructing a detection frame generation algorithm, and generating a corresponding region detection regression formula according to the detection frame generation algorithm;
receiving training learning rate and batch size input by a user for training the abnormal region detection model to be trained;
taking the normal agricultural product training set as a positive sample, taking the abnormal agricultural product training set as a negative sample, and inputting the negative sample into the abnormal area detection model to be trained;
according to the training learning rate and the batch size, performing first feature extraction on the positive sample and the negative sample by using the improved VGG network to obtain a first positive sample feature set and a first negative sample feature set;
performing second feature extraction on the first positive sample feature set and the first negative sample feature set by using a second preset number of the convolutional layers to obtain a second positive sample feature set and a second negative sample feature set;
performing region detection frame prediction on the second positive sample feature set and the second negative sample feature set by using the region detection regression formula to obtain a region detection frame set;
adjusting the training learning rate, the batch size and the internal parameters of the abnormal region detection model to be trained by using the region detection frame set, and returning to the first feature extraction step;
and determining the abnormal region detection model to be trained as the abnormal region detection model until the training learning rate, the batch size and the adjustment times of the internal parameters of the abnormal region detection model to be trained are adjusted to be larger than a specified adjustment threshold.
Wherein the constructing a detection box generation algorithm comprises:
the following detection box generation algorithm is adopted:
wherein the content of the first and second substances,represents the maximum and minimum proportion of the preset detection frame to each original training image in the original training images,representing the number of detection boxes in each original training image,the first to represent each original training diagramAnd predicting the position of each detection frame.
Further, the generating a corresponding region detection regression formula according to the detection box generation algorithm includes:
generating the region detection regression formula by adopting the following method:
wherein the content of the first and second substances,representing the region detection regression formula,is the number of positive samples included within the generated detection box,to indicate a function, take the value (0,1),to detect the predicted values of positive or negative samples within the box,in order to detect the position prediction value of the frame,position parameters of real frames of agricultural products in the inner circles of the positive sample and the negative sample,for adjusting the parameters, the parameters can be manually set in advance,represents a regression function of the position of the detection frame,is a check box class regression function.
In addition, the training learning rate indicates an internal parameter of the abnormal region detection model to be trained, the training learning rate, and an adjustment range of the batch size each time the abnormal region detection model to be trained is adjusted, and the batch size indicates the number of pictures of the positive samples or the negative samples input at each training.
It should be emphasized that, first, performing the first feature extraction by using the improved VGG network and performing the second feature extraction by using the convolutional layer are common technical means, and are not described herein again. Secondly, the settable interval of the specified adjustment threshold is [5000,20000], and the abnormal region detection model to be trained can be determined as the abnormal region detection model after the adjustment times are larger than the specified adjustment threshold
And S3, receiving the picture of the agricultural product to be detected, and executing picture conversion on the picture of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain the converted picture to be detected.
In detail, the receiving the picture of the agricultural product to be detected comprises:
receiving the geographical position of an agricultural product to be detected;
executing shooting at the geographic position by means of shooting equipment to generate a panoramic photo of the agricultural product to be detected;
and receiving the picture interception of the panoramic picture by the user to obtain the picture of the agricultural product to be detected.
Illustratively, if a grape orchard needs to be detected again, inputting the geographic position of the grape orchard to the shooting equipment, and shooting to obtain a panoramic view of the grape orchard, wherein the shooting equipment comprises a speed dome device, an unmanned aerial vehicle device and the like. Further, after a panoramic image of the grape orchard is obtained, according to the area in which the user is interested, the grape orchard is intercepted, and the picture of the agricultural product to be detected is obtained.
It should be explained that, because the abnormal region detection model has requirements on the input format of the picture, such as picture resolution, size, etc., the picture conversion is performed on the picture of the agricultural product to be detected according to the input requirements, so as to obtain the converted picture to be detected.
And S4, inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection, and obtaining an abnormal area detection result.
In detail, after the converted picture to be detected is input to the abnormal region detection model, the improved VGG network is sequentially used to perform first feature extraction on the converted picture to be detected and perform second feature extraction on the second preset number of convolutional layers until the region detection regression formula is used to perform region detection frame prediction to obtain a region detection frame, wherein the region detection frame includes whether the region is an abnormal region or a normal region and the position of each region.
Illustratively, the converted image to be detected corresponding to the grape orchard is input into the abnormal region detection model, and the abnormal region detection model can detect which regions in the converted image to be detected are normal grape regions and which regions are virus grape regions, so that abnormal region detection is completed.
It can be seen that, in the embodiment of the present invention, an original training atlas of an agricultural product is received, according to whether each original training atlas in the original training atlas includes an abnormal region, the original training atlas is divided to obtain a normal agricultural product training set and an abnormal agricultural product training set, and abnormal region detection of the agricultural product can be performed only after an abnormal region detection model to be trained needs to be trained, so that training of the abnormal region detection model to be trained can be realized through the normal agricultural product training set and the abnormal agricultural product training set obtained by dividing the abnormal region, thereby providing necessary conditions for subsequent abnormal region detection, in addition, according to the input requirement of the abnormal region detection model, image transformation is performed on the image of the agricultural product to be detected to obtain a transformed image, and further, the abnormal region detection model is utilized to complete final abnormal region detection, compared with the background technology, the whole detection process is an automatic detection process without human intervention. Therefore, the abnormal region detection method based on the agricultural products, provided by the embodiment of the invention, can solve the problems of low detection efficiency and low detection intelligence degree of the abnormal region detection based on the agricultural products.
Fig. 4 is a functional block diagram of the abnormal region detection apparatus based on agricultural products according to the present invention.
The abnormal region detection apparatus 400 based on agricultural products according to the present invention may be installed in an electronic device. According to the realized functions, the agricultural product-based abnormal region detection device may include a training atlas classification module 401, a model training module 402, a picture transformation module 403, and an abnormal region detection module 404. A module according to the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the training image set classification module 401 is configured to receive an original training image set of agricultural products, and perform division on the original training image set according to whether each original training image in the original training image set includes an abnormal region, to obtain a normal agricultural product training set and an abnormal agricultural product training set;
the model training module 402 is configured to construct an abnormal region detection model of an agricultural product to be trained, and train the abnormal region detection model to be trained by using the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
the image conversion module 403 is configured to receive an image of an agricultural product to be detected, and perform image conversion on the image of the agricultural product to be detected according to an input requirement of the abnormal region detection model to obtain a converted image to be detected;
the abnormal region detection module 404 is configured to input the converted to-be-detected picture to the abnormal region detection model to perform abnormal region detection, so as to obtain an abnormal region detection result.
In detail, when the modules in the abnormal region detection apparatus 400 based on agricultural products according to the embodiment of the present invention are used, the same technical means as the abnormal region detection method based on agricultural products described in fig. 1 and fig. 3 are adopted, and the same technical effects can be produced, which is not described herein again.
Fig. 5 is a schematic structural diagram of an electronic device for implementing the method for detecting an abnormal area based on agricultural products according to the present invention.
The electronic device may include a processor 50, a memory 51, a communication bus 52, and a communication interface 53, and may further include a computer program, such as an agricultural product-based abnormal area detection program, stored in the memory 51 and executable on the processor 50.
In some embodiments, the processor 50 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, and includes one or more Central Processing Units (CPUs), a microprocessor, a digital Processing chip, a graphics processor, a combination of various control chips, and the like. The processor 50 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (for example, executing an abnormal area detection program based on agricultural products, etc.) stored in the memory 51 and calling data stored in the memory 51.
The memory 51 includes at least one type of readable storage medium including flash memory, removable hard disks, multimedia cards, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disks, optical disks, etc. The memory 51 may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The memory 51 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the electronic device. Further, the memory 51 may also include both an internal storage unit and an external storage device of the electronic device. The memory 51 may be used not only to store application software installed in the electronic device and various types of data, such as codes of an abnormal region detection program based on agricultural products, but also to temporarily store data that has been output or is to be output.
The communication bus 52 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 51 and at least one processor 50 or the like.
The communication interface 53 is used for communication between the electronic device and other devices, and includes a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which are typically used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a Display (Display), an input unit such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
Fig. 5 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 5 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 50 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
It is to be understood that the embodiments described are for illustrative purposes only and that the scope of the claimed invention is not limited to this configuration.
The agricultural product-based abnormal area detection program stored in the memory 51 of the electronic device is a combination of a plurality of computer programs, and when running in the processor 50, can realize:
receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set;
constructing an abnormal region detection model of an agricultural product to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
receiving an image of an agricultural product to be detected, and performing image conversion on the image of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain a converted image to be detected;
and inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection to obtain an abnormal area detection result.
Specifically, the processor 50 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the computer program, which is not described herein again.
Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a non-volatile computer-readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set;
constructing an abnormal region detection model of an agricultural product to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
receiving an image of an agricultural product to be detected, and performing image conversion on the image of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain a converted image to be detected;
and inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection to obtain an abnormal area detection result.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. An abnormal area detection method based on agricultural products, which is characterized by comprising the following steps:
receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set;
constructing an abnormal region detection model of an agricultural product to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
receiving an image of an agricultural product to be detected, and performing image conversion on the image of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain a converted image to be detected;
and inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection to obtain an abnormal area detection result.
2. The agricultural product-based abnormal region detection method of claim 1, wherein the constructing of the abnormal region detection model to be trained of the agricultural product comprises:
acquiring a trained 16-layer VGG network;
replacing the fully-connected layers in the 16-layer VGG network with a first preset number of convolutional layers to obtain an improved VGG network;
and adding a second preset number of convolutional layers at the tail part of the improved VGG network to obtain the abnormal region detection model to be trained.
3. The agricultural product-based abnormal region detection method according to claim 2, wherein training the abnormal region detection model to be trained using the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model comprises:
constructing a detection frame generation algorithm, and generating a corresponding region detection regression formula according to the detection frame generation algorithm;
receiving training learning rate and batch size input by a user for training the abnormal region detection model to be trained;
taking the normal agricultural product training set as a positive sample, taking the abnormal agricultural product training set as a negative sample, and inputting the negative sample into the abnormal area detection model to be trained;
according to the training learning rate and the batch size, performing first feature extraction on the positive sample and the negative sample by using the improved VGG network to obtain a first positive sample feature set and a first negative sample feature set;
performing second feature extraction on the first positive sample feature set and the first negative sample feature set by using a second preset number of the convolutional layers to obtain a second positive sample feature set and a second negative sample feature set;
performing region detection frame prediction on the second positive sample feature set and the second negative sample feature set by using the region detection regression formula to obtain a region detection frame set;
adjusting the training learning rate, the batch size and the internal parameters of the abnormal region detection model to be trained by using the region detection frame set, and returning to the first feature extraction step;
and determining the abnormal region detection model to be trained as the abnormal region detection model until the training learning rate, the batch size and the adjustment times of the internal parameters of the abnormal region detection model to be trained are adjusted to be larger than a specified adjustment threshold.
4. The agricultural product-based abnormal region detection method of claim 3, wherein the constructing a detection box generation algorithm comprises:
the following detection box generation algorithm is adopted:
wherein the content of the first and second substances,represents the maximum and minimum proportion of the preset detection frame to each original training image in the original training images,representing the number of detection boxes in each original training image,the first to represent each original training diagramAnd predicting the position of each detection frame.
5. The agricultural product-based abnormal region detection method according to claim 4, wherein the generating of the corresponding region detection regression formula according to the detection box generation algorithm comprises:
generating the region detection regression formula by adopting the following method:
wherein the content of the first and second substances,representing the region detection regression formula,is the number of positive samples included within the generated detection box,to indicate a function, take the value (0,1),to detect the predicted values of positive or negative samples within the box,in order to detect the position prediction value of the frame,position parameters of real frames of agricultural products in the inner circles of the positive sample and the negative sample,in order to adjust the parameters of the device,represents a regression function of the position of the detection frame,representing a check box class regression function.
6. The agricultural product-based abnormal area detection method of claim 1, wherein the receiving of the picture of the agricultural product to be detected comprises:
receiving the geographical position of an agricultural product to be detected;
executing shooting at the geographic position by means of shooting equipment to generate a panoramic photo of the agricultural product to be detected;
and receiving the picture interception of the panoramic picture by the user to obtain the picture of the agricultural product to be detected.
7. The method for detecting abnormal regions based on agricultural products according to claim 1, wherein the dividing the original training image set according to whether each original training image in the original training image set includes abnormal regions to obtain a normal agricultural product training set and an abnormal agricultural product training set comprises:
transmitting each of the original training images to a researcher studying the agricultural product;
receiving a returned result of whether the researcher includes diseases or not for each original training image;
and dividing the original training atlas according to the returned result to obtain the normal agricultural product training set and the abnormal agricultural product training set.
8. An abnormal area detection device based on agricultural products, the device comprising:
the training image set classification module is used for receiving an original training image set of agricultural products, and dividing the original training image set according to whether each original training image in the original training image set comprises an abnormal area or not to obtain a normal agricultural product training set and an abnormal agricultural product training set;
the model training module is used for constructing an abnormal region detection model of agricultural products to be trained, and training the abnormal region detection model to be trained by utilizing the normal agricultural product training set and the abnormal agricultural product training set to obtain an abnormal region detection model;
the image conversion module is used for receiving an image of the agricultural product to be detected, and executing image conversion on the image of the agricultural product to be detected according to the input requirement of the abnormal region detection model to obtain a converted image to be detected;
and the abnormal area detection module is used for inputting the converted picture to be detected into the abnormal area detection model to execute abnormal area detection so as to obtain an abnormal area detection result.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the agricultural product-based abnormal area detection method of any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the agricultural product-based abnormal area detection method according to any one of claims 1 to 7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111193335.6A CN113837316B (en) | 2021-10-13 | 2021-10-13 | Abnormal region detection method, device, equipment and medium based on agricultural products |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111193335.6A CN113837316B (en) | 2021-10-13 | 2021-10-13 | Abnormal region detection method, device, equipment and medium based on agricultural products |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113837316A true CN113837316A (en) | 2021-12-24 |
CN113837316B CN113837316B (en) | 2023-12-26 |
Family
ID=78968857
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111193335.6A Active CN113837316B (en) | 2021-10-13 | 2021-10-13 | Abnormal region detection method, device, equipment and medium based on agricultural products |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113837316B (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115407800A (en) * | 2022-09-13 | 2022-11-29 | 成都信息工程大学 | Unmanned aerial vehicle inspection system and inspection method in agricultural product storage and preservation warehouse |
CN116401417A (en) * | 2023-06-07 | 2023-07-07 | 深圳市中农网有限公司 | Hierarchical storage method based on massive agricultural product data |
CN116649160A (en) * | 2023-08-01 | 2023-08-29 | 南京康之春生物科技有限公司 | Edible fungus strain production monitoring system and monitoring method |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109558902A (en) * | 2018-11-20 | 2019-04-02 | 成都通甲优博科技有限责任公司 | A kind of fast target detection method |
CN110717903A (en) * | 2019-09-30 | 2020-01-21 | 天津大学 | Method for detecting crop diseases by using computer vision technology |
CN111160485A (en) * | 2019-12-31 | 2020-05-15 | 中国民用航空总局第二研究所 | Regression training-based abnormal behavior detection method and device and electronic equipment |
AU2020102405A4 (en) * | 2020-09-24 | 2020-11-05 | Suzhou Polytechnic Institute Of Agriculture | A non-destructive testing method for agricultural product quality indicators |
CN112581459A (en) * | 2020-12-23 | 2021-03-30 | 安徽高哲信息技术有限公司 | Crop classification system and method |
US20210133518A1 (en) * | 2017-04-07 | 2021-05-06 | Intel Corporation | Joint training of neural networks using multi-scale hard example mining |
CN112861906A (en) * | 2020-12-31 | 2021-05-28 | 深圳前海微众银行股份有限公司 | Method and device for detecting picture abnormity |
CN113052136A (en) * | 2021-04-22 | 2021-06-29 | 南京工业大学 | Pedestrian detection method based on improved Faster RCNN |
WO2021151277A1 (en) * | 2020-05-26 | 2021-08-05 | 平安科技(深圳)有限公司 | Method and apparatus for determining severity of damage on target object, electronic device, and storage medium |
CN113255605A (en) * | 2021-06-29 | 2021-08-13 | 深圳市城市交通规划设计研究中心股份有限公司 | Pavement disease detection method and device, terminal equipment and storage medium |
-
2021
- 2021-10-13 CN CN202111193335.6A patent/CN113837316B/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210133518A1 (en) * | 2017-04-07 | 2021-05-06 | Intel Corporation | Joint training of neural networks using multi-scale hard example mining |
CN109558902A (en) * | 2018-11-20 | 2019-04-02 | 成都通甲优博科技有限责任公司 | A kind of fast target detection method |
CN110717903A (en) * | 2019-09-30 | 2020-01-21 | 天津大学 | Method for detecting crop diseases by using computer vision technology |
CN111160485A (en) * | 2019-12-31 | 2020-05-15 | 中国民用航空总局第二研究所 | Regression training-based abnormal behavior detection method and device and electronic equipment |
WO2021151277A1 (en) * | 2020-05-26 | 2021-08-05 | 平安科技(深圳)有限公司 | Method and apparatus for determining severity of damage on target object, electronic device, and storage medium |
AU2020102405A4 (en) * | 2020-09-24 | 2020-11-05 | Suzhou Polytechnic Institute Of Agriculture | A non-destructive testing method for agricultural product quality indicators |
CN112581459A (en) * | 2020-12-23 | 2021-03-30 | 安徽高哲信息技术有限公司 | Crop classification system and method |
CN112861906A (en) * | 2020-12-31 | 2021-05-28 | 深圳前海微众银行股份有限公司 | Method and device for detecting picture abnormity |
CN113052136A (en) * | 2021-04-22 | 2021-06-29 | 南京工业大学 | Pedestrian detection method based on improved Faster RCNN |
CN113255605A (en) * | 2021-06-29 | 2021-08-13 | 深圳市城市交通规划设计研究中心股份有限公司 | Pavement disease detection method and device, terminal equipment and storage medium |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115407800A (en) * | 2022-09-13 | 2022-11-29 | 成都信息工程大学 | Unmanned aerial vehicle inspection system and inspection method in agricultural product storage and preservation warehouse |
CN115407800B (en) * | 2022-09-13 | 2023-09-01 | 成都信息工程大学 | Unmanned aerial vehicle inspection method in agricultural product storage fresh-keeping warehouse |
CN116401417A (en) * | 2023-06-07 | 2023-07-07 | 深圳市中农网有限公司 | Hierarchical storage method based on massive agricultural product data |
CN116401417B (en) * | 2023-06-07 | 2023-09-05 | 深圳市中农网有限公司 | Hierarchical storage method based on massive agricultural product data |
CN116649160A (en) * | 2023-08-01 | 2023-08-29 | 南京康之春生物科技有限公司 | Edible fungus strain production monitoring system and monitoring method |
CN116649160B (en) * | 2023-08-01 | 2023-10-10 | 南京康之春生物科技有限公司 | Edible fungus strain production monitoring system and monitoring method |
Also Published As
Publication number | Publication date |
---|---|
CN113837316B (en) | 2023-12-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113837316B (en) | Abnormal region detection method, device, equipment and medium based on agricultural products | |
WO2021087985A1 (en) | Model training method and apparatus, storage medium, and electronic device | |
CN109948497B (en) | Object detection method and device and electronic equipment | |
CN110378297B (en) | Remote sensing image target detection method and device based on deep learning and storage medium | |
CN113159147B (en) | Image recognition method and device based on neural network and electronic equipment | |
Okayasu et al. | Vision-based classification of mosquito species: Comparison of conventional and deep learning methods | |
Li et al. | Improved YOLO v5 wheat ear detection algorithm based on attention mechanism | |
CN105809146A (en) | Image scene recognition method and device | |
CN110222718B (en) | Image processing method and device | |
CN114758249B (en) | Target object monitoring method, device, equipment and medium based on field night environment | |
CN111860652B (en) | Method, device, equipment and medium for measuring animal body weight based on image detection | |
CN110296660B (en) | Method and device for detecting livestock body ruler | |
CN109344738A (en) | The recognition methods of crop diseases and pest crop smothering and device | |
Song et al. | Tsr-yolo: A chinese traffic sign recognition algorithm for intelligent vehicles in complex scenes | |
Sabzi et al. | Designing a fruit identification algorithm in orchard conditions to develop robots using video processing and majority voting based on hybrid artificial neural network | |
Wang et al. | Lightweight SM-YOLOv5 tomato fruit detection algorithm for plant factory | |
Choi et al. | CNN-based illumination estimation with semantic information | |
CN115100732A (en) | Fishing detection method and device, computer equipment and storage medium | |
Kwon et al. | Visible and near-infrared image acquisition and fusion for night surveillance | |
Genaev et al. | Classification of fruit flies by gender in images using smartphones and the YOLOv4-tiny neural network | |
CN115082968A (en) | Behavior identification method based on infrared light and visible light fusion and terminal equipment | |
CN114842240A (en) | Method for classifying images of leaves of MobileNet V2 crops by fusing ghost module and attention mechanism | |
Yuan et al. | Sensitivity examination of YOLOv4 regarding test image distortion and training dataset attribute for apple flower bud classification | |
Xie et al. | Occluded pedestrian detection techniques by deformable attention-guided network (DAGN) | |
Nazir et al. | EfficientPNet—An Optimized and Efficient Deep Learning Approach for Classifying Disease of Potato Plant Leaves |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |