CN117953349A

CN117953349A - Method, device, equipment and storage medium for detecting plant diseases and insect pests of traditional Chinese medicinal materials

Info

Publication number: CN117953349A
Application number: CN202410330593.1A
Authority: CN
Inventors: 余应淮; 王焌毅; 彭小红; 汤文浩; 黄湘粤; 徐明瑜; 吴承恩; 叶旭晖; 温慧娴; 颜俊熹; 肖泰安; 欧贻燊
Original assignee: Guangdong Ocean University
Current assignee: Guangdong Ocean University
Priority date: 2024-03-22
Filing date: 2024-03-22
Publication date: 2024-04-30
Anticipated expiration: 2044-03-22
Also published as: CN117953349B

Abstract

The invention discloses a disease and pest detection method, device, equipment and storage medium for traditional Chinese medicine, wherein a multi-scale feature fusion neural network model is subjected to model training through multi-angle traditional Chinese medicine disease and pest pretreatment images, so that global features and local features of the traditional Chinese medicine disease and pest pretreatment images are extracted by the multi-scale feature fusion neural network model, the global features and the local features of the images are obtained and spliced, weight values of the spliced features are calculated, the global features and the local features of the images are subjected to feature fusion based on the weight values, the multi-scale feature fusion neural network model is subjected to model parameter optimization based on the obtained image fusion features, a traditional Chinese medicine disease and pest detection model is generated based on optimal model parameters, and the received traditional Chinese medicine images to be detected are input into the traditional Chinese medicine disease and pest detection model to output traditional Chinese medicine disease and pest results.

Description

Method, device, equipment and storage medium for detecting plant diseases and insect pests of traditional Chinese medicinal materials

Technical Field

The invention relates to the technical field of intelligent agriculture, in particular to a method, a device, equipment and a storage medium for detecting plant diseases and insect pests of traditional Chinese medicinal materials.

Background

The traditional Chinese medicinal materials have important roles in traditional medicine and modern medicine, but the quality and the safety of the traditional Chinese medicinal materials are influenced by plant diseases and insect pests, the traditional Chinese medicinal material plant diseases and insect pests detection method is mainly based on manual observation and experience judgment, the problems of strong subjectivity, low efficiency, low accuracy and the like exist, and along with the expansion of the industrial scale of the traditional Chinese medicinal materials, a more efficient and accurate plant diseases and insect pests detection method is needed to ensure the quality of the traditional Chinese medicinal materials.

The existing automatic disease and pest detection technology is mainly applied to the agricultural field, but has certain limitation on the special field of traditional Chinese medicinal materials, and mainly is difficult to process complex textures and forms on the surface of the traditional Chinese medicinal materials when the existing disease and pest detection technology is applied to the disease and pest detection of the traditional Chinese medicinal materials, so that the detection accuracy is limited.

Therefore, in order to solve the above problems, it is necessary to develop an intelligent detection method for plant diseases and insect pests of Chinese medicinal materials.

Disclosure of Invention

The invention aims to solve the technical problems that: the method, the device, the equipment and the storage medium for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials can improve the accuracy of detecting the plant diseases and insect pests of the traditional Chinese medicinal materials.

In order to solve the technical problems, the invention provides a method for detecting plant diseases and insect pests of Chinese medicinal materials, which comprises the following steps:

collecting multi-angle Chinese medicinal plant disease and insect pest images, and performing image preprocessing on the Chinese medicinal plant disease and insect pest images at each angle to obtain multi-angle Chinese medicinal plant disease and insect pest preprocessed images;

Performing model training on a multi-scale feature fusion neural network model based on the traditional Chinese medicine plant disease and insect pest pretreatment image, so that the multi-scale feature fusion neural network model performs global feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain image global features, performs local feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain image local features, performs splicing processing on the image global features and the image local features, calculates weight values of the splicing features, performs feature fusion processing on the image global features and the image local features based on the weight values to obtain image fusion features, performs model parameter optimization processing on the multi-scale feature fusion neural network model based on the image fusion features, and determines optimal model parameters;

Based on the optimal model parameters, a traditional Chinese medicine plant disease and insect pest detection model is generated, and the received traditional Chinese medicine plant image to be detected is input into the traditional Chinese medicine plant disease and insect pest detection model, so that the traditional Chinese medicine plant disease and insect pest detection model outputs traditional Chinese medicine plant disease and insect pest results.

In one possible implementation manner, the collecting the multi-angle plant diseases and insect pests image of the traditional Chinese medicine specifically includes:

setting a traditional Chinese medicine plant disease and insect pest image acquisition device and a transparent conveyor belt, wherein the traditional Chinese medicine plant disease and insect pest image acquisition device comprises an automatic rotation platform and image acquisition equipment, and the image acquisition equipment is arranged in the automatic rotation platform;

controlling the image acquisition equipment to rotate the transparent conveyor belt according to a preset rotation angle interval based on the automatic rotation platform;

in the rotation process, based on the image acquisition equipment, image acquisition processing is carried out on the traditional Chinese medicine crops placed on the transparent conveyor belt, and multi-angle traditional Chinese medicine plant diseases and insect pests images are obtained.

In one possible implementation manner, the image preprocessing is performed on the plant disease and insect pest image of the traditional Chinese medicine at each angle to obtain a multi-angle plant disease and insect pest preprocessed image of the traditional Chinese medicine, which specifically includes:

Respectively carrying out image filtering treatment on the Chinese herbal medicine plant disease and insect pest images at each angle to obtain Chinese herbal medicine filtering images, and carrying out histogram equalization treatment on the Chinese herbal medicine filtering images to obtain Chinese herbal medicine equalization images;

performing color space conversion on the traditional Chinese medicine equalization image to obtain a target color space image, and performing saliency detection on the target color space image to obtain a first salient region image corresponding to the target color space image;

Normalizing the first significant region image to obtain a normalized significant region image, and multiplying the normalized significant region image and the traditional Chinese medicinal material equalization image to obtain a second significant region image;

Threshold segmentation is performed on the second salient region image so that the second salient region image is divided into foreground regions, and an initialization mask is set based on the foreground regions;

Determining an interested region of the second significant region image, inputting the interested region, the initialization mask and the traditional Chinese medicine equalization image into a preset matting algorithm, so that the matting algorithm carries out iterative optimization processing on a segmentation result of the traditional Chinese medicine equalization image to obtain an image segmentation result;

And carrying out image segmentation processing on the traditional Chinese medicine equalization image based on the image segmentation result to obtain a traditional Chinese medicine plant disease and insect pest pretreatment image.

In one possible implementation manner, the global feature extraction is performed on the plant disease and pest pretreatment image of the traditional Chinese medicinal materials to obtain an image global feature, which specifically includes:

the multi-scale feature fusion neural network model comprises a deep convolution network;

Mapping the traditional Chinese medicine plant disease and insect pest pretreatment image into a depth feature space based on the depth convolution network to obtain a plurality of first feature images;

Performing convolution processing on the plurality of first features based on a preset convolution check to obtain a plurality of second feature images, and performing combination processing on the plurality of second feature images according to channels to obtain a feature result image;

And carrying out point-by-point convolution processing on the characteristic result graph to obtain the image global characteristic.

In one possible implementation, the deep convolutional network performs a global feature extraction formula as follows:

；

in the method, in the process of the invention, As a global feature of the image,For depth space separable convolution operations, this generally involves applying to the input imageIs used for the convolution processing of (1),Operations are performed for a point-by-point convolution,In order to be a characteristic result graph,For the depth space separable convolutional network parameters,The network parameters are convolved for vgg's 16,For the ratio of depth space separable convolutional network parameters to VGG16 convolutional network parameters,For the number of parameters of the depth-space separable convolutional network,For the convolution kernel size,For the number of convolution kernels,Is thatNumber of convolution kernels.

In one possible implementation manner, the extracting the local feature of the preprocessed image of the plant diseases and insect pests of the traditional Chinese medicine to obtain the local feature of the image specifically includes:

the multi-scale feature fusion neural network model comprises a shift window converter network, wherein the shift window converter network comprises a multi-head self-attention mechanism, a multi-level perception mechanism and an offset window multi-head self-attention mechanism;

Respectively carrying out linear mapping processing on the plurality of first feature images to obtain corresponding linear first feature images, respectively carrying out weighting processing on the linear first feature images based on the multi-head self-attention mechanism to obtain first weighted feature images, and carrying out addition processing on the first weighted feature images and the corresponding first feature images to obtain residual connection feature images;

Performing linear mapping processing on the residual error connection characteristic map to obtain a linear residual error connection characteristic map, and performing nonlinear transformation on the linear residual error connection characteristic map based on the multi-level perception mechanism to obtain a nonlinear residual error connection characteristic map;

adding the nonlinear residual connection characteristic diagram and the residual connection characteristic diagram to obtain an offset rate window input characteristic diagram;

Performing linear mapping processing on the offset window input feature map to obtain a corresponding linear offset window input feature map, respectively performing weighting processing on the linear offset window input feature map based on the offset window multi-head self-attention mechanism to obtain a weighted linear offset window input feature map, and performing addition processing on the weighted linear offset window input feature map and the corresponding offset window input feature map to obtain an offset window residual error connection feature map;

Performing linear mapping processing on the offset window residual error connection characteristic map to obtain a linear offset window residual error connection characteristic map, and performing nonlinear transformation on the linear offset window residual error connection characteristic map based on the multi-level sensing mechanism to obtain a nonlinear offset window residual error connection characteristic map;

And adding the nonlinear offset rate window residual error connection characteristic diagram and the offset rate window residual error connection characteristic diagram to obtain an image local characteristic.

In one possible implementation manner, the stitching processing is performed on the global image feature and the local image feature, and the calculating a weight value of the stitching feature specifically includes:

Performing dimension conversion on the image global feature based on a full connection layer to obtain a target dimension global feature, performing dimension conversion on the image local feature based on the full connection layer to obtain a target dimension local feature, and performing splicing processing on the target dimension global feature and the target dimension local feature to obtain a spliced feature;

Performing weight value calculation on the splicing features based on a preset weight value calculation formula to obtain weight values of the splicing features; the preset weight value calculation formula is as follows:

；

in the method, in the process of the invention, Representing the dimension conversion operation of the global connectivity layer,Representing the sigmoid function,As a global feature of the image,As a local feature of the image,Is a weight value.

The invention also provides a plant disease and insect pest detection device for the traditional Chinese medicinal materials, which comprises: the system comprises an image acquisition module, a multi-scale feature fusion neural network model training module and a traditional Chinese medicine grass pest detection module;

The image acquisition module is used for acquiring multi-angle Chinese herbal medicine disease and insect pest images, and carrying out image preprocessing on the Chinese herbal medicine disease and insect pest images at each angle to obtain multi-angle Chinese herbal medicine disease and insect pest preprocessed images;

The multi-scale feature fusion neural network model training module is used for carrying out model training on a multi-scale feature fusion neural network model based on the traditional Chinese medicine plant disease and insect pest pretreatment image so that the multi-scale feature fusion neural network model carries out global feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain an image global feature, carries out local feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain an image local feature, carries out splicing treatment on the image global feature and the image local feature, calculates a weight value of the splicing feature, carries out feature fusion treatment on the image global feature and the image local feature based on the weight value to obtain an image fusion feature, carries out model parameter optimization treatment on the multi-scale feature fusion neural network model based on the image fusion feature, and determines optimal model parameters;

The traditional Chinese medicine plant disease and insect pest detection module is used for generating a traditional Chinese medicine plant disease and insect pest detection model based on the optimal model parameters, inputting the received traditional Chinese medicine image to be detected into the traditional Chinese medicine plant disease and insect pest detection model, and enabling the traditional Chinese medicine plant disease and insect pest detection model to output traditional Chinese medicine plant disease and insect pest results.

In one possible implementation manner, the image acquisition module is configured to acquire multi-angle images of plant diseases and insect pests of traditional Chinese medicinal materials, and specifically includes:

In one possible implementation manner, the image acquisition module is configured to perform image preprocessing on the plant diseases and insect pests images of the traditional Chinese medicinal materials at each angle to obtain a multi-angle plant diseases and insect pests preprocessed image, and specifically includes:

In one possible implementation manner, the multi-scale feature fusion neural network model training module is configured to perform global feature extraction on the traditional Chinese medicine plant disease and pest pretreatment image to obtain an image global feature, and specifically includes:

；

In one possible implementation manner, the multi-scale feature fusion neural network model training module is configured to perform local feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain an image local feature, and specifically includes:

In one possible implementation manner, the multi-scale feature fusion neural network model training module is configured to perform a stitching process on the image global feature and the image local feature, and calculate a weight value of the stitching feature, where the method specifically includes:

；

The invention also provides a terminal device which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the method for detecting the plant diseases and insect pests of the traditional Chinese medicine materials is realized when the processor executes the computer program.

The invention also provides a computer readable storage medium, which comprises a stored computer program, wherein the equipment where the computer readable storage medium is located is controlled to execute the method for detecting the plant diseases and insect pests of the traditional Chinese medicine according to any one of the above when the computer program runs.

Compared with the prior art, the method, the device, the equipment and the storage medium for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials have the following beneficial effects:

Because the images at different angles can provide more comprehensive information, the preprocessing is helpful for eliminating noise and enhancing image characteristics, and therefore, the detection accuracy of the plant diseases and insect pests of the traditional Chinese medicine materials in the subsequent model training process can be improved by collecting and preprocessing the multi-angle plant disease and insect pest images of the traditional Chinese medicine materials; and when the model training is carried out on the multi-scale feature fusion neural network model based on the traditional Chinese medicine plant disease and insect pest pretreatment image, the global features and the local features of the traditional Chinese medicine plant disease and insect pest pretreatment image are extracted, the extracted image global features and the extracted image local features are fused, the whole information and the local details of the image can be better captured, the parameter optimization is carried out on the multi-scale feature fusion neural network model subsequently, the optimal model parameters can be obtained, the model performance and the generalization capability are improved, so that the detection accuracy and the detection robustness are improved, finally, the traditional Chinese medicine plant disease and insect pest detection model is generated based on the optimal model parameters, the input traditional Chinese medicine plant disease and insect pest detection can be accurately carried out on the traditional Chinese medicine plant disease and insect pest detection accuracy is improved.

Drawings

FIG. 1 is a schematic flow chart of an embodiment of a method for detecting plant diseases and insect pests of a Chinese medicinal material;

Fig. 2 is a schematic structural view of an embodiment of a plant disease and insect pest detection device for a traditional Chinese medicine provided by the invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Embodiment 1, referring to fig. 1, fig. 1 is a schematic flow chart of an embodiment of a method for detecting plant diseases and insect pests of a traditional Chinese medicine, as shown in fig. 1, the method includes steps 101 to 103, specifically as follows:

Step 101: collecting multi-angle Chinese medicinal plant disease and insect pest images, and carrying out image preprocessing on the Chinese medicinal plant disease and insect pest images at each angle to obtain multi-angle Chinese medicinal plant disease and insect pest preprocessed images.

In one embodiment, since the pest and disease damage is not limited to a specific area and may occur at each part of the Chinese medicinal material, multiple angles of Chinese medicinal material pest and disease damage images are collected, and appearance information of the Chinese medicinal material crops to be detected under different angles can be obtained through the multiple angles of Chinese medicinal material pest and disease damage images, which is helpful to cover each part of the surface of the Chinese medicinal material, including hidden areas which may be affected by the pest and disease damage; and the visual angle diversity can provide more comprehensive information, is favorable for more accurately detecting potential diseases and insect pests, improves the subsequent detection robustness to the diseases and insect pests, and can acquire effective information from other angles even if certain areas are shielded or illumination conditions change.

In one embodiment, for collecting multi-angle Chinese herbal medicine plant diseases and insect pests images, a Chinese herbal medicine plant disease and insect pests image collecting device and a transparent conveyor belt are arranged, wherein the Chinese herbal medicine plant disease and insect pests image collecting device comprises an automatic rotating platform and image collecting equipment, and the image collecting equipment is arranged in the automatic rotating platform; controlling the image acquisition equipment to rotate the transparent conveyor belt according to a preset rotation angle interval based on the automatic rotation platform; and in the rotating process, the image acquisition equipment is used for carrying out image acquisition processing on the traditional Chinese medicine crops placed on the transparent conveyor belt to obtain multi-angle traditional Chinese medicine plant diseases and insect pests images.

Specifically, by constructing a stable automatic rotating platform, for example, a turntable controlled by a motor, the automatic rotating platform is ensured to stably rotate in the horizontal direction; selecting a suitable high resolution image acquisition device, such as a high pixel count camera or a professional scanner; the method comprises the steps of installing image acquisition equipment on an automatic rotating platform so as to capture images of traditional Chinese medicinal materials in an omnibearing manner, selecting materials with high transparency and good light transmittance as a conveyor belt, ensuring that the conveyor belt is smooth and has no obvious lines or flaws, placing the traditional Chinese medicinal materials on the transparent conveyor belt, rotating the conveyor belt according to preset rotating angle intervals by controlling the automatic rotating platform, and simultaneously carrying out image acquisition processing on the traditional Chinese medicinal materials placed on the transparent conveyor belt through the image acquisition equipment in a good illumination environment to obtain multi-angle traditional Chinese medicinal material disease and pest images, wherein the preset rotating angle intervals can be set according to user requirements, for example, the preset rotating angle intervals are set to be 10 degrees; thus, the image of the plant diseases and insect pests of the traditional Chinese medicinal materials at a plurality of angles can be ensured to be acquired.

In one embodiment, noise and impurities exist in the image due to the influence of factors such as illumination and the like possibly occurring in the image acquisition process; therefore, the image preprocessing is also required to be carried out on the traditional Chinese medicine plant diseases and insect pests images at each angle, so that the image quality is improved, and the subsequent analysis and processing are more accurate and reliable.

In one embodiment, the image filtering processing is performed on the plant diseases and insect pests images of the traditional Chinese medicinal materials at each angle to obtain a filtered image of the traditional Chinese medicinal materials, and the histogram equalization processing is performed on the filtered image of the traditional Chinese medicinal materials to obtain an equalized image of the traditional Chinese medicinal materials.

Specifically, performing image filtering processing on the traditional Chinese medicine plant disease and insect pest images at each angle by adopting a median filtering mode to obtain traditional Chinese medicine filtering images, wherein the median filtering is a nonlinear filtering method which replaces the gray value of a central pixel based on the median of the gray values of pixels in a neighborhood; processing may be performed using a common image processing library such as the median filtering functions provided in OpenCV or MATLAB; selecting a suitable filter size according to specific requirements, it is generally recommended to select a square filter template of a suitable size; the method has good denoising effect on salt and pepper noise, speckle noise and the like, and can effectively smooth images and remove outliers because the method is not influenced by extreme values; through median filtering, high-frequency noise in the image can be eliminated, and the quality and definition of the image are improved, so that the subsequent image analysis and processing are more accurate and reliable.

Specifically, the histogram equalization is a method for enhancing the contrast of an image, which redistributes pixel values according to the gray level of the image, enhances the contrast and details of the image, and makes the histogram of the image more uniform, thereby improving the visual effect of the image, and can also use histogram equalization functions provided in image processing libraries such as OpenCV, MATLAB for processing.

In an embodiment, the color space conversion is performed on the traditional Chinese medicine equalization image to obtain a target color space image, and the saliency detection is performed on the target color space image to obtain a first salient region image corresponding to the target color space image.

Specifically, the traditional Chinese medicine equalization image is converted from an original color space to a target color space based on a color space conversion function.

Preferably, the original color space includes, but is not limited to, an RGB color space, and the target color space includes, but is not limited to, a Lab color space, an HSV color space, and the like; color space conversion functions include, but are not limited to, functions in the OpenCV library, such as cv2.cvtcolor ().

Specifically, a GBVS algorithm is called, a target color space image is used as input of the GBVS algorithm, a saliency value of each pixel point in the target color space image is calculated based on color features, contrast features and direction features of the target color space image, and a first salient region image corresponding to the target color space image is obtained based on the saliency value.

For each pixel point, GBVS algorithm will consider its target location in color space and the color differences from surrounding pixels to determine its saliency; meanwhile, the GBVS algorithm analyzes the contrast of different areas in the image to determine which areas are more attractive and significant; and by analyzing the information of the local gradient, the edge direction and the like of the image, the algorithm can identify the region with obvious direction characteristics as a salient region.

In an embodiment, the normalization processing is performed on the first significant region image to obtain a normalized significant region image, and the multiplication processing is performed on the normalized significant region image and the traditional Chinese medicine equalization image to obtain a second significant region image.

Specifically, the first salient region image is normalized to be a floating point number ranging from 0 to 1, a normalized salient region image is obtained, and the traditional Chinese medicine equalization image is multiplied with the normalized salient region image to highlight the salient region, and a second salient region image is obtained.

In one embodiment, the second salient region image is thresholded such that the second salient region image is divided into foreground regions, and an initialization mask is set based on the foreground regions; determining an interested region of the second significant region image, inputting the interested region, the initialization mask and the traditional Chinese medicine equalization image into a preset matting algorithm, so that the matting algorithm performs iterative optimization processing on a segmentation result of the traditional Chinese medicine equalization image to obtain an image segmentation result.

Specifically, a target threshold is set, a saliency value corresponding to each pixel point in the second saliency region image is traversed based on the target threshold, a region with higher saliency is marked as a foreground region through threshold segmentation, an initial mask is created according to the saliency map after threshold segmentation, the region with higher saliency is marked as a foreground region, and other regions are marked as background regions; an initial mask is applied to the image to indicate whether the region is foreground or background, such an initial mask may provide preliminary information about the foreground and background in the image, helping the segmentation algorithm to work better.

Specifically, the region of interest refers to a region which is particularly concerned or needs to be processed by a user in image processing, and the region of interest in the second significant region image is specified by creating a rectangle, so that an algorithm can be told to perform initialization work of a foreground and a background in the region, and the accuracy and the efficiency of segmentation are improved.

Specifically, the preset matting algorithm is a GrabCut algorithm, such as a cv2.grabcut () function.

Specifically, when a cv2.GrabCut () function is called, the region of interest, the initialization mask and the traditional Chinese medicine equalization image are taken as input parameters, and a Grabcut algorithm starts a foreground and background segmentation optimization iteration process of the traditional Chinese medicine equalization image according to the provided input parameters; in each iteration, the GrabCut algorithm continuously adjusts the segmentation result according to the current foreground and background estimation and the characteristic information of the image so as to expect to obtain more accurate foreground and background segmentation; in each iteration process, the GrabCut algorithm checks the difference condition between the current segmentation result and the last segmentation result until a certain convergence condition is met; the convergence condition generally comprises the condition that the specified iteration times are reached or the variation of the segmentation result is smaller than a certain threshold value; once the GrabCut algorithm reaches the set convergence condition, i.e. the segmentation result is relatively stable and no significant change occurs any more, the GrabCut algorithm will stop iterating, at which time the GrabCut algorithm will return the final image segmentation result, i.e. the segmentation result of the foreground region and the background region.

Specifically, the image segmentation result is a mask.

In one embodiment, the image segmentation result is based on image segmentation processing of the traditional Chinese medicine equalization image to obtain a traditional Chinese medicine plant disease and insect pest pretreatment image.

Step 102: performing model training on a multi-scale feature fusion neural network model based on the traditional Chinese medicine plant disease and insect pest pretreatment image, so that the multi-scale feature fusion neural network model performs global feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain image global features, performs local feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain image local features, performs splicing processing on the image global features and the image local features, calculates weight values of the splicing features, performs feature fusion processing on the image global features and the image local features based on the weight values to obtain image fusion features, performs model parameter optimization processing on the multi-scale feature fusion neural network model based on the image fusion features, and determines optimal model parameters.

In one embodiment, the multi-scale feature fusion neural network model comprises a deep convolutional network.

In an embodiment, the multi-scale feature fusion neural network model performs global feature extraction on the traditional Chinese medicine plant disease and pest pretreatment image, and when obtaining an image global feature, maps the traditional Chinese medicine plant disease and pest pretreatment image into a depth feature space based on the depth convolution network to obtain a plurality of first feature images; performing convolution processing on the plurality of first features based on a preset convolution check to obtain a plurality of second feature images, and performing combination processing on the plurality of second feature images according to channels to obtain a feature result image; and carrying out point-by-point convolution processing on the characteristic result graph to obtain the image global characteristic.

In particular, the deep convolutional network includes a convolutional layer and a pooling layer, the combination of which is repeatedly stacked multiple times to progressively extract more abstract and advanced feature representations. The deeper the depth of the deep convolutional network, the more rich the feature representation the model can learn.

Specifically, the traditional Chinese medicine plant disease and pest pretreatment image is subjected to convolution processing based on each convolution layer in the depth convolution network, wherein each convolution processing generates a feature map, namely a first feature map, which represents feature response of a corresponding position, and a plurality of first feature maps are obtained by integrating all the first feature maps。

Specifically, the number of the used components is x, and the size isIs to check a plurality of first feature patternsAnd performing convolution operation to obtain a plurality of second feature maps. The convolution operation may further extract different features in the image, such as edges, textures, etc.; in deep learning, different channels of a feature map often contain feature information of different aspects; for a plurality of second feature graphs, the combination processing can be carried out according to channels so as to obtain richer feature representations; after the combined multiple second feature images are obtained, the features can be integrally processed in a global range through point-by-point convolution operation to obtain global feature representation of the image, namely, the multiple second feature images are subjected to convolution network operation through a convolution nerve kernels with the two-dimensional products of 1 multiplied by 1; this helps capture the overall structure and features in the image.

Specifically, the formula for global feature extraction is performed by the deep convolutional network, as follows:

；

The formula for executing global feature extraction by the deep convolutional network is to optimize a standard VGG16 convolutional network calculation formula, so that overhead can be reduced on the premise of not reducing model performance, and the conventional standard convolutional block is replaced by the deep separable convolution, namely when the number x of convolution kernels in the deep convolutional network is larger, the improved VGG-16 network parameter is equivalent to one-half of the y square of the standard VGG-16 model parameter calculation amount.

The traditional plant disease and insect pest identification model generally adopts a single feature extraction mode, and the single feature extraction mode usually only focuses on global features, so that the importance of local detail information in an image is ignored. In pest and disease damage identification, local features such as edges and textures are important for accurately identifying pest and disease damage areas, and the overall features cannot fully represent complex pest and disease damage areas in an image, so that the learned features of a model are not specific and accurate enough, and the final classification effect is affected; based on this, in this embodiment, a shift window converter network Swin converter is also set in the multi-scale feature fusion neural network model, and local features in the image are extracted based on the shift window converter network Swin converter.

In one embodiment, the multi-scale feature fusion neural network model includes a shifted window transformer network, wherein the shifted window transformer network includes a multi-headed self-attention mechanism, a multi-level perception mechanism, and an offset window multi-headed self-attention mechanism.

Specifically, because the global coding network based on the attention mechanism in the prior art has the problem that the details of the detection target are easy to lose, based on the problem, the multi-head self-attention mechanism introducing the offset window better keeps the detail information of the target by selectively focusing on the local area of interest; meanwhile, attention calculation can be performed on different positions and scales, so that the perception range is improved, more comprehensive context information is captured, and the performance and effect of an image processing task are improved.

In an embodiment, when local feature extraction is performed on the traditional Chinese medicine plant disease and pest pretreatment image to obtain an image local feature, respectively performing linear mapping processing on the plurality of first feature images to obtain corresponding linear first feature images, respectively performing weighting processing on the linear first feature images based on the multi-head self-attention mechanism to obtain first weighted feature images, and performing addition processing on the first weighted feature images and the corresponding first feature images to obtain residual connection feature images; performing linear mapping processing on the residual error connection characteristic map to obtain a linear residual error connection characteristic map, and performing nonlinear transformation on the linear residual error connection characteristic map based on the multi-level perception mechanism to obtain a nonlinear residual error connection characteristic map; adding the nonlinear residual connection characteristic diagram and the residual connection characteristic diagram to obtain an offset rate window input characteristic diagram; performing linear mapping processing on the offset window input feature map to obtain a corresponding linear offset window input feature map, respectively performing weighting processing on the linear offset window input feature map based on the offset window multi-head self-attention mechanism to obtain a weighted linear offset window input feature map, and performing addition processing on the weighted linear offset window input feature map and the corresponding offset window input feature map to obtain an offset window residual error connection feature map; performing linear mapping processing on the offset window residual error connection characteristic map to obtain a linear offset window residual error connection characteristic map, and performing nonlinear transformation on the linear offset window residual error connection characteristic map based on the multi-level sensing mechanism to obtain a nonlinear offset window residual error connection characteristic map; and adding the nonlinear offset rate window residual error connection characteristic diagram and the offset rate window residual error connection characteristic diagram to obtain an image local characteristic.

Specifically, the shift window transformer network performs a formula for local feature extraction as follows:

；

where LN is a linear mapping function, Is a multi-level sensing mechanism, which is characterized in that,The feature values entered for the multi-headed self-attention window, i.e. the first feature map,In order to be a multi-headed self-attention mechanism,A residual connection feature map completed for the residual connection,The feature map is connected for a non-linear residual,Features are input for the offset rate window, i.e. feature maps are input for the offset rate window,To shift the window multi-headed self-attention mechanism,For the representation of the characteristics after residual connection of the offset window, namely the residual connection characteristic diagram of the offset window,Is the feature sequencing order between the marked difference windows,The feature map is connected for a non-linear offset window residual,Is a local feature of the image.

Specifically, by accessing the offset window multi-head self-attention mechanism in the shift window converter network, the target texture information can be further refined based on multi-multiplying power downsampling to obtain a plurality of layers of features.

In an embodiment, performing dimension conversion on the image global feature based on a full connection layer to obtain a target dimension global feature, performing dimension conversion on the image local feature based on the full connection layer to obtain a target dimension local feature, and performing splicing processing on the target dimension global feature and the target dimension local feature to obtain a spliced feature; and calculating the weight value of the splicing characteristic based on a preset weight value calculation formula to obtain the weight value of the splicing characteristic.

Specifically, the preset weight value calculation formula is as follows:

；

In an embodiment, the weight value, the global image feature and the local image feature are substituted into a feature fusion formula to perform feature fusion processing, so as to obtain an image fusion feature, wherein the feature fusion formula is as follows:

in the method, in the process of the invention, As a global feature of the image,As a local feature of the image,As the weight value of the weight,Is an image fusion feature.

Specifically, the full connection layer FC is adopted to conduct dimension conversion and splicing on the image global features and the image local features extracted by the improved deep convolution network and the shift window converter network, and the sigmoid function is utilized to calculate feature weight distribution, so that the neural network construction is achieved in the multi-scale feature fusion neural network model by adopting fusion features.

In an embodiment, after the fusion feature is obtained, the fusion feature is further input into a global average pooling layer, so that the dimension reduction processing is performed on the fusion feature based on the global average pooling layer to obtain a feature vector with a fixed length, and the feature vector is input into a batch normalization layer, so that the batch normalization layer normalizes the feature vector to obtain a normalized feature vector, and stability and convergence rate of the model are improved.

In an embodiment, the normalized feature vector is further input into a full-connection layer, so that the full-connection layer performs matrix multiplication operation on the normalized feature vector and weights of the model to obtain a matrix feature vector, nonlinearity is introduced through an activation function to generate a nonlinear matrix feature vector, the nonlinear matrix feature vector output by the full-connection layer is normalized through a Softmax function to obtain a traditional Chinese medicine plant disease and insect pest prediction result, namely probability distribution of plant disease and insect pest categories, and detection operation of the traditional Chinese medicine plant disease and insect pests is achieved.

In one embodiment, the difference between the traditional Chinese medicine plant disease and insect pest prediction result and the actual label is measured by using a cross entropy loss function and is used as a model training target; and (3) adjusting model parameters according to the loss rate, updating the parameters of the model by adopting a p2p (point-to-point) optimization method through algorithms such as gradient descent and the like, gradually reducing a loss function, improving the performance and generalization capability of the model until the model converges, and determining optimal model parameters.

Step 103: based on the optimal model parameters, a traditional Chinese medicine plant disease and insect pest detection model is generated, and the received traditional Chinese medicine plant image to be detected is input into the traditional Chinese medicine plant disease and insect pest detection model, so that the traditional Chinese medicine plant disease and insect pest detection model outputs traditional Chinese medicine plant disease and insect pest results.

In one embodiment, after determining the optimal model parameters, the structure of the optimal model parameters and the generated traditional Chinese medicine plant disease and insect pest detection model is saved by adopting an open neural network open format ONNX, and export operation is performed on the model in a PyTorch framework through a torch.

In one embodiment, the user interface suitable for the mobile device and the web browser is designed and developed, so that the user can interact with the drug pest detection model in the terminal device through the user interface in the mobile device and the web browser.

Specifically, the user interface is used for receiving the traditional Chinese medicine image to be detected input by the user, inputting the traditional Chinese medicine image to be detected to the traditional Chinese medicine disease and pest detection model, so that the traditional Chinese medicine disease and pest detection model detects the traditional Chinese medicine disease and pest of the traditional Chinese medicine image to be detected, and outputting the traditional Chinese medicine disease and pest result.

In one embodiment, after the plant diseases and insect pests of the traditional Chinese medicinal materials are output, the plant diseases and insect pests of the traditional Chinese medicinal materials are visually displayed based on a display module of the terminal equipment, so that a user can directly observe the plant diseases and insect pests.

Embodiment 2, referring to fig. 2, fig. 2 is a schematic structural diagram of an embodiment of a plant disease and insect pest detection device for a traditional Chinese medicine provided by the invention, as shown in fig. 2, the device includes an image acquisition module 201, a multi-scale feature fusion neural network model training module 202 and a traditional Chinese medicine plant disease and insect pest detection module 203, and specifically includes the following steps:

The image acquisition module 201 is configured to acquire multi-angle plant diseases and insect pests images of the traditional Chinese medicinal materials, and perform image preprocessing on the plant diseases and insect pests images of the traditional Chinese medicinal materials at each angle to obtain multi-angle plant diseases and insect pests preprocessed images of the traditional Chinese medicinal materials.

The multi-scale feature fusion neural network model training module 202 is configured to perform model training on a multi-scale feature fusion neural network model based on the traditional Chinese medicine plant disease and insect pest pretreatment image, so that the multi-scale feature fusion neural network model performs global feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain an image global feature, performs local feature extraction on the traditional Chinese medicine plant disease and insect pest pretreatment image to obtain an image local feature, performs stitching processing on the image global feature and the image local feature, calculates a weight value of the stitching feature, performs feature fusion processing on the image global feature and the image local feature based on the weight value to obtain an image fusion feature, performs model parameter optimization processing on the multi-scale feature fusion neural network model based on the image fusion feature, and determines an optimal model parameter.

The herbal plant disease and insect pest detection module 203 is configured to generate a herbal plant disease and insect pest detection model based on the optimal model parameters, and input the received image of the herbal plant to be detected into the herbal plant disease and insect pest detection model, so that the herbal plant disease and insect pest detection model outputs a herbal plant disease and insect pest result.

In one embodiment, the image acquisition module 201 is configured to acquire images of plant diseases and insect pests of the traditional Chinese medicinal materials at multiple angles, and specifically includes: setting a traditional Chinese medicine plant disease and insect pest image acquisition device and a transparent conveyor belt, wherein the traditional Chinese medicine plant disease and insect pest image acquisition device comprises an automatic rotation platform and image acquisition equipment, and the image acquisition equipment is arranged in the automatic rotation platform; controlling the image acquisition equipment to rotate the transparent conveyor belt according to a preset rotation angle interval based on the automatic rotation platform; in the rotation process, based on the image acquisition equipment, image acquisition processing is carried out on the traditional Chinese medicine crops placed on the transparent conveyor belt, and multi-angle traditional Chinese medicine plant diseases and insect pests images are obtained.

In one embodiment, the image acquisition module 201 is configured to perform image preprocessing on the plant diseases and insect pests images of the traditional Chinese medicinal materials at each angle to obtain a multi-angle plant diseases and insect pests preprocessed image, and specifically includes: respectively carrying out image filtering treatment on the Chinese herbal medicine plant disease and insect pest images at each angle to obtain Chinese herbal medicine filtering images, and carrying out histogram equalization treatment on the Chinese herbal medicine filtering images to obtain Chinese herbal medicine equalization images; performing color space conversion on the traditional Chinese medicine equalization image to obtain a target color space image, and performing saliency detection on the target color space image to obtain a first salient region image corresponding to the target color space image; normalizing the first significant region image to obtain a normalized significant region image, and multiplying the normalized significant region image and the traditional Chinese medicinal material equalization image to obtain a second significant region image; threshold segmentation is performed on the second salient region image so that the second salient region image is divided into foreground regions, and an initialization mask is set based on the foreground regions; determining an interested region of the second significant region image, inputting the interested region, the initialization mask and the traditional Chinese medicine equalization image into a preset matting algorithm, so that the matting algorithm carries out iterative optimization processing on a segmentation result of the traditional Chinese medicine equalization image to obtain an image segmentation result; and carrying out image segmentation processing on the traditional Chinese medicine equalization image based on the image segmentation result to obtain a traditional Chinese medicine plant disease and insect pest pretreatment image.

In one embodiment, the multi-scale feature fusion neural network model training module 202 is configured to perform global feature extraction on the plant disease and pest pretreatment image of the traditional Chinese medicinal materials to obtain an image global feature, and specifically includes: the multi-scale feature fusion neural network model comprises a deep convolution network; mapping the traditional Chinese medicine plant disease and insect pest pretreatment image into a depth feature space based on the depth convolution network to obtain a plurality of first feature images; performing convolution processing on the plurality of first features based on a preset convolution check to obtain a plurality of second feature images, and performing combination processing on the plurality of second feature images according to channels to obtain a feature result image; and carrying out point-by-point convolution processing on the characteristic result graph to obtain the image global characteristic.

In one embodiment, the deep convolutional network performs a global feature extraction formula as follows:

；

In one embodiment, the multi-scale feature fusion neural network model training module 202 is configured to perform local feature extraction on the plant disease and insect pest pretreatment image of the traditional Chinese medicinal materials to obtain image local features, and specifically includes: the multi-scale feature fusion neural network model comprises a shift window converter network, wherein the shift window converter network comprises a multi-head self-attention mechanism, a multi-level perception mechanism and an offset window multi-head self-attention mechanism; respectively carrying out linear mapping processing on the plurality of first feature images to obtain corresponding linear first feature images, respectively carrying out weighting processing on the linear first feature images based on the multi-head self-attention mechanism to obtain first weighted feature images, and carrying out addition processing on the first weighted feature images and the corresponding first feature images to obtain residual connection feature images; performing linear mapping processing on the residual error connection characteristic map to obtain a linear residual error connection characteristic map, and performing nonlinear transformation on the linear residual error connection characteristic map based on the multi-level perception mechanism to obtain a nonlinear residual error connection characteristic map; adding the nonlinear residual connection characteristic diagram and the residual connection characteristic diagram to obtain an offset rate window input characteristic diagram; performing linear mapping processing on the offset window input feature map to obtain a corresponding linear offset window input feature map, respectively performing weighting processing on the linear offset window input feature map based on the offset window multi-head self-attention mechanism to obtain a weighted linear offset window input feature map, and performing addition processing on the weighted linear offset window input feature map and the corresponding offset window input feature map to obtain an offset window residual error connection feature map; performing linear mapping processing on the offset window residual error connection characteristic map to obtain a linear offset window residual error connection characteristic map, and performing nonlinear transformation on the linear offset window residual error connection characteristic map based on the multi-level sensing mechanism to obtain a nonlinear offset window residual error connection characteristic map; and adding the nonlinear offset rate window residual error connection characteristic diagram and the offset rate window residual error connection characteristic diagram to obtain an image local characteristic.

In an embodiment, the multi-scale feature fusion neural network model training module 202 is configured to perform a stitching process on the global feature of the image and the local feature of the image, and calculate a weight value of the stitched feature, and specifically includes: performing dimension conversion on the image global feature based on a full connection layer to obtain a target dimension global feature, performing dimension conversion on the image local feature based on the full connection layer to obtain a target dimension local feature, and performing splicing processing on the target dimension global feature and the target dimension local feature to obtain a spliced feature; performing weight value calculation on the splicing features based on a preset weight value calculation formula to obtain weight values of the splicing features; the preset weight value calculation formula is as follows:

；

It will be clear to those skilled in the art that, for convenience and brevity of description, reference may be made to the corresponding process in the foregoing method embodiment for the specific working process of the above-described apparatus, which is not described in detail herein.

It should be noted that, the embodiments of the plant diseases and insect pests detecting device of the above-mentioned traditional Chinese medicinal materials are only schematic, where the modules described as separate components may or may not be physically separated, and the components displayed as the modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

On the basis of the embodiment of the method for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials, another embodiment of the invention provides plant diseases and insect pests of the traditional Chinese medicinal materials, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to realize the method for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials of any embodiment of the invention.

Illustratively, in this embodiment the computer program may be partitioned into one or more modules, which are stored in the memory and executed by the processor to perform the present invention. The one or more modules may be a series of instruction segments of a computer program capable of performing a specific function, where the instruction segments are used to describe the execution process of the computer program in the pest detection terminal device of the traditional Chinese medicine.

The plant diseases and insect pests detection terminal equipment of the traditional Chinese medicinal materials can be computing equipment such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The plant diseases and insect pests detection terminal equipment of the traditional Chinese medicinal materials can comprise, but is not limited to, a processor and a memory.

The Processor may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The general processor may be a microprocessor or any conventional processor, etc., and the processor is a control center of the plant disease and insect pest detection terminal device of the traditional Chinese medicine, and is connected with various parts of the plant disease and insect pest detection terminal device of the whole traditional Chinese medicine by various interfaces and lines.

The memory can be used for storing the computer program and/or the module, and the processor can realize various functions of the plant diseases and insect pests detection terminal device of the traditional Chinese medicinal materials by running or executing the computer program and/or the module stored in the memory and calling the data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the cellular phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.

On the basis of the embodiment of the method for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials, another embodiment of the invention provides a storage medium, which comprises a stored computer program, wherein when the computer program runs, equipment where the storage medium is located is controlled to execute the method for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials of any embodiment of the invention.

In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, where the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form, and so on. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

In summary, the method, the device, the equipment and the storage medium for detecting the plant diseases and insect pests of the traditional Chinese medicine provided by the invention are characterized in that model training is carried out on the multi-scale feature fusion neural network model through the multi-angle traditional Chinese medicine plant disease and insect pest preprocessing image, so that global features and local features of the traditional Chinese medicine plant disease and insect pest preprocessing image are extracted by the multi-scale feature fusion neural network model, the global features and the local features of the image are obtained and spliced, the weight value of the spliced features is calculated, the global features and the local features of the image are subjected to feature fusion based on the weight value, the model parameter optimization processing is carried out on the multi-scale feature fusion neural network model based on the obtained image fusion features, the traditional Chinese medicine plant disease and insect pest detection model is generated based on the optimal model parameters, and the received traditional Chinese medicine plant disease and insect pest image to be detected is input into the traditional Chinese medicine plant disease and insect pest detection model, and the traditional Chinese medicine plant disease and insect pest result is output.

The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and substitutions can be made by those skilled in the art without departing from the technical principles of the present invention, and these modifications and substitutions should also be considered as being within the scope of the present invention.

Claims

1. The method for detecting the plant diseases and insect pests of the traditional Chinese medicinal materials is characterized by comprising the following steps of:

2. The method for detecting plant diseases and insect pests of a traditional Chinese medicine according to claim 1, wherein the collecting of multi-angle plant diseases and insect pests images of the traditional Chinese medicine specifically comprises:

3. The method for detecting plant diseases and insect pests of a Chinese medicinal material according to claim 1, wherein the image preprocessing is performed on the plant diseases and insect pests image of the Chinese medicinal material at each angle to obtain a multi-angle plant diseases and insect pests preprocessed image, specifically comprising:

4. The method for detecting plant diseases and insect pests of a traditional Chinese medicine according to claim 1, wherein the step of performing global feature extraction on the plant diseases and insect pests pretreatment image of the traditional Chinese medicine to obtain an image global feature comprises the following steps:

5. The method for detecting plant diseases and insect pests of Chinese medicinal materials according to claim 4, wherein the formula of global feature extraction is performed by the deep convolution network as follows:

；

in the method, in the process of the invention, For global features of the image,/>Separable convolution operations for depth space, including on input images/>Convolution processing of/>Performing operations for point-by-point convolution,/>Is a characteristic result graph,/>For depth space separable convolutional network parameters,/>Convolving the network parameters for vgg < 16 >Is the ratio of the depth space separable convolutional network parameters to the VGG16 convolutional network parameters,/>For the number of parameters of a depth-space separable convolutional network,/>For convolution kernel size,/>For the number of convolution kernels,/>For/>Number of convolution kernels.

6. The method for detecting plant diseases and insect pests of a Chinese medicinal material according to claim 4, wherein the extracting the local features of the plant diseases and insect pests pretreatment image of the Chinese medicinal material to obtain the local features of the image specifically comprises:

7. The method for detecting plant diseases and insect pests of a Chinese medicinal material according to claim 1, wherein the performing a stitching process on the global image feature and the local image feature, and calculating a weight value of the stitching feature specifically comprises:

；

in the method, in the process of the invention, Dimension conversion operation representing global connection layer,/>Representing a sigmoid function,/>For global features of the image,/>Is a local feature of the image,/>Is a weight value.

8. The utility model provides a disease and pest detection device of chinese-medicinal material, its characterized in that includes: the system comprises an image acquisition module, a multi-scale feature fusion neural network model training module and a traditional Chinese medicine grass pest detection module;

9. A terminal device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method for detecting the plant diseases and insect pests of the Chinese herbal medicine according to any one of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, wherein the computer readable storage medium comprises a stored computer program, and wherein the computer program when run controls a device in which the computer readable storage medium is located to perform the method for detecting plant diseases and insect pests of the chinese herbal medicine according to any one of claims 1 to 7.