CN115063453A - Plant leaf stomata individual behavior detection and analysis method, system and storage medium - Google Patents
Plant leaf stomata individual behavior detection and analysis method, system and storage medium Download PDFInfo
- Publication number
- CN115063453A CN115063453A CN202210730549.0A CN202210730549A CN115063453A CN 115063453 A CN115063453 A CN 115063453A CN 202210730549 A CN202210730549 A CN 202210730549A CN 115063453 A CN115063453 A CN 115063453A
- Authority
- CN
- China
- Prior art keywords
- frame
- target detection
- stomata
- air hole
- air holes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V10/422—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation for representing the structure of the pattern or shape of an object therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The application provides a plant leaf stomata individual behavior detection and analysis method, a system and a storage medium. According to the method, the positions of the air holes in the video image are identified through a multi-target tracking algorithm, the opening and closing states of the air holes in the video image are detected through a target detection algorithm, the air holes are matched, the opening and closing time of the air holes is marked, and the tracking analysis of the opening and closing time sequence of the air holes is realized. According to the method, the plant leaf videos are detected and analyzed, and the integrated system of the multi-deep learning task is utilized to obtain various properties such as the opening and closing state, the length and the width, the area and the circumference of the stomata. Compared with the traditional pore property research, the invention realizes the qualitative tracking of the pore opening and closing state and the quantitative analysis of the pore shape characteristics by the overtime sequence analysis technology of the pore video and combining the algorithms of target detection, semantic segmentation and the like, is beneficial to analyzing the physiological mechanism of the opening and closing rhythm of the pore individuals and reveals the motion rule of the pore individuals and the population level.
Description
Technical Field
The application relates to the technical field of plant information monitoring, in particular to a method and a system for detecting and analyzing individual behaviors of stomata of plant leaves and a storage medium.
Background
Stomata on plant leaves are important channels for the plant to exchange CO2 and water with the external environment. Morphological changes of stomata directly affect important physiological processes of photosynthesis, transpiration, water utilization efficiency, stress resistance and the like of plants.
Under the common regulation and control of endogenous signals and external environmental factors, the individual behaviors of stomata show great difference. Due to the limitation of observation methods, the response of a single pore to external stimuli such as environment and the like is considered to be independent and similar in the early days, and the behavior of the pore only shows slight random differences. However, a number of observations and studies have shown that the behaviour of stomata in plant leaves often shows significant spatiotemporal heterogeneity: for example, in some regions of the same plant leaf, stomata are open, and in other regions, stomata close. This heterogeneity of stomatal behaviour is considered to be an important mechanism for plant adaptation to the environment.
However, existing analysis techniques for the behavior of individual stomata are not complete. Traditional stomata phenotype character analysis relies on manual calibration, and is tedious, time-consuming and labor-consuming. In recent years, researchers have developed software that can identify stomata or divide epidermal cells, but such software has limited accuracy and requires a great deal of effort from the user to perform manual correction. In addition, although the existing deep learning image analysis algorithm can accurately quantify parameters such as stomatal density, the detailed character analysis of the individual behaviors of the stomata cannot be performed.
In addition, the stomata of the plants are quick and sensitive to the response process of drought and other stress environments, and time sequence data is favorable for more accurately reflecting stomata changes. However, the conventional pore observation methods mostly use a silica gel oil blotting method or a destructive skin peeling method, and these methods have difficulty in achieving intensive time-series observation. In the prior art, continuous observation of the pore individuals is difficult, and the defects of the continuous observation technology also cause that the existing pore analysis technology is mostly concentrated on a single-point image analysis algorithm, thereby causing a blank research on the state time sequence analysis of the pore individuals.
Disclosure of Invention
Aiming at the defects of the prior art, the method, the system and the storage medium for detecting and analyzing the individual behaviors of the air holes of the plant leaves are provided, the prediction tracking of the air hole detection frame is realized by matching a Kalman filtering algorithm with a Hungarian algorithm, the relevance among all frame sequences of a video image can be described through a multi-target tracking algorithm, an identity card is given to each air hole, and then the opening and closing state and the shape characteristics of each air hole are identified and tracked respectively. The technical scheme is specifically adopted in the application.
Firstly, in order to achieve the above object, a plant leaf stomata individual behavior detection and analysis method is provided, which performs the following steps for each frame of image according to the sequence of each frame of plant leaf video image: firstly, identifying target detection frames corresponding to air holes in each frame image of a video through a multi-target tracking algorithm, and marking air hole identity information corresponding to each target detection frame; secondly, respectively identifying the opening and closing states of the air holes in each frame of image of the video through a target detection algorithm, matching the opening and closing states of the air holes with the identity information of the air holes, and respectively marking the opening and closing states and the opening and closing time corresponding to the air holes; and thirdly, inputting the corresponding pore pictures into the semantic segmentation model according to the target detection frames, detecting to obtain pore complex areas in the pore pictures, performing mask output on the video images according to the pore complex areas, and respectively extracting morphological characteristics corresponding to the pores on the basis of the mask images.
Optionally, the method for detecting and analyzing the individual behavior of stomata of plant leaves as described above, wherein the multi-target tracking algorithm includes the following steps: 101, obtaining target detection frames respectively corresponding to air holes on a blade through detection of a target detector, and generating a frame prediction frame according to the target detection frame of the previous frame through a Kalman filtering algorithm; and 102, respectively identifying the association condition of the air holes in each frame according to the matching degree between the target detection frame and the current frame prediction frame, and marking the identity information of the air holes corresponding to each target detection frame.
Optionally, the method for detecting and analyzing the individual behavior of stomata of plant leaves as described above, wherein step 102 specifically includes the following steps: matching a target detection frame with a frame prediction frame through a Hungarian algorithm, if only the target detection frame exists but the frame prediction frame matched with the target detection frame does not exist, marking the identity information of the air hole newly added to the air hole corresponding to the target detection frame, and adding a track corresponding to the identity information of the air hole; if the target detection frame is matched with the frame prediction frame, continuously updating the target detection frame through a Kalman filtering algorithm to generate a prediction frame for matching the next frame target detection frame; and if the target detection frame matched with the frame prediction frame does not exist, deleting the track corresponding to the air hole identity information to which the frame prediction frame belongs.
Optionally, in the method for detecting and analyzing individual behaviors of plant leaf stomata, a target detection frame corresponding to each stomata identified by the multi-target tracking algorithm is marked as [ x1, y1, x2, y2, ID ], where (x1, y1) and (x2, y2) respectively represent pixel coordinates of two opposite corners of the target detection frame, and the ID represents identity information of the stomata corresponding to the target detection frame; in the second step, the open-close state of the air holes in each frame of image of the video is respectively identified through a target detection algorithm, the open-close state of each air hole is matched with the identity information of the air hole, and the specific steps of respectively marking the open-close state and the open-close time corresponding to each air hole comprise: respectively identifying the open-close state of the air holes in each frame of image of the video through a two-classification target detection algorithm based on YOLOv3-tiny, and recording as a state detection frame [ x ', y', x ", y", category ], wherein (x ', y') and (x ", y") respectively represent pixel coordinates of two opposite angles of the state detection frame, and the category represents the open-close state of the air holes corresponding to the state detection frame; acquiring time information in a frame video image through a character recognition algorithm; comparing the pixel coordinate distances between the state detection frame [ x ', y', x ", y", category ] and the target detection frame [ x1, y1, x2, y2, ID ], updating the open/closed state of the target detection frame to the open/closed state of the state detection frame closest to the pixel coordinate distance thereof, and outputting [ x1, y1, x2, y2, ID, category, time ].
Optionally, in the method for detecting and analyzing individual behaviors of stomata in plant leaves, the semantic segmentation algorithm is a deep learning model based on uet and is formed by a network structure of an encoder-decoder; the encoder performs feature extraction on image downsampling of a target detection block layer by layer, each layer of encoder fuses extracted features to a decoder of a corresponding layer through a depth separable convolution structure, each layer of decoder performs upsampling on the layer by layer and then adjusts the dimension of a feature layer to be 2 by using 2-dimensional convolution, then expands the features, and performs binary classification on each pixel point by using a softmax activation function to detect and determine whether each pixel point belongs to an air pore complex area.
Optionally, the method for detecting and analyzing individual plant leaf stomata behavior includes 4 layers, wherein only the three middle layers of the encoder in the semantic segmentation model are respectively feature-fused with the upsampling decoder, and after 3 times of feature layer fusion operations, the feature layer output by the last decoder is adjusted to have a dimension of 2 by using 2-dimensional convolution so as to respectively mark stomata pixel points and background pixel points.
Optionally, the method for detecting and analyzing individual behaviors of stomata of plant leaves as described above, wherein the step of respectively extracting morphological features corresponding to each stomata according to the stomata complex region includes: calculating the outline corresponding to the air hole complex area, calculating the air hole perimeter according to the outline, calculating the air hole area according to the number of pixel points in the outline, and calculating the length and width of the air hole according to the difference of extreme values of horizontal and vertical coordinates on the outline.
Optionally, the method for detecting and analyzing the individual behavior of stomata of the plant leaf as described above, wherein the step of calculating the contour corresponding to the stomata complex area specifically includes: and corroding and expanding the mask image of the air hole complex area, and after removing the noise points, calculating the contour of the residual communication area as the contour corresponding to the air hole complex area.
Simultaneously, in order to realize above-mentioned mesh, this application still provides a plant leaf gas pocket individual behavior detection analytic system, and it includes: the microscope is used for shooting video images of stomata in the plant leaves; a detection and analysis unit for executing the plant leaf stomata individual behavior detection and analysis method; and an output unit which outputs video images or graph data by masking according to the air hole identity information, morphological characteristics and/or open-close state and/or air hole complex area of each air hole obtained by the detection and analysis unit.
The present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processing device, implements the method as described in any one of the above.
Advantageous effects
According to the method and the device, the positions of the air holes in the video image are identified through the multi-target tracking algorithm, the opening and closing states of the air holes in the video image are detected through the target detection algorithm, then the air holes are matched, and the opening and closing time of the air holes is marked, so that the analysis of the opening and closing time sequence of the air holes is realized. According to the method, through detection and analysis of the plant leaf stomata video, a plurality of stomata individual characters including the opening and closing states, the length and the width, the area, the perimeter and the like of the stomata are obtained by utilizing an integrated system of a multi-deep learning task. Compared with the traditional pore property research, the invention can qualitatively track the open and close state of the pore and quantitatively detect the morphological characteristics of the pore by the overtime sequence analysis technology of the pore video and combining algorithms such as target detection, semantic segmentation and the like, thereby helping scientific researchers to better analyze the physiological mechanism of the open and close of the pore individuals and disclosing the motion rule of the pore individuals and the population level.
The method utilizes a YOLOv3-tiny algorithm to respectively carry out multi-target tracking and target detection of the open and close states of the air holes, wherein the first multi-target detection only returns information of a rectangular target detection frame of the air holes, and the second open and close state detection returns a state detection frame for marking the open and close states of the air holes. This application will carry out the tracking of gas pocket target respectively and to the detection of gas pocket switching state, only detect the gas pocket for the first time alone and be favorable to improving the precision performance of tracking algorithm. If a conventional design is adopted, the tracking of the opening and closing type of the air hole is directly added in the first target detection, and the algorithm can respectively judge the air holes in the two states of opening and closing as two different objects, so that the target tracking performance can be seriously influenced. The air hole information obtained by the two detections can be matched one by one through the distance between the pixel coordinates of the detection frame returned by the detection algorithm, so that the air hole target states obtained by the two detections are fused into a uniform state detection result [ x1, y1, x2, y2, ID, category, time ]. This application accessible detects the central point interval between the frame twice, matches the coordinate position under the frame is detected twice near to same gas pocket identity information, judges the independent time sequence tracking that realizes each gas pocket target through simple coordinate position from this.
In addition, the method further utilizes the Unet deep learning model to carry out air hole semantic segmentation, improves an encoder in an encoder-decoder network structure of the method, and fuses the features extracted by the encoder into corresponding layers through a deep separable convolution structure. According to the method and the device, the conventional convolution is replaced by the depth separable convolution structure, so that the parameter quantity of the model is less, and the detection speed of the model is further accelerated.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application.
Drawings
The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application and not limit the application. In the drawings:
FIG. 1 is a schematic flow chart of the steps of the method for detecting and analyzing the individual behavior of stomata of plant leaves according to the present application;
FIG. 2 is a schematic diagram of a semantic segmentation model as used in the present application;
FIG. 3 is a real-time video image of a blade pore output by the method of the present application;
FIG. 4 is a trace image of each air hole obtained by the present application performing timing tracking on different air holes;
fig. 5 is a graph of the motion law of each air hole obtained by performing time-series tracking on different air holes according to the present application.
Detailed Description
In order to make the purpose and technical solutions of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings of the embodiments of the present application. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the application without any inventive step, are within the scope of protection of the application.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The meaning of "and/or" as used herein is intended to include both the individual components or both.
The term "connected" as used herein may mean either a direct connection between components or an indirect connection between components via other components.
Fig. 1 is a computer program execution unit for a plant leaf stomata individual behavior detection and analysis system according to the application. The system comprises:
a microscope for taking video images of stomata in plant leaves like that shown in fig. 3;
and the detection and analysis unit is used for correspondingly executing a computer program stored in a readable storage medium by computer processing equipment such as a GPU, a CPU, a singlechip, an FPGA and the like so as to respectively execute the following steps on each frame of image according to the sequence of each frame of the plant leaf video image in the following way to realize the detection and analysis of the individual behaviors of each air hole on the plant leaf: firstly, identifying target detection frames corresponding to air holes in each frame image of a video through a multi-target tracking algorithm, and marking air hole identity information corresponding to each target detection frame; secondly, respectively identifying the opening and closing states of the air holes in each frame of image of the video through a target detection algorithm, matching the opening and closing states of the air holes with the identity information of the air holes, and respectively marking the opening and closing states and the opening and closing time corresponding to the air holes; inputting the corresponding pore pictures into a semantic segmentation model according to the target detection frames, detecting to obtain pore complex areas in the pore pictures, performing mask output on the video images according to the pore complex areas, and respectively extracting morphological characteristics corresponding to the pores on the basis of the mask images;
and the output unit can be set as a display or a display unit of the mobile terminal equipment, and can also selectively output a video image similar to that shown in fig. 3 or corresponding graph data similar to that shown in fig. 4 and 5 by a mask according to the air hole identity information, morphological characteristics and/or opening and closing states and/or air hole complex areas of each air hole obtained by the detection and analysis unit in a form of table data.
Therefore, the system can analyze the difference among the individual plant stomata behaviors by taking the dynamic leaf stomata video shot by the portable microscope as input and through an integrated system consisting of deep learning algorithms such as multi-target tracking, target detection, character recognition, semantic segmentation and the like, can realize full-automatic, high-precision and high-efficiency analysis of the individual plant stomata behaviors of the plant stomata, and provides an effective tool for further exploring the physiological mechanism of the plant stomata.
The plant leaf stomata individual behavior detection and analysis method adopted by the detection and analysis unit can be realized in the following way in specific practice:
step one, a multi-target tracking algorithm is called to endow each air hole in each frame of image picture of the video with a unique identifier, and the identity information of the air hole corresponding to each target detection frame is recorded; the multi-target tracking algorithm has the following working principle:
101, detecting and obtaining target detection frames respectively corresponding to air holes on a blade by using a lightweight yolov3-tiny target detection technology through a multi-target tracking algorithm (SORT) (simple Online and real tracking), and then generating a frame prediction frame according to the target detection frame of the previous frame through a Kalman filtering algorithm;
102, respectively identifying the correlation condition of the air holes in each frame according to the matching degree between the target detection frame and the prediction frame of the frame, and marking the identity information of the air holes corresponding to each target detection frame, so as to return the target detection frame marked as [ x1, y1, x2, y2, ID ]; wherein (x1, y1) may represent the coordinates of the upper left corner point of the boundary of the target detection box, (x2, y2) may represent the coordinates of the lower right corner of the boundary of the target detection box, ID is used to identify the air vent identity information corresponding to the target detection box, and the pixel coordinates of the other two opposite corners of the target detection box may also be used to mark the air vent coordinate position.
Secondly, calling a target detection algorithm to respectively identify the open-close states of the air holes in each frame of image of the video, matching the open-close states of the air holes with the identity information of the air holes, and respectively marking the open-close states and the open-close times corresponding to the air holes; the working principle of the target detection algorithm is as follows:
step 201, respectively identifying the open and close states of air holes in each frame of image of the video through a two-classification target detection algorithm based on YOLOv3-tiny, and marking as a state detection frame [ x ', y', x ", y", category ], wherein (x ', y') and (x ", y") respectively represent pixel coordinates of two opposite angles of the state detection frame, and the category represents the open and close states of the air holes corresponding to the state detection frame;
step 202, acquiring time information in a frame video image through an OCR character recognition algorithm; in order to improve the operation precision performance, YOLOv3-tiny is respectively carried out on the frame image twice, and a target detection frame and a state detection frame which are independent of each other are respectively obtained to carry out target detection so as to respectively mark the air hole position ID and the switch state, so that the method also needs to match the state detection frame [ x ', y', x ', y', category ] with the target detection frame [ x1, y1, x2, y2, ID ] one by one through comparison of pixel coordinate distances. In the matching process, a central point set of the state detection frame and a central point set of the target detection frame are respectively obtained according to the information of the coordinate points of the detection frames, one central point in the target detection frame set is selected, Euclidean distances are calculated from all the central points in the state detection frame set, the point with the shortest distance is selected for matching according to the distance between the two points, the opening and closing state of the target detection frame can be updated to the opening and closing state of the state detection frame with the closest pixel coordinate distance, and [ x1, y1, x2, y2, ID, category and time ] is output.
Inputting the corresponding air hole picture into a semantic segmentation model according to each target detection frame to detect each detection frame pixel by pixel to obtain an air hole complex area in the air hole picture, then performing morphological analysis according to a mask image of the air hole complex area, further calculating information such as the length, the width, the area, the perimeter and the like of the air hole, performing mask output on a video image according to the air hole complex area, and respectively extracting morphological characteristics corresponding to the air hole; the semantic segmentation model can be realized by the Unet-based semantic segmentation model shown in FIG. 2:
the semantic segmentation model is composed of a network structure of an encoder-decoder, receives an air hole picture intercepted according to coordinate information [ x1, y1, x2 and y2] of a detection frame (bounding box), performs feature extraction on layer-by-layer down-sampling of a target detection frame image through the encoder, fuses extracted features of each layer of the target detection frame image to a decoder of a corresponding layer through a depth separable convolution structure respectively through encoders in the model, performs up-sampling on each layer of the decoder layer by layer, adjusts the dimension of a feature layer to be 2 by using 2-dimensional convolution, expands the features, and performs secondary classification on each pixel by using a softmax activation function to detect whether each pixel belongs to an air hole complex area or not.
Therefore, the semantic segmentation algorithm is based on a deep learning model of Unet, and can replace the conventional convolution with the deep separable convolution at a decoder part for feature extraction, so that the parameter quantity of the model is less, and the detection speed of the model is effectively accelerated. In the alternative process, the depth separable convolution and the conventional convolution can be modularly replaced: depth separable convolutions, when embodied, consist of depth convolutions and point convolutions. The deep convolution is responsible for carrying out independent convolution operation on each channel; the point convolution is responsible for adjusting the number of output channels. Since the depth separable convolution is the same shape as the output of a conventional convolution, it can be modularly replaced in the model structure.
In other implementation manners, the detection analysis method of the present application may further perform the following operations on a given original frame of the video to track the behavior of the plant leaf stomata individuals:
1. and (4) carrying out stomata individual identification based on a multi-target tracking algorithm.
In order to endow each air hole in a video with a fixed ID and further study the behavior change of each air hole, a target detector YOLOv3-tiny (counting) is operated to detect, so that a target detection box is obtained through lightweight target detection, and the individual air hole tracks of a time sequence are obtained by matching with a multi-target tracking technology realized by a Hungary algorithm and a Kalman filtering algorithm.
The multi-target tracking algorithm relies on the detection results of target detection. In order to realize high-efficiency multi-target tracking, firstly, a light-weight and high-efficiency YOLOv3-tiny target detector is utilized to detect the positions of all gas holes in a first frame of a video, and the positions are marked by a rectangular frame (bounding box), namely a target detection frame;
and then, the detection frame obtained by the first frame is transmitted to a Kalman filtering algorithm so as to predict the motion variable of the next frame according to the current motion variable through the Kalman filtering algorithm. Predicting the next frame position of the first frame detection frame through a Kalman filtering algorithm to obtain a second frame prediction frame;
and continuously carrying out air hole target detection on the second frame by the YOLOv3-tiny target detector to obtain a second frame target detection frame, and in order to realize the best matching of the detection frame and the prediction frame, distributing by a Hungarian algorithm to enable the prediction frame to find the best matching detection frame, thereby achieving the tracking effect.
In the matching process of the Hungarian algorithm, three matching situations can occur: (1) if the target detection frame is not matched with the prediction frame, only the prediction frame is needed, and the target detection frame is not matched with the current frame prediction frame, the air hole is indicated to possibly move and disappear in the picture, and at the moment, the track corresponding to the identity information of the air hole to which the current frame prediction frame belongs needs to be deleted; (2) if the object is not detected, only a target detection frame exists but no frame prediction frame matched with the target detection frame exists, the air hole appears in the picture for the first time, so that the identity information of the air hole corresponding to the target detection frame needs to be marked separately and newly added with a track corresponding to the identity information of the air hole; (3) and matching the object, wherein the prediction frame is matched with the prediction frame of the current frame, which indicates that the previous frame and the next frame are successfully tracked, and at the moment, the target detection frame can be continuously updated through a Kalman filtering algorithm to generate the prediction frame detection frame matching for matching the target detection frame of the next frame. The individual pore tracks of the time sequence shown in fig. 4 can be obtained by repeating the above steps until the video frame is finished. Take the first exhaust hole 1 of fig. 4 as an example. The air holes are opened at the early timing point on the left side, and the openings between the air holes are gradually closed at the late timing point on the right side, so that the opening and closing time sequence track of a single air hole is embodied.
2. Monitoring of open and closed states of stomata individual based on target detection
In order to obtain the open-close state of each air hole in each frame, a two-classification target detection algorithm based on Yolov3-tiny (identification open-close) is used for carrying out open-close detection on the air hole, and a state detection frame and the corresponding open-close state of the air hole are obtained. The method comprises the steps of matching a state detection frame and a target detection frame at the shortest distance, judging the association degree between the two detection frames, and matching the target detection frame to the state detection frame at the shortest distance from the state detection frame, so that data association is realized according to the opening and closing type information marked by the state detection frame, marking the detection frame and the state detection frame which belong to one air hole by the same air hole identity information ID, and distributing the opening and closing state type of an air hole target for each target detection frame object.
And acquiring time information of the upper left corner of the current frame of the video through character recognition, further marking the time information for different opening and closing states of each target detection frame, and finally outputting the opening and closing conversion process of each air hole in a mode of fig. 5.
3. Pore morphological parameter analysis based on semantic segmentation
In order to further analyze morphological parameters of each air hole, a single air hole complex mask image is obtained by using a semantic segmentation algorithm, and properties such as the length, the width, the area and the perimeter of the air hole are analyzed according to a mask region.
In the specific implementation process, firstly, according to the coordinate information [ x1, y1, x2 and y2] of the detection box in the target detection process, screenshot is carried out on the position of the air hole, the picture is input into the semantic segmentation model of fig. 2, the semantic segmentation model can automatically adjust the screenshot picture to be in a uniform size, or resihape is carried out on the picture in a manual mode to adjust the size of the picture, then, a deep learning algorithm based on Unet is used, the original input image is subjected to feature extraction of layer-by-layer down-sampling by using an encoder on the left side of fig. 2 through a network structure of the encoder-decoder, and the decoder on the right side of fig. 2 is used for up-sampling. In order to realize multi-scale feature fusion, three layers f1, f2 and f3 in the middle layer of an encoder in a semantic segmentation model are respectively subjected to feature fusion with a right-side up-sampling decoder to obtain a better segmentation result; after 3 times of feature layer fusion operation, adjusting the dimension of the feature layer to be 2 by using 2-dimensional convolution on the feature layer output by the last layer of decoder so as to divide each pixel point in the image into two categories of background pixels and pore pixels; expanding the features by using reshape, performing secondary classification on each pixel point by using softmax, and outputting the probability that each pixel point is a background and pore complex; and finally, detecting the pore complex area according to a threshold value of the probability, and outputting a mask image for marking the pore position.
After the mask image is obtained, further performing corrosion and expansion operation on the mask image of the air hole complex region according to the semantic segmentation result of the air hole complex region to remove unnecessary noise; calculating the contours of the denoised connected regions as the contours corresponding to the air hole complex regions, and calculating the number of pixel points in the connected regions to be recorded as the areas of the air holes; after the outline of the connected region is obtained by using a findcontours function of opencv, the perimeter of the air hole can be further calculated according to the outline information; and obtaining information such as the length, width, eccentricity, opening, area, perimeter and the like of the air hole according to the difference of the extreme values of the x and y coordinates of the pixels in the communicated region.
When the eccentricity is calculated, an ellipse can be used for fitting the pore mask image, and then the calculation is carried out according to the eccentricity formula of the ellipse: the ratio of the distance from the center of the ellipse to the semi-major axis of the ellipse to obtain the eccentricity e of the ellipse fitted to the air hole as c/a. Where c is the focal length and a is the length of the semi-major axis.
When the opening degree is calculated, the area (S) of the mask image in different opening and closing states of the pores can be calculated, the opening degree is (S-Smin)/(Smax-Smin), and the opening degree can be represented by a numerical value in a range of [0,1 ]. Where Smin represents the minimum value of the pore area in the sequence and S represents the maximum value of the pore area in the sequence.
TABLE 1 time sequence chart of the detection results of each air hole
And (4) circulating the steps 1-3 until the video is finished.
In summary, the present application establishes a link between video frames through a target tracking algorithm, as shown in table 1, the method gives a unique ID to each pore in a video, thereby capturing morphological properties such as area, length, etc. of each pore at each moment, realizing continuity of time scale, and realizing detection and time sequence recording of motion characteristics of individual pores with finer granularity;
the method and the device combine with a target detection algorithm to realize the analysis of individual pore motion characteristics with finer granularity. As shown in fig. 4 and 5, the binary problem of the open and closed states of the air vents is achieved by target detection, and the difference of the circadian rhythm of the opening and closing of each air vent is quantified based on the time continuity achieved by the target tracking algorithm.
Therefore, the stomata behavior detection and analysis method provided by the application is beneficial to quantitative expression of the space-time heterogeneity among individual stomata, provides a beneficial tool for research of plant stomata, and is expected to further improve the existing plant stomata theory.
The above are merely embodiments of the present application, and the description is specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the protection scope of the present application.
Claims (10)
1. A plant leaf stomata individual behavior detection and analysis method is characterized in that the following steps are respectively executed on each frame of image according to the sequence of each frame of plant leaf video image:
firstly, identifying target detection frames corresponding to air holes in each frame image of a video through a multi-target tracking algorithm, and marking air hole identity information corresponding to each target detection frame;
secondly, respectively identifying the opening and closing states of the air holes in each frame of image of the video through a target detection algorithm, matching the opening and closing states of the air holes with the identity information of the air holes, and respectively marking the opening and closing states and the opening and closing time corresponding to the air holes;
and thirdly, inputting the corresponding pore pictures into the semantic segmentation model according to the target detection frames, detecting to obtain pore complex areas in the pore pictures, performing mask output on the video images according to the pore complex areas, and respectively extracting morphological characteristics corresponding to the pores on the basis of the mask images.
2. The method for detecting and analyzing the individual behavior of stomata of plant leaves as claimed in claim 1, wherein the multi-target tracking algorithm comprises the following steps:
101, obtaining target detection frames respectively corresponding to air holes on a blade through detection of a target detector, and generating a frame prediction frame according to the target detection frame of the previous frame through a Kalman filtering algorithm;
and 102, respectively identifying the correlation condition of the air holes in each frame according to the matching degree between the target detection frame and the frame prediction frame, and marking the identity information of the air holes corresponding to each target detection frame.
3. The method for detecting and analyzing the individual behavior of stomata of plant leaves as claimed in claim 2, wherein the step 102 specifically comprises the following steps:
by matching the target detection box with the frame prediction box through the Hungarian algorithm,
if only the target detection frame exists but the frame prediction frame matched with the target detection frame does not exist, marking the identity information of the air hole which is newly added for the air hole corresponding to the target detection frame, and additionally adding a track corresponding to the identity information of the air hole;
if the target detection frame is matched with the frame prediction frame, continuously updating the target detection frame through a Kalman filtering algorithm to generate a prediction frame for matching the next frame target detection frame;
and if the target detection frame matched with the frame prediction frame does not exist, deleting the track corresponding to the air hole identity information to which the frame prediction frame belongs.
4. The method for detecting and analyzing the individual behavior of stomata in plant leaves as claimed in claims 1 to 3, wherein the target detection boxes corresponding to each stomata identified by the multi-target tracking algorithm are marked as [ x1, y1, x2, y2, ID ], where (x1, y1) and (x2, y2) respectively represent the pixel coordinates of two opposite corners of the target detection box, and ID represents the identity information of the stomata corresponding to the target detection box;
in the second step, the open-close state of the air holes in each frame of image of the video is respectively identified through a target detection algorithm, the open-close state of each air hole is matched with the identity information of the air hole, and the specific steps of respectively marking the open-close state and the open-close time corresponding to each air hole comprise:
respectively identifying the open-close state of the air holes in each frame of image of the video through a two-classification target detection algorithm based on YOLOv3-tiny, and recording as a state detection frame [ x ', y', x ", y", category ], wherein (x ', y') and (x ", y") respectively represent pixel coordinates of two opposite angles of the state detection frame, and the category represents the open-close state of the air holes corresponding to the state detection frame;
acquiring time information in a frame video image through a character recognition algorithm;
comparing the pixel coordinate distances between the state detection frame [ x ', y', x ", y", category ] and the target detection frame [ x1, y1, x2, y2, ID ], updating the open/closed state of the target detection frame to the open/closed state of the state detection frame closest to the pixel coordinate distance thereof, and outputting [ x1, y1, x2, y2, ID, category, time ].
5. The method for detecting and analyzing the individual behavior of stomata of plant leaves as claimed in claim 1, characterized in that the semantic segmentation algorithm is a deep learning model based on Unet and is composed of a network structure of an encoder-decoder;
the encoder performs feature extraction on image downsampling of a target detection block layer by layer, each layer of encoder fuses extracted features to a decoder of a corresponding layer through a depth separable convolution structure, each layer of decoder performs upsampling on the layer by layer and then adjusts the dimension of a feature layer to be 2 by using 2-dimensional convolution, then expands the features, and performs binary classification on each pixel point by using a softmax activation function to detect and determine whether each pixel point belongs to an air pore complex area.
6. The method for detecting and analyzing plant leaf stomata individual behaviors as claimed in claim 5, wherein the encoder comprises 4 layers, only three layers in the middle layer of the encoder are respectively subjected to feature fusion with an up-sampling decoder in a semantic segmentation model, and after 3 times of feature layer fusion operations, the dimension of a feature layer output by the last layer of decoder is adjusted to be 2 by using 2-dimensional convolution so as to respectively mark stomata pixel points and background pixel points.
7. The method for detecting and analyzing the individual behavior of stomata of plant leaves according to claims 1 to 6, wherein the step of extracting morphological features corresponding to each stomata according to the stomata complex area comprises the following steps:
calculating the outline corresponding to the air hole complex area, calculating the perimeter of the air hole according to the outline, calculating the area of the air hole according to the number of pixels in the outline, and calculating the length and the width of the air hole according to the difference of extreme values of horizontal and vertical coordinates of the outline.
8. The method for detecting and analyzing the individual behavior of stomata of plant leaves according to claim 7, wherein the step of calculating the corresponding contour of the stomata complex area specifically comprises the following steps:
and corroding and expanding the mask image of the air hole complex area, and after removing the noise points, calculating the contour of the residual communication area as the contour corresponding to the air hole complex area.
9. A plant leaf stomata individual behavior detection and analysis system is characterized by comprising:
the microscope is used for shooting video images of stomata in the plant leaves;
a detection and analysis unit for executing the plant leaf stomata individual behavior detection and analysis method according to any one of claims 1 to 8;
and an output unit which outputs video images or graph data by masking according to the air hole identity information, morphological characteristics and/or open-close state and/or air hole complex area of each air hole obtained by the detection and analysis unit.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processing device, carries out the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210730549.0A CN115063453B (en) | 2022-06-24 | 2022-06-24 | Plant leaf stomata individual behavior detection and analysis method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210730549.0A CN115063453B (en) | 2022-06-24 | 2022-06-24 | Plant leaf stomata individual behavior detection and analysis method, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115063453A true CN115063453A (en) | 2022-09-16 |
| CN115063453B CN115063453B (en) | 2023-08-29 |
Family
ID=83202173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210730549.0A Active CN115063453B (en) | 2022-06-24 | 2022-06-24 | Plant leaf stomata individual behavior detection and analysis method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115063453B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112052836A (en) * | 2020-10-09 | 2020-12-08 | 南京农业大学 | System and method for monitoring opening and closing states of plant leaf air holes in real time |
| CN112801068A (en) * | 2021-04-14 | 2021-05-14 | 广东众聚人工智能科技有限公司 | Video multi-target tracking and segmenting system and method |
| CN112950700A (en) * | 2021-02-25 | 2021-06-11 | 安徽农业大学 | Plant leaf stomatal conductance measuring method based on deep learning and image processing |
| WO2022127180A1 (en) * | 2020-12-17 | 2022-06-23 | 深圳云天励飞技术股份有限公司 | Target tracking method and apparatus, and electronic device and storage medium |
-
2022
- 2022-06-24 CN CN202210730549.0A patent/CN115063453B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112052836A (en) * | 2020-10-09 | 2020-12-08 | 南京农业大学 | System and method for monitoring opening and closing states of plant leaf air holes in real time |
| WO2022127180A1 (en) * | 2020-12-17 | 2022-06-23 | 深圳云天励飞技术股份有限公司 | Target tracking method and apparatus, and electronic device and storage medium |
| CN112950700A (en) * | 2021-02-25 | 2021-06-11 | 安徽农业大学 | Plant leaf stomatal conductance measuring method based on deep learning and image processing |
| CN112801068A (en) * | 2021-04-14 | 2021-05-14 | 广东众聚人工智能科技有限公司 | Video multi-target tracking and segmenting system and method |
Non-Patent Citations (2)
| Title |
|---|
| ZHUANGZHUANG SUN 等: "An Integrated Method for Tracking and Monitoring Stomata Dynamics from Microscope Videos", 《 PLANT PHENOMICS》 * |
| 孙壮壮 等: "单子叶作物叶片气孔自动识别与计数技术", 《农业工程学报》, vol. 35, no. 23 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115063453B (en) | 2023-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Rao et al. | Selfie video based continuous Indian sign language recognition system | |
| CN111223088B (en) | Casting surface defect identification method based on deep convolutional neural network | |
| CN110276264B (en) | Crowd density estimation method based on foreground segmentation graph | |
| CN107633226B (en) | Human body motion tracking feature processing method | |
| CN106022231A (en) | Multi-feature-fusion-based technical method for rapid detection of pedestrian | |
| CN112241762B (en) | Fine-grained identification method for pest and disease damage image classification | |
| CN111783576A (en) | Pedestrian re-identification method based on improved YOLOv3 network and feature fusion | |
| CN108734200B (en) | Human target visual detection method and device based on BING (building information network) features | |
| CN112434599B (en) | Pedestrian re-identification method based on random occlusion recovery of noise channel | |
| CN113763424B (en) | Real-time intelligent target detection method and system based on embedded platform | |
| CN110147833A (en) | Facial image processing method, apparatus, system and readable storage medium storing program for executing | |
| CN111738164B (en) | Pedestrian detection method based on deep learning | |
| CN110599463A (en) | Tongue image detection and positioning algorithm based on lightweight cascade neural network | |
| CN110533100A (en) | A method of CME detection and tracking is carried out based on machine learning | |
| CN102314591B (en) | Method and equipment for detecting static foreground object | |
| CN109448307A (en) | A kind of recognition methods of fire disaster target and device | |
| CN113706481A (en) | Sperm quality detection method, sperm quality detection device, computer equipment and storage medium | |
| CN110363799B (en) | Multi-motion human target tracking method based on vision under man-machine coexistence environment | |
| Feng | Mask RCNN-based single shot multibox detector for gesture recognition in physical education | |
| CN114038011A (en) | Method for detecting abnormal behaviors of human body in indoor scene | |
| CN111640138A (en) | Target tracking method, device, equipment and storage medium | |
| CN109859244B (en) | Visual tracking method based on convolution sparse filtering | |
| Wang et al. | Single shot multibox detector with deconvolutional region magnification procedure | |
| CN110889418A (en) | Gas contour identification method | |
| CN108985216B (en) | Pedestrian head detection method based on multivariate logistic regression feature fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |