WO2021229248A1

WO2021229248A1 - Method for detection and classification of biotic - abiotic stress in crops from thermal photographs using artificial intelligence

Info

Publication number: WO2021229248A1
Application number: PCT/GR2021/000024
Authority: WO
Inventors: Georgios FEVGAS
Original assignee: Fevgas Georgios
Priority date: 2020-05-12
Filing date: 2021-04-28
Publication date: 2021-11-18
Also published as: GR1009898B

Abstract

In the grayscale thermal image (Fig. 1), the Otsu method is applied (Fig. 2). Finding the pixels group of the OTSU method representing the crop is done by selecting the corresponding group with the most pixels in shades of green in the aligned digital photo (Fig. 3). From the metadata of the thermal photography of each pixel of the selected category, an array is constructed where it contains coordinates and temperatures, and k-Means clustering is applied to group the temperatures. Each pixel, depending on the group it belongs to, assigned a specific color to it (Fig. 4). Then, the coloring and the coordinates of the pixels representing the highest temperatures are selected. The pseudo-coloring is placed on the corresponding (same coordinates) pixels of the digital photo (Fig. 5). The digital pseudo-color photo is powered to a trained Convolutional Neural Network, which performs classification of stressed sub-area within the crop with an outline and verbal visualization (Fig. 6, C1 stress).

Description

DESCRIPTION

METHOD FOR DETECTION AND CLASSIFICATION OF BIOTIC- ABIOTIC STRESS IN CROPS FROM THERMAL PHOTOGRAPHS USING ARTIFICIAL INTELLIGENCE.

The present invention relates to a method for detection and classification of biotic - abiotic stress in crops from thermal photographs using Artificial Intelligence and in particular to an automated processing method of thermal photographs and classification using Convolutional Neural Networks - CNNs in order to construction digital photographs of standardized pseudo-color of the crop with demarcated contours of the sub-areas with a verbal representation of the classification of stress.

Due to the rapid technological development in recent years, various sensors have been developed that can be used to collect data in crops. These are digital camera (RGB), spectral sensor, Light Detection and Ranging (LiDAR), and thermal camera. Some of their applications are plant height control, biomass, Leaf Area Index (LAI), and other physiological characteristics (Rahaman et al., 2015. Advanced phenotyping and phenotype data analysis for the study of plant growth and development. Front. Plant Sci. 6.; Zhang and Kovacs J.M., 2012. The application of small unmanned aerial systems for precision agriculture: a review. Precis. Agric. 13, 693-712).

The use of digital cameras (RGB) is more common compared to other sensors because they have low cost, lightweight, and simple data processing. Their disadvantages are low radiometric resolution and lack of proper calibration (Ballesteros et al., 2014. Applications of georeferenced high-resolution images obtained with unmanned aerial vehicles. Part I: Description of image acquisition and processing. Precis. Agric. 15, 579-592.; Bendig et al., 2014. Estimating Biomass of Barley Using Crop Surface Models (CSMs) Derived from UAV-Based RGB Imaging. Remote Sens. 6, 10395-10412.). They can be used in the rapid acquisition of color photographs to calculate the height of the crop, the LAI, and the color of the leaves so that through already developed algorithms with the method of photo processing, the damaged "dry" leaves can be detected. However, this method lags in obtaining phenotype information and crop characteristics due to the lack of a digital camera to capture the invisible spectrum (A1 Hiary et al., 2011a. Fast and Accurate Detection and Classification of Plant Diseases. Int. J. Comput. Appl. 17, 31-38; Singh and Misra, 2017b. Detection of plant leaf diseases using image segmentation and soft computing techniques. Inf. Process. Agric. 4, 41-49.).

Spectral sensors have the ability to receive radiation in the visible and non- visible spectrum so they can be used to obtain the phenotype of a crop (Candiago et al., 2015. Evaluating Multispectral Images and Vegetation Indices for Precision Farming Applications from UAV Images. Remote Sens. 7, 4026-4047;). Their main disadvantages are complex data processing and sensitivity to weather conditions (Colomina and Molina, 2014. Unmanned aerial systems for photogrammetry and remote sensing: A review. ISPRS J. Photogramm. Remote Sens. 92, 79-97; Nasi et al., 2015. Using UAV-Based Photogrammetry and Hyperspectral Imaging for Mapping Bark Beetle Damage at Tree-Level. Remote Sens. 7, 15467-15493; Thorp et al., 2015. Proximal hyperspectral sensing and data analysis approaches for field- based plant phenomics. Comput. Electron. Agric. 118, 225-236; Zarco-Tejada et al., 2013. Spatio-temporal patterns of chlorophyll fluorescence and physiological and structural indices acquired from hyperspectral imagery as compared with carbon fluxes measured with eddy covariance. Remote Sens. Environ. 133, 102-115.).

The Light Detection and Ranging (LiDAR) sensor determining ranges by targeting an object with a laser and uses the photoelectric detection method. It can be used to measure biomass and plant height. Its advantage is the effective acquisition of high precision horizontal and vertical structure of the vegetation. Its disadvantages are the high cost of acquisition and the large amounts of data processing (Ota et al., 2015. Above ground Biomass Estimation Using Structure from Motion Approach with Aerial Photographs in a Seasonal Tropical Forest. Forests 6, 3882-3898.; Wallace etal., 2012. Development of a UAV-LiDAR System with Application to Forest Inventory. Remote Sens. 4, 1519-1543.).

The analysis methodology and control of chlorophyll content, the LAI, and the leaf nitrogen content using remote sensing is fully developed (Ballesteros et al.,

2014. Applications of georeferenced high-resolution images obtained with unmanned aerial vehicles. Part I: Description of image acquisition and processing. Precis. Agric. 15, 579-592.; Bendig et al., 2014. Estimating Biomass of Barley Using Crop Surface Models (CSMs) Derived from UAV -Based RGB Imaging. Remote Sens. 6, 10395-10412.). Therefore, we can have accurate plant growth information because the leaves’ spectral characteristics are directly related to the above indices.

The thermal camera uses infrared radiation. Therefore it can be used to measure the canopy temperature of a crop and the rate of water vapor leave from leaves, and the carbon dioxide (C02) entering in leaves (Baluja et al., 2012. Assessment of vineyard water status variability by thermal and multispectral imagery using an unmanned aerial vehicle (UAV). Irrig. Sci. 30, 511-522.). In this way, it is possible to determine the growth status of the crop indirectly. Disadvantages governing the use of the thermal camera are the sensitivity to weather conditions and the difficulty of eliminating the soil’s effect (Gonzalez-Dugo et al.,

2015. Using High-Resolution Hyperspectral and Thermal Airborne Imagery to Assess Physiological Condition in the Context of Wheat Phenotyping. Remote Sens. 7, 13586-13605.; Jones et al., 2009. Thermal infrared imaging of crop canopies for the remote diagnosis and quantification of plant responses to water stress in the field. Funct. Plant Biol. 36, 978.; Sugiura et al., 2007. Correction of Low-altitude Thermal Images applied to estimating Soil Water Status. Biosyst. Eng. 96, 301-313.).

The treatment method applicable to a thermal image of a crop (in the whole area of cultivation, and not at each plant separately) is the Crop Water Stress Index - CWSI, which determines the water availability of a crop through infrared temperature measurements (thermal photography) of the plant crown. The plant’s crown’s temperature is an indicator of the culture's aqueous condition because of the foliage's stomata closing response to the water's depletion, causing a decrease in transpiration and increasing the temperature of the leaf. On the contrary, the adequacy of water in the soil keeps the stomata open and a strong respiration rate, resulting in a decrease in the foliage's temperature compared to the atmospheric temperature above the crop (Idso et ah, 1981. Normalizing the stress-degree-day parameter for environmental variability. Agric. Meteorol. 24, 45-55.). Finally, plants with biotic or abiotic stress have been shown to exhibit higher canopy temperatures than healthy plants (Zarco- Tejada et al., 2012. Fluorescence, temperature and narrow-band indices acquired from a UAV platform for water stress detection using a micro-hyperspectral imager and a thermal camera. Remote Sens. Environ. 117, 322-337.; Zarco-Tejada et al., 2018. Previsual symptoms of Xylella fasti diosa infection revealed in spectral plant- trait alterations. 7, 432-439.).

Considering the above, a method was found to find biotic-abiotic stress in crops from thermal photographs using Artificial Intelligence and, in particular using, Convolutional Neural Networks. The process of automated thermal photo processing is initially performed by converting the thermal photo (pseudo-coloring) to grayscale (Fig. 1). Then the OTSU method is used to isolate the background, i.e., the soil (Fig. 2). The OTSU algorithm assumes that the image contains two classes of pixels that follow the bimodal histogram (foreground pixels and background pixels), then calculates the optimal threshold separating the two classes so that their combined spread (minimum variability) is minimal or equivalent (because the sum of the pairs of square distances is constant) so that their variability is maximal.

Finding the class of pixels of the OTSU method representing the crop is done by comparing the OTSU method’s two classes of pixels with the corresponding (same coordinates) pixels of the aligned digital photograph (Fig. 3). More specifically, a comparison is made between the two classes of pixels of the aligned digital photo (RGB) which has been converted to HSV color model (Hue, Saturation, Value), in order to find the class that has the more pixels in shades of green. After finding the class with the most pixels with shades of green, the corresponding class of pixels of thermal photography is selected. Then, export the metadata of the thermal image which is the temperature of each pixel of the class selected by the above procedure. In this way, a table is made which contains the coordinates and the temperature of each pixel.

Next, the k-Means clustering method is applied to the data in the table above. The k-Means clustering method groups the pixel temperatures into a specific number of groups. Clustering is the process of organizing patterns (temperatures) into groups, where the group members are similar. Depending on the group you belong to, you assign a specific color to each pixel (e.g., the pixel belonging to the group with the lowest temperature, you assign the dark green). The purpose of using standard pseudo-coloring in crop pixels is to normalize the temperatures of any thermal photograph depicting crop or crops.

Using the coordinates and the coloring of each pixel, a new photo is created. The new normalized photo shows only the crop's temperature fluctuation with standard pseudo-coloring (Fig. 4). Then, the coloring and the coordinates of the pixels representing the highest temperatures are selected. The pseudo-coloring is placed on the corresponding (same coordinates) pixels of the digital photo (Fig. 5). The pseudo-coloring digital photo is then fed to a trained Convolutional Neural Network, which performs stressed sub-areas classification within the crop with a specific delimitation of the stressed region with contour.

The final result is the digital photograph of standardized pseudo-coloring of the crop with delimited outlines of the sub-areas with a verbal representation of stress classification (Fig. 6, Cl stress).

Claims

1. Method for detection and classification of biotic-abiotic stress in crops with automated thermal photographs processing using Convolutional Neural Networks - CNNs which includes the following phases: a. conversion of thermal photography (pseudo-coloring) to Grayscale

(Fig- 1). b. Using the OTSU method to isolate the background, which is the soil (Fig. 2). The algorithm OTSU assumes that the image contains two classes of pixels following a bimodal histogram (foreground pixels and background pixels). c. Finding the class of pixels of the OTSU method representing the crop is done by comparing the OTSU method’s two classes of pixels with the corresponding (same coordinates) pixels of the aligned digital photograph (Fig. 3). More specifically, a comparison is made between the two classes of pixels of the aligned digital photo (RGB) converted to HSV color model (Hue, Saturation, Value), to find the class that has the more pixels in shades of green. After finding the class with the most pixels with shades of green, the corresponding class of thermal photography pixels is selected. d. exporting the metadata of the thermal image, which is the temperature of each class’s pixel selected by the above procedure. e. construction of an array that contains the coordinates and the temperature of each pixel. f. application of the k-means clustering method in the data of the above array. The k-means clustering method groups the pixel temperatures into a specific number of groups. Clustering is the process of grouping similar objects (temperatures) into different groups, or more precisely, the partitioning of a data set into subsets so that the data in each subset according to some defined distance measure. g. assigning a specific color to each pixel depending on the group it belongs to; for example, the pixel that belongs to the group with the lowest temperature assigns the dark green. The purpose of using standard pseudo-coloring in crop pixels is to normalize the temperatures of any thermal photograph depicting crop or crops. h. constructing a new photo using the coordinates and coloring of each pixel. The new normalized photo (Fig. 4) depicts only the crop with standard pseudo-coloring. i. selection of the coloring and the coordinates of the pixels representing the highest temperatures and places the pseudo-color on the corresponding (same coordinates) pixels of the digital photograph (Fig.

5)· j. feeding of digital pseudo-coloring photo in a trained Convolutional Neural Network. k. classification stressed sub-areas within the crop with a specific delimitation of the stressed region with contour. l. illustration of the digital photograph of a standardized pseudo-color of the crop with demarcated contours of the sub-areas with a verbal representation of stress classification (Fig. 6, Cl stress).

2. Method for detection and classification of biotic-abiotic stress in crops with automated thermal photographs processing using Convolutional Neural Networks - CNNs according to claim 1 wherein the crop is a vineyard.

3. Method for detection and classification of biotic-abiotic stress in crops with automated thermal photographs processing using Convolutional Neural Networks - CNNs according to claim 1 wherein the crop is treelike.

4. Method for detection and classification of biotic-abiotic stress in crops with automated thermal photographs processing using Convolutional Neural Networks - CNNs according to claim 1 wherein the crop is an industrial crop.

5. Method for detection and classification of biotic-abiotic stress in crops with automated thermal photographs processing using Convolutional Neural Networks - CNNs according to claim 1 wherein the crop is vegetables.