JP2023007457A - Chlorophyll amount learning estimation system, chlorophyll amount estimation model generation system, chlorophyll amount estimation system, chlorophyll amount estimation method, and computer program - Google Patents

Abstract

To allow a measurer to measure the amount of chlorophyll in the entire target plant without using a chlorophyll meter.SOLUTION: Based on chlorophyll amount and color information for each of multiple locations of a first individual that is a particular type of plant and used as a sample, a learning estimation server calculates a model representing the relationship between the chlorophyll amount and color information for the plant (#702). Then, based on an image of a second individual of the plant subject to estimation, and the calculated model, the server estimates the amount of chlorophyll in a part captured in each image element of the image (#704).SELECTED DRAWING: Figure 13

Description

Patent Act Article 30, Paragraph 2 application filed Announcement via telecommunication line Publisher name: MDPI Publication name: Remote Sensing Volume: Volume 13 Issue: Issue 4 Publication date Date: February 13, 2021 Title: A Robust Vegetation Index Based on Different UAV RGB Images to Estimate SPAD Values of Naked Barley Leaves URL: https://www. mdpi. com/2072-4292/13/4/686 https://www. mdpi. com/2072-4292/13/4/686/pdf [Publications, etc.] Publication in publication Publisher name: Society for Ecological Engineering Publication name: Eco-Engineering Volume: Volume 33 Issue: Issue 2 Year of publication Month: April 2021 Article title: Assessment of naked barley leaf SPAD values using RGB values under different growth stages at both the leaf and canopy levels URL: https://www. j stage. jst. go. jp/article/seitaikogaku/33/2/33_31/_article/-char/ja/ [Publications, etc.] Presentation at academic conferences Academic name: Agricultural Information Society 2022 Annual Conference Publication date: May 22, 2022 [Published Materials, etc.] Presentation at an academic conference Name of academic conference: Japan Society for Environmental and Environmental Engineering Online Next Generation Research Presentation Date: November 2, 2021

The present invention relates to a technique for estimating the amount of chlorophyll in plants.

Photosynthetic plants change the amount of chlorophyll in their leaves as they grow. Taking advantage of this property, the amount of chlorophyll is sometimes referred to when determining when to harvest plants such as crops or flowers cultivated in fields and when to fertilize fields.

Chlorophyll content is conventionally measured by a chlorophyllometer. For example, according to the chlorophyll meter described in Non-Patent Document 1, when a leaf is pinched with a thumb-sized clip-shaped head, a part of the pinched leaf is irradiated with light, and SPAD (Soil & Plant Analyzer Development) value based on the reflected light.

Chlorophyll meter "SPAD-502Plus", Web page on Konica Minolta website: https://www.konicaminolta.jp/instruments/products/color/chlorophyll/index.html, Retrieved June 4, 2021

As described above, according to the conventional chlorophyll meter, only the amount of chlorophyll in the portion irradiated with light can be measured. Therefore, measuring the amount of chlorophyll in plants cultivated in vast fields has conventionally required a great deal of time and manpower.

SUMMARY OF THE INVENTION In view of such problems, an object of the present invention is to enable a measurer to measure the chlorophyll content of the entire target plant without using a chlorophyll meter.

A chlorophyll amount learning and estimating system according to one embodiment of the present invention is a specific type of plant and is based on the chlorophyll amount and color information of each of a plurality of positions of a first individual used as a sample. model calculation means for calculating a model representing a relationship with color information; and an image of the second individual which is a plant and is an object of estimation, and based on the model, the portion shown in each pixel of the image. an estimation means for estimating the chlorophyll amount.

According to the present invention, the measurer can measure the chlorophyll content of the whole target plant without using a chlorophyll meter.

It is a figure which shows the example of the whole structure of a chlorophyll amount estimation system. It is a figure which shows the example of the hardware constitutions of a learning estimation server. FIG. 4 is a diagram showing an example of a functional configuration of a learning inference server; It is a figure which shows the example of measured value data. FIG. 4 is a diagram showing an example of intermediate data; FIG. 4 is a diagram showing an example of learning data; It is a figure which shows the example of a linear model function. It is a figure which shows the example of the state of the imaging|photography of the agricultural field by a drone. It is a figure which shows the example of an agricultural field image. FIG. 10 is a diagram showing an example of a partial image; FIG. 4 is a diagram showing an example of the positional relationship of multiple partial images; It is a figure which shows the example of a SPAD value distribution map. 4 is a flowchart for explaining an example of the overall flow of processing by a learning estimation program; 10 is a flowchart for explaining an example of the flow of learning processing; 10 is a flowchart for explaining an example of the flow of estimation processing; It is a figure which shows the example of a linear model function.

FIG. 1 is a diagram showing an example of the overall configuration of a chlorophyll amount estimation system 5. As shown in FIG. FIG. 2 is a diagram showing an example of the hardware configuration of the learning estimation server 1. As shown in FIG. FIG. 3 is a diagram showing an example of the functional configuration of the learning inference server 1. As shown in FIG.

The chlorophyll amount estimation system 5 is a system for estimating the chlorophyll amount of plants cultivated in a field by AI (Artificial Intelligence) based on aerial photographs. client 21, second client 22, drone 31, chlorophyll meter 32, colorimeter 33, communication line 4, and the like.

The learning estimation server 1 , first client 21 and second client 22 can communicate with each other via the communication line 4 . As the communication line 4, the Internet, a public line, a LAN (Local Area Network), or the like is used.

The drone 31 is a small UAV (Unmanned Aerial Vehicle) equipped with a digital camera and a communication function, and photographs a field from above. A commercially available drone is used as the drone 31 .

The chlorophyll meter 32 measures the amount of chlorophyll in plant leaves. A commercially available chlorophyll meter is used as the chlorophyll meter 32 . A case where a chlorophyll meter "SPAD-502 Plus" manufactured by Konica Minolta Co., Ltd. is used will be described below as an example. According to this chlorophyll meter, a SPAD (Soil & Plant Analyzer Development) value is calculated and output as a value representing the amount of chlorophyll.

A colorimeter 33 measures the color of the leaves of the plant. A commercially available color difference meter is used as the colorimeter 33 . A case in which a Konica Minolta color difference meter “CR-400” is used will be described below as an example. According to this color difference meter, the value representing the color is the value of the XYZ color system of CIE (Commission Internationale de l'Eclairage) (hereinafter referred to as "first color value (X, Y, Z)") ) is calculated and output.

The first client 21 is a terminal device for causing the learning estimation server 1 to perform machine learning. The second client 22 is a terminal device for causing the learning estimation server 1 to estimate the amount of chlorophyll in plants cultivated in fields. As the first client 21 and the second client 22, computers having communication functions, such as commercially available personal computers, tablet computers, or smartphones, are used.

The learning estimation server 1 performs machine learning based on instructions from the first client 21 , or estimates the amount of chlorophyll based on instructions from the second client 22 . A so-called server device or a cloud server is used as the learning estimation server 1 . A case where a server device is used as the learning estimation server 1 will be described below as an example.

The learning inference server 1, as shown in FIG. 2, includes a main body 10, a display 11, a keyboard 12, a pointing device 13, and the like. The main body 10 includes a processor 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, an auxiliary storage device 104, a network controller 105, an input/output interface 106, and the like.

A learning estimation program 14 (see FIG. 3) is installed in the ROM 103 or the auxiliary storage device 104 . The learning estimation program 14 is a program for generating data for estimating the amount of chlorophyll by machine learning and estimating the amount of chlorophyll in plants in a field. The learning estimation program 14 is loaded into RAM 102 and executed by processor 101 .

The network controller 105 is a communication device such as a NIC (Network Interface Card), and is used to communicate with the first client 21, the second client 22, and the like.

The input/output interface 106 is an input/output board conforming to standards such as USB (Universal Serial Bus), to which the display 11, the keyboard 12, and the pointing device 13 are connected.

The display 11 displays a screen for inputting commands or information, a screen showing execution results of processing by the processor 101, and the like. A keyboard 12 and pointing device 13 are used to enter commands or information.

By executing the learning estimation program 14 by the processor 101, each function of the machine learning unit 15, the model data storage unit 16, and the estimation unit 17 shown in FIG. 3 is realized. Hereinafter, each function will be described along with how to use the first client 21, the second client 22, the drone 31, the chlorophyll meter 32, and the colorimeter 33, taking the case of estimating the chlorophyll amount of cabbage as an example.

[Machine learning]
FIG. 4 is a diagram showing an example of the measured value data 6A. FIG. 5 is a diagram showing an example of intermediate data 6C. FIG. 6 is a diagram showing an example of learning data 6D. FIG. 7 is a diagram showing an example of the linear model function f ₂ (P).

A worker prepares several individuals of cabbage. These individuals are desirably cultivated in a plurality of fields with different soil conditions. For example, five cabbage plants are prepared from each of six fields with different soil conditions.

Then, the operator measures the SPAD values S and the first colorimetric values (X, Y, Z) at a plurality of positions on the leaves of each of these individuals using the chlorophyll meter 32 and the colorimeter 33 . The SPAD value S is sent from the chlorophyll meter 32 to the first client 21 and the first colorimetric values (X, Y, Z) are sent from the colorimeter 33 to the first client 21 .

As a result, SPAD values S and first colorimetric values (X, Y, Z) for a plurality of positions of each individual (cabbage) are input to the first client 21 as shown in FIG. . The individual identifier is a code for identifying an individual, and the position identifier is a code for identifying the position where the SPAD value S and the first colorimetric values (X, Y, Z) are measured. . Both identifiers are given for convenience and are not used during machine learning.

Hereinafter, the data indicating the SPAD value S and the first colorimetric values (X, Y, Z) measured by the chlorophyll meter 32 and the colorimeter 33 from the same position of the same individual will be referred to as "measurement value data 6A". Describe. When the SPAD value S and the first colorimetric values (X, Y, Z) are measured at N ₂ points each from each of N ₁ individuals, the first client 21 has (N ₁ ×N ₂ ) Measured value data 6A is obtained. Although it is preferable that the number of samples, that is, the number of pieces of measured value data 6A is large, it is preferable to collect the measured value data 6A by evenly measuring from positions of various densities.

Furthermore, the operator inputs the SPAD value S and identification information (for example, plant name) of the plant whose first colorimetric values (X, Y, Z) are measured. In this example, "cabbage" is entered.

Then, the first client 21 accesses a web page for machine learning using a web browser, and converts these measured value data 6A to learning command data 6B indicating machine learning commands and input identification information (plant name). together with the learning estimation server 1.

In the learning estimation server 1, the machine learning unit 15 is composed of a measured value reception unit 151, a color space conversion unit 152, a feature amount calculation unit 153, a regression analysis unit 154, and the like, as shown in FIG.

The measured value reception unit 151 receives the measured value data 6A and the learning command data 6B from the first client 21 via the web page for machine learning.

Then, the color space conversion unit 152 converts the first colorimetric values (X, Y, Z) indicated in each measurement value data 6A to RGB colorimetric system values (hereinafter referred to as “second colorimetric values (R, G, B)”) to generate intermediate data 6C as shown in FIG. As well-known conversion formulas, for example, the following formulas (1) to (7) are used.

R=255×f ₁ (α(R)) (5)
G=255×f ₁ (α(G)) (6)
B=255×f ₁ (α(B)) (7)

The feature quantity calculation unit 153 calculates the feature quantity P is calculated to generate learning data 6D as shown in FIG.
For example, the following formula (8) is used as the cabbage feature amount calculation formula.
P=(GB)/(G+B) … (8)

For example, the feature amount P ₀₁₀₁ of the first learning data 6D is obtained by substituting the second colorimetric values (R ₀₁₀₁ , B ₀₁₀₁ , G ₀₁₀₁ ) shown in the first intermediate data 6C into the equation (8),
P ₀₁₀₁ =(G ₀₁₀₁ -B ₀₁₀₁ )/(G ₀₁₀₁ +B ₀₁₀₁ )
is calculated as

Here, the case where the plant is cabbage is described as an example, but the plant is not limited to crops such as cabbage and barley, and may be ornamental plants such as roses and tulips. In addition, one feature quantity calculation formula can be commonly used for a plurality of types of plants. For example, formula (8) can be used as a feature quantity calculation formula not only for cabbage but also for onions, lettuce, green onions, Chinese chives, and citrus fruits.

However, since onions and green onions have tubular leaves (tubular leaves), it is difficult for light to pass therethrough. Therefore, when measuring the amount of chlorophyll with the chlorophyll meter 32 in collecting learning data, these leaves must be split and thinned. However, in the inference phase, which will be described later, the feature amount P can be calculated and the SPAD value S can be estimated based on the image captured by the digital camera without the chlorophyll meter 32 measuring. there is no need to divide the In addition, even for leaves other than tubular leaves, if the thickness of the leaves makes it difficult for light to pass through, the amount of chlorophyll can be measured by splitting the leaves thinly. However, even in this case, the SPAD value S can be estimated without splitting these leaves in the inference phase, which will be described later.

The regression analysis unit 154 generates a linear model function representing the correlation between the SPAD value S and the feature amount P by performing regression analysis based on the SPAD value S and the feature amount P indicated in each learning data 6D. . At this time, the SPAD value S is used as an objective variable (explained variable), and the feature amount P is used as an explanatory variable.

As a regression analysis method, a known method such as the least squares method is used. In this case, assuming that the correlation between the SPAD value S and the feature amount P is represented by a linear function, the regression analysis unit 154 uses the least squares method to calculate the slope and intercept of the linear function. Generate a linear model function f ₂ (P) as shown in .

Then, the regression analysis unit 154 associates the model data 6E indicating the generated linear model function with the identification information (in this example, the plant name “cabbage”) indicated in the learning instruction data 6B, and stores the model data storage unit 16 in the model data storage unit 16. be memorized.

Model data 6E of cabbage is generated and stored in the model data storage unit 16 by the work of the worker and the processing of each device.

The learning estimation server 1 may check whether or not the generated linear model function is significant by RMSE (Root Mean Squared Error) or NRMSE (Normalized Mean Squared Logarithmic Error). In this case, if the RMSE or NRMSE is equal to or less than a predetermined value, it is determined to be significant, and otherwise it is determined to be non-significant.

For example, the regression analysis unit 154 selects a portion (for example, half) of the learning data 6D for verification. The selected learning data 6D is hereinafter referred to as "verification data". Verification data is not used for generating model data 6E.

The regression analysis unit 154 converts the RGB values (R, G, B) indicated in one piece of verification data into a feature amount P using a feature amount calculation formula. The SPAD value S is calculated by substituting each feature amount P into the linear model function shown in the model data 6E. The RMSE or NRMSE of the calculated SPAD value S and the SPAD value S shown in its verification data is calculated as the error of both SPAD values S. If the error is less than a predetermined value, it is determined as "accepted", and if it is equal to or greater than the predetermined value, it is determined as "failed". Errors are similarly calculated for the rest of the verification data, and pass/fail determination is made. The linear model is determined to be significant if the number of passed tests with respect to the number of verification data is equal to or greater than a predetermined ratio (for example, 95%), and is determined to be insignificant if the ratio is less than the predetermined ratio.

Then, if significant, data representing the linear model function is stored in the model data storage unit 16 as learning data 6D. If it is not significant, the linear model function is discarded and the sample data, that is, the measured value data 6A is increased to regenerate the linear model function.

The model data 6E of plants other than cabbage are also generated by similar work and processing, and stored in the model data storage unit 16 in association with the identification information of the plants.

However, the feature amount calculation formula is used properly for each plant. For example, the following formula (9) is used as the feature value calculation formula for naked barley.
P=GB … (9)

〔Estimated〕
FIG. 8 is a diagram showing an example of how a field 800 is photographed by the drone 31. As shown in FIG. FIG. 9 is a diagram showing an example of the field image 8A. FIG. 10 is a diagram showing an example of a partial image 8C. FIG. 11 is a diagram showing an example of the positional relationship of multiple partial images 8C. FIG. 12 is a diagram showing an example of the SPAD value distribution map 8F.

The user can cause the chlorophyll amount estimation system 5 to estimate the chlorophyll amount of plants cultivated in the field. Hereinafter, the user's work and the processing of each part of each device will be described, taking as an example a case of estimating the overall chlorophyll amount of a large number of individuals of cabbage cultivated in a field 800 shown in FIG.

The user places the color reference card 801 at a predetermined position in the field 800 substantially horizontally. Color reference card 801 is a paper or board such as pure white or 18 percent gray for white balancing of images, commonly referred to as a "white card" or "gray card".

The user flies the drone 31 over the field 800 and makes the drone 31 photograph the field 800 from _a height h1. The height h ₁ is on the order of 50 to 120 meters. An example of a case where a field image 8A, which is an image of the entire field 800 as shown in FIG. 9, is obtained in one shot will be described below. The image of the color reference card 801 is included in the field image 8A as the color reference image 8B. Although FIG. 9 shows a monochrome image as the field image 8A due to limitations of patent drawings, it is actually a color image. FIG. 10 is also the same.

Further, the user causes the drone 31 to photograph the field 800 part by part from a height h2 _{lower than} the height h1. _h2 is on the order of 6 meters. As a result, a partial image 8C, which is an image of a portion of the field 800, as shown in FIG. 10, is obtained. Since the partial image 8C is captured at a lower altitude than the field image 8A, the field 800 is shown more clearly than the field image 8A.

A case where i×j partial images are obtained as the partial image 8C will be described below as an example. Each partial image 8C is described separately as "partial image _8C11 ", "partial image _8C12 ", ..., "partial image _8Cij " as shown in FIG. Each position of the field 800 is shown in one of these partial images 8C. It should be noted that the same portion of the field 800 may somewhat overlap in the adjacent partial images 8C.

The drone 31 transmits the image data of the field image 8A and the image data of each of the partial images 8C to the second client 22 .

Thereby, the second client 22 can obtain the image data of the entire field 800 and the image data of each part of the field 800 . Furthermore, the user inputs the plant name of the plant cultivated in the field 800 to the second client 22 as the identification information of the plant. In this example, the plant name "cabbage" is entered. The second client 22 displays a list of plant type identification information (plant names) corresponding to each of the learning data 6D stored in the model data storage unit 16 of the learning estimation server 1. You may select the identification information from the list.

Then, the second client 22 accesses the web page for the estimation service, and presents the image data of the field image 8A and the image data of the partial image 8C, as well as the instruction for estimation and the input identification information (plant name). The estimation command data 6F is transmitted to the learning estimation server 1.

In the learning estimation server 1, as shown in FIG. 176 and the like.

The image data reception unit 171 receives the image data of the field image 8A, the image data of each of the partial images 8C, and the estimation command data 6F from the second client 22 via the estimation service web page.

Then, the brightness adjustment unit 172 performs white balance processing of the field image 8A as follows.

If the image data of the field image 8A is non-RAW data such as JPEG (Joint Photographic Experts Group), color temperature data is not included, so the color reference image 8B included in the field image 8A is used. Specifically, the brightness adjustment unit 172 corrects the entire field image 8A so that the color of the color reference image 8B becomes the original color (the color of the color reference card 801).

On the other hand, when the image data of the field image 8A is RAW data, the brightness adjustment unit 172 corrects the field image 8A so as to have a predetermined color temperature. Also in this case, as in the case of non-RAW data, the field image 8A may be corrected so that the color of the color reference image 8B becomes the original color.

Hereinafter, the field image 8A corrected by the brightness adjustment unit 172 is referred to as "first corrected image 8D".

By the way, the agricultural field 800 is provided with a passage. In addition, soil or weeds may be visible between adjacent cabbages. Therefore, the field image 8A may include images of paths, soil, and weeds. In addition, cabbage sometimes has worm-eaten. Pathways, soil, and weeds are not part of the cabbage and should be excluded from the estimation of SPAD values. Also, moth-eaten parts should be excluded as they lack chlorophyll. In addition, shadowed areas should be excluded.

Therefore, the noise removal unit 173 removes pixels of portions other than leaves, that is, pixels representing paths, soil, weeds, wormholes, or shadows, from the corrected field image 8A, that is, the first corrected image 8D. Do as follows.

The noise removal unit 173 determines pixels representing leaves, paths, soil, weeds, wormholes, and shadows from the first corrected image 8D. These pixels may be determined, for example, as follows. A large number of pixels representing leaves, paths, soil, weeds, wormholes, and shadows are collected in advance, and a classifier (learned model) for classifying these is generated by machine learning. This classifier is then used to determine whether each pixel of the first corrected image 8D represents leaves, paths, soil, weeds, wormholes, or shadows. At this time, a known algorithm such as MLC (Maximum Likelihood Classifier) is used. ENVI5.5 from Harris Geospatial Solutions is known as software to which MLC is applied. You may discriminate|determine by this software. A classifier created by a system other than the learning estimation server 1 may be used for determination.

Pixels representing any of paths, soil, weeds, wormholes, and shadows are hereinafter referred to as "noise pixels." Then, the noise removal unit 173 removes the determined noise pixels from the first corrected image 8D.

Alternatively, the noise removal unit 173 removes noise pixels from the first corrected image 8D as follows. A noise image (in this example, pixels representing any of path, soil, weeds, worm-eaten, and shadow) is discriminated from the partial image 8C of each portion of the field 800 . The determination method is as described above.

Furthermore, the noise removal unit 173 identifies pixels in the first corrected image 8D that represent the same positions as the noise pixels identified in the partial image 8C. This “position” is a position within the field 800 . In other words, when the determined noise pixel is a pixel representing the position (L _x , _Ly ) in the field 800, the noise removal unit 173 removes the position (L _x , _Ly ) from the first corrected image 8D. ) are identified. It can be said that this pixel is also a noise pixel. Then, the noise removing section 173 removes the identified noise pixels from the first corrected image 8D.

According to this method, noise pixels can be removed more reliably than the method of directly determining and removing noise pixels from the first corrected image 8D without using the partial image 8C. However, information on the correspondence relationship between the pixels of the partial image 8C and the pixels of the first corrected image 8D is required. This information can be obtained by a known method, for example, as follows.

When each of the field image 8A and the partial image 8C is captured by the drone 31, position information such as GPS (Global Positioning System) is obtained. Then, the noise removing section 173 calculates which pixel of the first corrected image 8D each pixel of the partial image 8C corresponds to based on the acquired position information.

Alternatively, the noise removing section 173 reduces the partial image 8C to match the reduced scale (actual length per pixel) of the first corrected image 8D. Then, by matching the reduced partial image 8C and the first corrected image 8D, each pixel of the reduced partial image 8C corresponds to which pixel of the first corrected image 8D.

Hereinafter, the first corrected image 8D from which noise pixels have been removed will be referred to as "second corrected image 8E". If the estimation target is bare wheat, the pixels of the ears may be added to the noise pixels. Alternatively, in the case of apples, oranges, and the like, the pixels of the fruit and the trunk and branches of the tree may be added to the noise pixels.

The SPAD value calculator 174 calculates the SPAD value S for each pixel of the second corrected image 8E as follows, based on the linear model function indicated in the plant name model data 6E indicated in the estimated command data 6F. .

The SPAD value calculation unit 174 calculates a feature amount P for each pixel of the second corrected image 8E by using the feature amount calculation formula of the plant with the pixel value (RGB value) of the plant name. Then, the SPAD value S is calculated by substituting the calculated feature amount P into the linear model function of the plant.

In this example, the plant is cabbage, so the SPAD value calculation unit 174 calculates the feature amount P of each pixel using equation (8). Then, the SPAD value S of each pixel is calculated by substituting the feature amount P of each pixel into the linear model function (see FIG. 7) shown in the cabbage model data 6E. For example, when the RGB values of a certain pixel shown in the second corrected image 8E are (R _a , G _a , B _a ), the characteristic amount P _a of the pixel is (G _a −B _a )/ Calculate (G _a + B _a ). Then, f(P _a ) is calculated as the SPAD value S _a of that pixel.

The distribution map generator 175 generates a distribution map 8F that expresses in grayscale the magnitude of the SPAD value S of each pixel of the second corrected image 8E calculated by the SPAD value calculator 174, as shown in FIG. For example, the upper limit (maximum value) of the SPAD value S is S _MAX , the gradation is 256 (0 to 255), and the number of pixels is U _MAX ×V _MAX . When the map 8F is generated, the pixel value of the pixel at the coordinates (u, v) of the SPAD value distribution map 8F is (255/S _MAX )×S _{(u, v)} . In addition, decimal places may be rounded up, rounded down, or rounded off. u=1,2,...,U _MAX and v=1,2,...,V _MAX .

Alternatively, the distribution map generator 175 may generate a map representing the magnitude of the SPAD value S of each pixel on a color scale as the distribution map 8F. For example, the maximum value (ie, 255) may be represented by 100% red intensity, the intermediate value (127) by 100% green intensity, and the minimum value (0) by 100% blue intensity.

The estimation result transmission unit 176 transmits the estimation result data 6G indicating the SPAD value S of each pixel of the second corrected image 8E to the second client 22 together with the image data of the SPAD value distribution map 8F.

When the second client 22 receives the estimation result data 6G and the image data of the distribution map 8F from the learning estimation server 1, it displays the SPAD value of each pixel shown in the estimation result data 6G. Alternatively, the distribution map 8F is displayed.

FIG. 13 is a flowchart for explaining an example of the overall flow of processing by the learning estimation program 14. As shown in FIG. FIG. 14 is a flowchart illustrating an example of the flow of learning processing. FIG. 15 is a flowchart illustrating an example of the flow of estimation processing.

Next, the overall processing flow of the learning estimation server 1 will be described with reference to a flowchart. The learning estimation server 1 performs processing according to the procedure shown in FIG. 13 according to the learning estimation program 14 .

Upon receiving the learning command data 6B together with the measured value data 6A (Yes in #701 of FIG. 13), the learning estimation server 1 performs machine learning according to the procedure shown in FIG. 14 (#702).

The learning estimation server 1 converts the XYZ values (X, Y, Z) indicated in each measurement value data 6A into RGB values (R, G, B) (#711 in FIG. 14), and converts them into the learning command data 6B. A feature amount P is calculated by substituting the RGB values (R, G, B) into the plant feature amount calculation formula of the identification information shown (#712).

Then, the learning estimation server 1 generates a linear model function indicating the relationship between the feature amount P and the SPAD value S by regression analysis (#713), and uses the data indicating the generated linear model function as the identification information of the plant. It is saved as model data 6E in correspondence (#714).

Returning to FIG. 13, the learning estimation server 1 receives the estimation command data 6F together with the image data of the field image 8A (see FIG. 9) and the image data of the partial image 8C (see FIG. 10) (Yes in #703). , estimation (inference) processing is performed according to the procedure shown in FIG. 15 (#704).

The learning estimation server 1 generates the first corrected image 8D by adjusting the brightness of the field image 8A based on the reference (#721 in FIG. 15). A second corrected image 8E is generated by removing noise pixels from the first corrected image 8D (#722). By substituting the pixel value (RGB value) of each pixel of the second corrected image 8E into the linear model function of the model data 6E corresponding to the identification information indicated in the estimated command data 6F, the feature amount P of each pixel is obtained. Calculate (#723). An image, that is, a distribution map 8F (see FIG. 12) representing the magnitude of the SPAD value S of each pixel in grayscale is generated (#724).

Then, the learning estimation server 1 returns (transmits) the estimation result data 6G indicating the feature amount P of each pixel of the second corrected image 8E to the transmission source of the estimation command data 6F together with the image data of the estimation command data 6F. (#725).

Returning to FIG. 13, while the learning estimation server 1 is providing the service of estimating the amount of chlorophyll (SPAD value) (Yes in #705), the processing of step #702 is performed each time learning command data 6B is received. , and the process of step #704 is executed each time the estimated command data 6F is received.

According to this embodiment, the user (measurer) can measure the chlorophyll amount (SPAD value S) of the entire target plant without using a chlorophyll meter. Moreover, since the SPAD value S is estimated based on the color temperature of the image of the object to be measured, it is less likely to be affected by conditions such as the season, time of day, and weather, and can be stably estimated.

In the present embodiment, the feature amount calculation unit 153 calculates (GB)/(G+B) as the feature amount P of cabbage as shown on the right side of the equation (8). In addition, GB was calculated as the feature amount P of naked barley, as shown on the right side of equation (9). In this way, both feature amounts P are calculated based on GB.

The feature amount P of bare barley may be calculated by the following equation (10).
P=2G-R-B (10)

The right side of equation (10) is transformed into (GB)+(GR). In other words, the feature amount P is calculated based on GB also when using formula (10).

FIG. 16 is a diagram showing an example of the linear model function f ₃ (P). Depending on the type of plant, the feature quantity is calculated by formulas other than the formulas (8) to (10) to generate the linear model function. The feature amount calculator 153 and the regression analysis unit 154 generate, for example, a tomato leaf linear model function f ₃ (P) as follows.

The operator measures the SPAD values S and the first colorimetric values (X, Y, Z) at a plurality of positions on the leaves of each of the plurality of tomato plants using the chlorophyll meter 32 and the colorimeter 33 .

The feature amount calculation unit 153 calculates the feature amount P for each position using the equation (11) based on the measured first colorimetric values (X, Y, Z).
P=xy (11)
However, x=X/(X+Y+Z), and y=Y/(X+Y+Z).

That is, the feature amount P is calculated based on the color values x and y of the Yxy color system. As a result, the combination data of the feature amount P and the SPAD value S for each position, which corresponds to the learning data 6D shown in FIG. 6, is obtained.

Regression analysis unit 154 generates a linear model function representing the correlation between SPAD value S and feature amount P by performing regression analysis based on SPAD value S and feature amount P, as in the case of cabbage. . Then, the linear model function f ₃ (P) as shown in equation (12) or FIG. 16 is obtained.
f ₃ (P)=aP+b (12)

a is a negative value, -809.33 in the example of FIG. b is a positive value, 150.84 in the example of FIG. Since a is a negative value, the SPAD value S decreases as the product of x and y, that is, the feature amount P increases.

Then, the estimation unit 17 estimates the SPAD value S of the tomato based on the linear model function f ₃ (P).

The learning estimation server 1 may calculate the feature amount P based on the values of the L ^* a ^* b ^* color system. For example, L ^* may be calculated as the feature amount P. Alternatively, a ^* may be calculated as the feature amount P.

In this embodiment, the case where the machine learning unit 15, the model data storage unit 16, and the estimation unit 17 are integrated into the learning estimation server 1 has been described as an example, but they may be distributed among a plurality of devices or systems. For example, the machine learning unit 15 may be realized by a machine learning computer or system, and the model data storage unit 16 and the estimation unit 17 may be realized by an inference computer or system.

In the present embodiment, the estimation unit 17 estimates the SPAD value S based on the image of the plant photographed from above the field 800, but may be estimated based on the image of the plant photographed from the side. For example, when estimating the SPAD value of a leaf in the middle part of a tomato, it is effective to base it on an image taken from the side. It is also effective in estimating the SPAD value of leaves of plants grown in plant factories or greenhouses.

In the present embodiment, the learning and estimation server 1 performs machine learning and estimation for each type such as "cabbage" and "naked barley", but may perform machine learning and estimation according to more specific types. For example, machine learning and inference may be performed for each variety such as "winter cabbage", "spring cabbage", "green ball", and "red cabbage". Similarly, in the case of tomatoes, machine learning and estimation may be performed for each variety such as "large tomato", "medium tomato", and "mini tomato".

Alternatively, the type and another attribute may be combined for machine learning and inference. For example, machine learning and estimation of cabbage may be performed for each property of the soil in the field where it is grown. That is, the learning estimation server 1 generates the model data 6E of cabbage for each property of the soil of the field where the cabbage is grown. Then, the SPAD value S of the cabbage cultivated in the field is estimated using the learning data 6D corresponding to the property of the soil of the field to be estimated.

As described above, according to the present embodiment, it is possible to stably estimate without being affected by conditions such as season, time of day, and weather. In order to further improve performance, one linear model function may be prepared for each condition and used properly. For example, linear model functions for weather conditions such as fine weather, cloudy weather, and rainy weather may be prepared and used separately. Alternatively, a linear model function may be prepared for each time period, such as the time period around sunrise, the time period around noon, and the time period around sunset, and used accordingly.

In the present embodiment, the learning estimation server 1 represents the estimated SPAD value S of each portion (pixel) of the field 800 in a two-dimensional map such as the SPAD value distribution map 8F (see FIG. 12). It may be represented by the original map. For example, it may be represented by a three-dimensional bar graph. In other words, the field 800 may be represented by a vertical×horizontal plane, and the SPAD value S may be represented by a bar having a height corresponding to its size.

The learning estimation server 1 may further notify the second client 22 of the harvest time of the plants cultivated in the field 800, for example, as follows.

The user regularly (for example, every day or every few days) causes the drone 31 to photograph the entire field 800 and each part thereof as described above, and sends the image data of the field image 8A from the second client 22 to the learning estimation server 1. , the image data of the partial image 8C of each part, and the estimated command data 6F. In addition to the functions shown in FIG. 3, the learning estimation server 1 has a time determination unit and a notification unit.

Upon receiving these data, the estimation unit 17 of the learning estimation server 1 generates the second corrected image 8E as described above, and estimates the SPAD value S of each pixel of the second corrected image 8E. The time determination unit determines whether or not the characteristics of these estimated SPAD values S meet predetermined standards corresponding to the plant. Then, when it is determined that the predetermined standard has been reached, the notification unit notifies the second client 22 that it is time to harvest. For example, notification is made when the average value of these SPAD values S exceeds a predetermined value. Alternatively, notification is made when the ratio of the number of pixels with a SPAD value S larger than a predetermined value to the number of all pixels exceeds a predetermined value.

The learning estimation server 1 may determine the timing of fertilization, the timing of seeding, and the like by the timing determination unit and notify the timing by the notification unit in the same manner as the mechanism for notifying the timing of harvesting. Alternatively, the timing determining unit may continue to record the value (for example, average value) indicating the SPAD value S of the entire field 800 to predict the harvesting time, the fertilization time, or the seeding time. The notification unit may then notify the second client 22 of the predicted timing.

Alternatively, the time determination unit may notify other than the second client 22 . For example, the terminal device of the person in charge of the field 800 may be notified. The notification destination may be set according to the work. In other words, notification destinations may be set for each of harvest, fertilization, and sowing, and notifications may be sent to notification destinations according to the work.

In the present embodiment, the learning estimation server 1 generates the second corrected image 8E by removing noise pixels after adjusting the color temperature of the field image 8A. It may be produced by adjusting the temperature.

In this embodiment, the learning estimation server 1 generates a linear model function by linear regression analysis as a model for estimation (inference), but it may generate a trained model by deep learning. Then, the SPAD value S may be estimated by a trained model.

In this embodiment, the learning estimation server 1 estimates the SPAD values S of many individuals cultivated in the field, but may estimate the SPAD values S of one or several individuals. In this case, instead of acquiring the field image 8A with the drone 31, the user may pick up the digital camera and take an image of one or several individuals from the vicinity to acquire an image of the individual. Then, the learning estimation server 1 may estimate the SPAD value S after adjusting the color temperature of this image and removing noise pixels.

In addition, the chlorophyll amount estimation system 5, the learning estimation server 1, the first client 21, the second client 22, the whole or the configuration of each part, the content of the processing, the order of the processing, the configuration of the data, etc. are within the scope of the present invention. can be changed accordingly.

1 Learning estimation server (chlorophyll amount learning estimation system)
15 Machine learning part (model generation system for chlorophyll amount estimation)
151 measurement value reception unit (acquisition means)
152 color space converter (acquisition means)
153 feature quantity calculation unit (model calculation means)
154 Regression Analysis Department (Model Calculation Means)
16 model data storage unit (storage means)
17 Estimation unit (chlorophyll content estimation system)
171 image data reception unit (acquisition means)
172 Brightness adjustment unit (correction image generation means)
173 noise removal unit (corrected image generation means)
174 SPAD value calculation unit (estimation means)
6D learning data (chlorophyll content, color information)
6E model data (model)
8A Field image 8E Second corrected image (image, input image)

Claims (15)

Model calculation means for calculating a model representing the relationship between the chlorophyll content and the color information of the plant based on the chlorophyll content and the color information at each of the plurality of positions of the first individual which is a specific type of plant and is used as a sample. When,
an estimating means for estimating the chlorophyll content of the portion reflected in each pixel of the image based on the image and the model of the second individual that is the plant and is the target of estimation;
A chlorophyll amount learning estimation system characterized by having Acquisition means for acquiring chlorophyll content and color information for each of a plurality of positions of an individual plant of a specific type;
model calculation means for calculating a model representing the relationship between the chlorophyll amount and color information of the plant based on the chlorophyll amount and color information at each of the plurality of positions acquired by the acquisition means;
A model generation system for estimating the amount of chlorophyll, characterized by comprising: The color information is information including the difference between the lightness of green and the lightness of blue.
The model generation system for estimating the amount of chlorophyll according to claim 2. The plant is a plant having chlorophyll,
The model generation system for estimating the amount of chlorophyll according to claim 2 or 3. the plant is cabbage, onion, or barley;
The color information is information containing the ratio of the difference to the sum of the lightness of green and the lightness of blue.
The model generation system for estimating the amount of chlorophyll according to claim 4. the plant is a tomato,
The color information is information containing color values x and y in the Yxy color system.
The model generation system for estimating the amount of chlorophyll according to claim 4. Stores a model representing the relationship between the chlorophyll amount and the color information of the plant, which is calculated based on the chlorophyll amount and the color information at each of a plurality of positions of the first individual which is a specific type of plant and is used as a sample. a storage means;
estimating means for estimating the chlorophyll content of the portion reflected in each pixel of the input image based on the input image that is the image of the second individual that is the plant and is the target of estimation and the model;
A chlorophyll amount estimation system characterized by having Acquisition means for acquiring a field image of a field in which a plurality of the second individuals are cultivated, taken from above;
a corrected image generating means for generating a corrected image by removing noise pixels, which are pixels corresponding to objects other than leaves, from the field image;
has
The estimating means uses the corrected image as the input image, and estimates the amount of chlorophyll in the portion reflected in each pixel of the input image.
The chlorophyll amount estimation system according to claim 7. The corrected image generating means generates an image adjusted according to a color temperature as the corrected image,
The chlorophyll amount estimation system according to claim 8. a determining means for determining whether or not the time for work on the plant has come based on the amount of chlorophyll estimated by the estimating means;
notification means for notifying that the time has come when it is determined that the time has come;
having
The chlorophyll amount estimation system according to any one of claims 7 to 9. Prediction means for predicting the timing of work on the plant based on the amount of chlorophyll estimated by the estimation means;
notification means for notifying the predicted time;
having
The chlorophyll amount estimation system according to any one of claims 7 to 9. the operation is harvesting the plant, fertilizing the plant, or sowing the plant;
The chlorophyll amount estimation system according to claim 10. A computer calculates a model representing the relationship between the chlorophyll content and the color information of the plant based on the chlorophyll content and color information at each of a plurality of positions of the first individual which is a specific type of plant and is used as a sample,
Based on the image of the second individual that is the plant and is the target of estimation and the model, a computer estimates the amount of chlorophyll in the portion reflected in each pixel of the image.
A method for estimating the amount of chlorophyll, characterized by: causing a computer to perform acquisition processing for acquiring chlorophyll content and color information at each of a plurality of positions of an individual of a specific type of plant;
causing the computer to perform a calculation process for calculating a model representing the relationship between the chlorophyll amount and color information of the plant based on the chlorophyll amount and color information at each of the plurality of positions acquired by the acquisition process;
A computer program characterized by: causing a computer to perform acquisition processing for acquiring an input image, which is an image of an individual that is a specific type of plant and is an object to be estimated;
Based on the model representing the relationship between the chlorophyll amount and color information of the plant, which is calculated based on the chlorophyll amount and color information at each of the plurality of positions of the individual that is the plant and is used as a sample, by the acquisition process causing the computer to perform an estimation process for estimating the amount of chlorophyll in the portion captured in each pixel of the acquired input image;
A computer program characterized by:

Images