WO2021187231A1

WO2021187231A1 - Affected tree diagnosis system, affected tree diagnosis method, and affected tree diagnosis program

Info

Publication number: WO2021187231A1
Application number: PCT/JP2021/009241
Authority: WO
Inventors: 浩史森田; 隆司藤木; 裕介増本; 利晃荒井; 和人内島; 正典村瀬; 俊宏内山
Original assignee: 株式会社電通国際情報サービス; 株式会社ブリヂストン
Priority date: 2020-03-16
Filing date: 2021-03-09
Publication date: 2021-09-23
Also published as: JP2021145557A

Abstract

Provided is a technique which can diagnose a disease affecting a tree with high accuracy. This affected tree diagnosis system is provided with: a subject acquisition unit for acquiring an image of a tree of interest; a diagnosis unit for inputting the image of the tree of interest to a diagnosis model, which has experienced a machine learning for diagnosing a tree having affected leaves as an affected tree from the image of the tree, by employing, as learning data, an image of a tree, information showing an area in which leaves are therein in the image, and information showing whether or not a characteristic property of the affecting disease appears in the area, thereby diagnosing a tree having affected leaves as an affected tree; and a diagnosis result output unit for outputting the results of the diagnosis.

Description

Diseased tree diagnosis system, diseased tree diagnosis method, and diseased tree diagnosis program

The present invention relates to a diseased tree diagnosis system, a diseased tree diagnosis, and a diseased tree diagnosis program.

Conventionally, root rot is known as a disease of Hevea brasiliensis. Root rot kills trees by infecting roots with pathogens and causing tissue rot. If the affected tree is left unattended, it will be transmitted to the surrounding healthy trees, so it is desirable to detect the disease early and deal with it. The diagnosis of this disease was made, for example, by a skilled worker looking at the color and condition of the leaves to make an estimate and digging the roots of a tree suspected of having the disease to determine whether or not it was infected.

Further, in Patent Document 1, a plurality of images of plant diseases and corresponding diagnosis results are captured as learning data, image feature data related to plant diseases is created and held, and an input for inputting an image to be diagnosed is input. A plant disease diagnostic system including a unit and an analysis unit that identifies which diagnostic result the input image is classified by using the deep learning device has been proposed.

Japanese Unexamined Patent Publication No. 2016-168046

In order to detect the disease at an early stage, it is necessary to distinguish a slight lesion, and even an expert is difficult to diagnose, and there are cases where it is not possible to clearly diagnose whether the disease is diseased or healthy. In order to prevent oversight of the disease, it is necessary to diagnose the diseased tree except for the trees that can be clearly judged to be healthy. For this reason, when digging the roots of a tree diagnosed with a disease and checking the root condition, there are many cases where the result is that the roots are healthy, that is, the diagnosis is incorrect, and it takes time and effort to dig and check the roots wastefully. There was a problem that it took. Therefore, it has been desired to improve the accuracy of diagnosis, particularly the accuracy (recall rate) when diagnosing a disease.

Further, in Patent Document 1, an image of a plant disease and a diagnosis result are taken in to create image feature data, and which diagnosis result the input image corresponds to is identified. In this case, if the lesion is clear on the input image, it can be diagnosed, but if the main lesion appears at the root and the lesion on the ground is slight, such as a rubber tree, it should be diagnosed accurately. I couldn't. For example, the leaves of a diseased rubber tree have a stronger yellow tint than that of a healthy one, but young healthy leaves have a similar color and are difficult to diagnose by leaf color. Similarly, the shape of the leaves was not significantly different from that of the healthy ones. For example, even if the leaves of the diseased tree were individually photographed to create image feature data, the disease could not be diagnosed accurately.

Therefore, an object of the present invention is to provide a technique capable of diagnosing a disease of a tree with high accuracy.

In order to solve the above problems, the diseased tree diagnostic system of the present invention is used.
The target acquisition unit that acquires the image of the tree to be diagnosed,
The image of the tree, the information indicating the range in which the leaf group is shown in the image, and the information indicating whether or not the diseased characteristic is present in the range are used as teacher data, and the affected leaf group is included from the image of the tree. A diagnostic unit that diagnoses a tree including a diseased leaf group as a diseased tree by inputting an image of the tree to be diagnosed into a machine-learned diagnostic model for diagnosing a tree as a diseased tree.
It includes a diagnosis result output unit that outputs the result of the diagnosis.

The image of the tree may be a bird's-eye view image taken from above the tree.

In the diseased tree diagnostic system, the leaf group exists between a first height set a predetermined distance below the upper end of the tree and a second height set a predetermined distance below the first height. It may be something to do.

The diseased tree diagnosis system processes at least one of rotation, movement, enlargement, reduction, inversion, color change, noise addition, blurring, high frequency component enhancement, and distortion with respect to the image of the tree to be diagnosed. The image after image processing is used as teacher data different from the image before image processing, and a pre-processing unit for increasing the number of the teacher data is further provided.
The model creation unit may perform machine learning using the increased teacher data.

The diseased tree diagnosis system further includes a position specifying unit that identifies the position where the diseased tree exists based on the position in the image of the tree diagnosed as the diseased tree in the image to be diagnosed. May be good.

When the image to be diagnosed is photographed by an air vehicle flying over the tree, the positioning unit determines the position of the air vehicle when the image to be diagnosed is photographed and the diseased tree. The position where the diseased tree exists may be specified based on the position of the tree diagnosed as having the disease in the image.

In order to solve the above problems, the diseased tree diagnosis method of the present invention is used.
Steps to acquire an image of the tree to be diagnosed,
The image of the tree, the information indicating the range in which the leaf group is shown in the image, and the information indicating whether or not the diseased characteristic is present in the range are used as teacher data, and the affected leaf group is included from the image of the tree. A step of diagnosing a tree including a diseased leaf group as a diseased tree by inputting an image of the tree to be diagnosed into a machine-learned diagnostic model for diagnosing a tree as a diseased tree.
The step of outputting the result of the diagnosis and
Is executed by the computer.

The present invention may be a diseased tree diagnosis program for causing a computer to execute each step of the diseased tree diagnosis method. Further, the present invention may be a storage medium in which the diseased tree diagnosis program can be read by a computer and stored non-temporarily.

In order to solve the above problems, the machine learning device of the present invention
A data acquisition unit that acquires an image of a tree, information indicating a range in which a group of leaves is shown in the image, and information indicating whether or not the area has a disease characteristic as teacher data.
A model generation unit that generates a diagnostic model for diagnosing a tree including a diseased leaf group as a diseased tree from an image of the tree by machine learning using the teacher data.
To be equipped.

The present invention can provide a technique capable of diagnosing a disease of a tree with high accuracy.

It is a schematic block diagram of the diseased tree system which concerns on embodiment. It is a block diagram of an air vehicle. It is a block diagram of a diagnostic apparatus. It is a hardware block diagram of the diagnostic apparatus. It is a block diagram of a machine learning device. It is explanatory drawing of the process which uses image data as teacher data. It is a hardware block diagram of the machine learning device. It is a figure which shows the process which the diagnostic apparatus diagnoses a diseased tree with respect to the image of the tree to be diagnosed. It is a figure which shows an example of the log data. It is explanatory drawing of the process of identifying the position where the diseased tree exists based on the position in the image. It is a figure which shows the example which plotted the position of a diseased tree on a map. It is a figure which shows the SSD network structure. It is a figure which shows the breakdown of a plurality of data sets. It is a figure which shows an example of the diagnosis result.

<First Embodiment>
"Device configuration"
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic configuration diagram of a diseased tree diagnosis system 100 according to the present embodiment. The diseased tree diagnosis system 100 includes a flying object 10, a diagnostic device 20, and a machine learning device 30. The diseased tree diagnosis system 100 takes an image of the tree to be diagnosed from the sky by the flying object 10, and uses a diagnostic model (neural network) created by deep learning to obtain the diseased tree from the image of the tree to be diagnosed. Diagnose. In this diagnostic model, deep learning (machine learning) was performed using a bird's-eye view image of a tree taken from above and information indicating the range in which a group of leaves having a disease characteristic is shown in the image as teacher data. .. As described above, the diseased tree diagnosis system 100 of the present embodiment accurately grasps the characteristics of the disease appearing in the leaf group and diagnoses the diseased tree with high accuracy by performing the diagnosis based on the state of the leaf group photographed from above. I am trying to be able to do it.

In this embodiment, an example of disease diagnosis of Hevea brasiliensis will be described. Hevea brasiliensis root rot is a disease caused by infection of the roots of Hevea brasiliensis root rot (Rigidoporus microporus), which is a type of filamentous fungus. As the disease progresses, the roots rot and the trees die. In addition, as the disease progresses, pathogens infect the roots of surrounding healthy trees and cause the disease, so it is desirable to detect the disease at an early stage and deal with it. However, lesions in the early stages of illness occur mainly in the underground roots, with slight changes in the above-ground part. For example, the leaves turn yellow when the roots cannot absorb enough water. However, this yellowing was slight, and young leaves and leaves with less sunlight may exhibit the same color, so that the disease could not be diagnosed from the leaf color alone. Therefore, the diseased tree diagnosis system 100 of the present embodiment diagnoses the disease based on the image of the leaf group. This makes it possible to make an accurate diagnosis based not only on the color of the leaves but also on the way the leaves are attached, particularly the density of the leaves and the angle of the leaves with respect to the branches. In addition, by using an image taken from above the tree, the leaves shown in this image are in a position where they can be seen from the sky, and those that have turned yellow due to poor sunlight can be excluded, so that the diagnosis can be made accurately.

《Flying body》
FIG. 2 is a configuration diagram of the flying object 10. As shown in FIG. 2, the flying object 10 is a small unmanned aerial vehicle (drone) including a drive unit 11, a control unit 12, a photographing device 13, and a position information acquisition unit 14.

The drive unit 11 obtains buoyancy by rotating the propeller by a motor or an internal combustion engine, and obtains thrust by tilting the rotation axis of the propeller with respect to the vertical axis. The configuration of the drive unit is not limited to this, and other configurations may be used. For example, it may be configured to have a balloon for obtaining buoyancy and a propeller for obtaining thrust.

The control unit 12 receives an operation by the user, controls the drive unit 11 according to the operation, and controls the flight state of the flying object 10. In addition, the control unit 12 controls the photographing device 13 to control operations such as starting and stopping shooting and changing the shooting direction.

The photographing device 13 forms an image of a tree as a diagnosis target on the light receiving surface of the image sensor 13B by the photographing lens 13A, converts the image into an electric signal by the image pickup element 13B, and acquires the data of the photographed image. Further, the photographing device 13 of the present embodiment records the data of the photographed image together with the photographing time.

The position information acquisition unit 14 acquires position information by a satellite positioning system or the like, and records its own position (also referred to as a shooting position) when shooting with the shooting device 13 in association with the shot image. For example, position information (latitude, longitude, etc.) is acquired at a predetermined cycle and recorded as log data together with the acquisition time. As a result, by collating the shooting time of the shot image with the log data, it is possible to determine where the flying object 10 was located at the time of shooting. In addition, the log data may record attitude information indicating in which direction the flying object 10 is facing, together with the position information.

The flying object 10 is not limited to a small unmanned aerial vehicle, but may be a manned airplane, a helicopter, an artificial satellite, or the like.

《Diagnostic device》
FIG. 3 is a configuration diagram of the diagnostic device 20. As shown in FIG. 3, the diagnostic apparatus 20 includes a target acquisition unit 21, a diagnostic unit 22, a diagnosis result output unit 23, and a position specifying unit 24.

The target acquisition unit 21 acquires image data of the tree to be diagnosed from the flying object 10. The acquisition of the image data is connected to the flying object 10 via, for example, a communication cable or a communication network, and is received from the flying object 10 by communication. Further, the flying object 10 may store the image data of the tree to be diagnosed in a storage medium such as a memory card, and the diagnostic device 20 may read the image data from the storage medium and acquire the image data.

The diagnosis unit 22 receives the disease from the image of the tree using the image of the tree, the information indicating the range in which the leaf group is shown in the image, and the information indicating whether or not the disease has a characteristic of the disease in the range as teacher data. By inputting an image of the tree to be diagnosed into a diagnostic model in which machine learning is performed to diagnose a tree containing a diseased leaf group as a diseased tree, a tree including a diseased leaf group is diagnosed as a diseased tree. do.

The diagnosis result output unit 23 outputs the result of the diagnosis by the diagnosis unit 22. The output of the result of this diagnosis is, for example, display on a display device, printing by a printer, storage on a storage medium, transmission to another device, or the like.

The position specifying unit 24 identifies the position where the diseased tree exists based on the position in the image of the tree diagnosed as the diseased tree in the image to be diagnosed. For example, when the image of the diagnosis target is taken by an air vehicle flying over the tree, the position specifying unit determines the position of the air vehicle when the image of the diagnosis target is taken and the position of the air vehicle. Based on the position of the tree diagnosed as the diseased tree in the image, the position where the diseased tree exists is specified.

FIG. 4 is a hardware configuration diagram of the diagnostic device 20. The diagnostic device 20 is a computer having a processor 222, a memory 223, an input / output IF (interface) 224, and a communication IF 225 connected to each other by a connection bus 221. The processor 222 processes the input information and outputs the processing result to control the entire apparatus. The processor 222 is also called a CPU (Central Processing Unit) or an MPU (Micro-processing unit). The processor 222 is not limited to a single processor, and may have a multiprocessor configuration. Further, it may be a multi-core configuration having a plurality of cores in a single chip connected by a single socket.

Memory 223 includes a main storage device and an auxiliary storage device. The main storage device is used as a work area of the processor 222, a storage area for temporarily storing information processed by the processor 222, and a buffer area for communication data. The main storage device is a storage medium for the processor 222 to cache programs and data and expand a work area. The main storage device includes, for example, a RAM (RandomAccessMemory), a ROM (ReadOnlyMemory), and a flash memory. The auxiliary storage device is a storage medium that stores a program executed by the processor 222, data used for information processing, operation setting information, and the like. The auxiliary storage device is, for example, an HDD (Hard-disk Drive), an SSD (Solid State Drive), an EPROM (Erasable Programmable ROM), a flash memory, a USB memory, a memory card, or the like. In addition, the auxiliary storage device stores the image (photographed image) of the tree to be diagnosed received by the target acquisition unit 21 and the diagnosis result.

The input / output IF224 is an interface for inputting / outputting data to / from a device connected to the diagnostic device 20. The input / output IF224 inputs / outputs data to / from a device such as a disk drive, an operation unit, or a display device that reads data from a storage medium such as a CD or DVD. The operation unit is an input unit such as a mouse, a keyboard, and a touch panel into which information for the diagnostic device 20 is input by an operator's operation. The display device is an output unit that displays and outputs information such as processing results to the operator.

The communication IF25 is an interface (communication module) that communicates with other devices via the communication line N, and is also called a CCU (Communication Control Unit). A plurality of components of the diagnostic device 20 shown in FIG. 6 may be provided, or some components may not be provided.

In the diagnostic device 20 of the present embodiment, when the processor 222 executes the application program, the diagnostic device 20 is a processing unit such as a target acquisition unit 21, a diagnostic unit 22, a diagnosis result output unit 23, and a position specifying unit 24. Functions as. That is, the processor 222 can also be used as each processing unit depending on the software to be executed. However, some or all of the above processing units are dedicated LSIs (large scale integration) such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), logic circuits, and others. It may be formed by hardware such as a digital circuit of. Further, an analog circuit may be included in at least a part of each of the above processing units.

《Machine learning device》
FIG. 5 is a configuration diagram of the machine learning device 30. As shown in FIG. 5, the machine learning device 30 has a data acquisition unit 31 and a model generation unit 32.

The data acquisition unit 31 acquires an image of the tree, information indicating the range in which the leaf group is shown in the image, and information indicating whether or not the disease has a characteristic in the range as teacher data.

The model generation unit 32 generates a diagnostic model for diagnosing a tree including a diseased leaf group as a diseased tree from an image of the tree by machine learning using the teacher data.

FIG. 6 is an explanatory diagram of processing using image data as teacher data. In FIG. 6, image 41 is an original image taken by the flying object 10. From this image 41, the area in which one tree is shown is extracted as the tree area 42. In the present embodiment, the tree area 42 is cut out at 800 × 800 pixels to 1500 × 1500 pixels.

Next, using an annotation tool, specify the range 43 in which the leaf groups in these tree regions 42 are shown, enter whether or not the range 43 has the characteristics of illness, and the information indicating these ranges 43 and the information indicating these ranges 43. Metadata 44 is created in which information on whether or not the patient is ill is described in XML format. Then, the data of the tree region 42 and the metadata 44 are used for learning as teacher data. Here, the leaf group is a group of a plurality of leaves, and includes, for example, 5 to 500 leaves. The number of leaves forming a leaf group may be appropriately determined according to the size (trunk diameter, tree height, etc.) and type of the tree, the age of the tree, the target detection accuracy, etc., and may be 10 to 300 or 10 to 50. There may be. The leaf group may include branches that support the leaves. Further, one leaf group may be a set of leaves derived from the same branch or a set of leaves within a predetermined distance from the tip of the branch. Here, the predetermined distance is a value of 30 cm to 3 m, for example, 50 cm, 1 m, 2 m, and the like. As described above, in the present embodiment, the image of the leaf alone is not used as the teacher data, but the image of the leaf group (data of the tree region 42) is used as the teacher data. Therefore, the teacher data of the present embodiment includes information such as the shape of a plurality of leaves, the distance between each leaf (leaf density), and the angle of the leaf with respect to the branch (how the leaves are attached), in addition to the color of the leaves. .. For example, young leaves are soft and hang down as they move away from the branches, so the way the leaves are attached provides information on whether or not the leaves are young. Therefore, in the diagnostic model learned using the teacher data, if the leaves are dripping, it is judged to be healthy because the leaves are young, and if the leaves are not dripping, the leaves are judged to be healthy. It can be determined that there is a high possibility of yellowing. In addition, when the leaves are shrunk due to the disease and the water absorbed by the roots is reduced, the distance between the leaves and the adjacent leaves is wider than in the healthy case, and the proportion of the leaves in the image of the leaf group (leaf density). Will be low. Therefore, the diagnostic model learned using the teacher data can use the leaf density in the image of the leaf group as one of the information for diagnosing the disease.

As described above, in the present embodiment, the image of the tree is learned, but the data used for the learning is not limited to the image of the tree. For example, even if pre-learning is performed using an image set of general objects such as dogs and cats, and based on the pre-learned model, actual learning (transfer learning) is performed using images of trees to create a diagnostic model. good. As a result, a diagnostic model with high detection accuracy can be generated even if the number of tree images is small, and the load of creating teacher data can be suppressed.

In addition, when specifying the range 43 in which the leaf group in the tree area 42 is reflected by using the annotation tool, the first height set to be a predetermined distance below the upper end of the tree and the first height set to be a predetermined distance below the first height are specified. A group of leaves existing between the second height set on the side may be specified. For example, the first height is set at a distance of 50 cm to 5 m, such as 50 cm, 1 m, 3 m, etc., from the upper end of the tree. The second height is set to a distance of 50 cm to 10 m, such as 1 m, 3 m, 5 m, etc., from the first height. Further, the first height may be set to a distance corresponding to 4% to 30% of the tree height, such as 4%, 6%, 20% of the tree height, from the upper end of the tree. Further, the second height may be set to a distance corresponding to 6% to 50% of the tree height, such as 6%, 20%, 30% of the tree height, from the first height. The first height and the second height may be set according to the height (tree height) of the target tree. In the present embodiment, since the target tree is Hevea brasiliensis, the tree height in this case is, for example, 10 m to 20 m. Not limited to this, the target tree may be a tall tree of 5 m or more.

Further, the image of the tree to be diagnosed is subjected to at least one of rotation, movement, enlargement, reduction, inversion, color change, noise addition, blurring, high frequency component enhancement, and distortion, and after image processing. The image of is used as teacher data different from the image before image processing, and the number of teacher data may be increased. As a result, it is possible to generate a diagnostic model with high detection accuracy while suppressing the influence of brightness, image orientation, and the like.

FIG. 7 is a hardware configuration diagram of the machine learning device 30. The machine learning device 30 is a computer having a processor 322, a memory 323, an input / output IF (interface) 324, and a communication IF 325 connected to each other by a connection bus 321. The processor 322 processes the input information and outputs the processing result to control the entire apparatus. The processor 322 is also called a CPU (Central Processing Unit) or an MPU (Micro-processing unit). The processor 322 is not limited to a single processor, and may have a multiprocessor configuration. Further, it may be a multi-core configuration having a plurality of cores in a single chip connected by a single socket.

Memory 323 includes a main storage device and an auxiliary storage device. The main storage device is used as a work area of the processor 322, a storage area for temporarily storing information processed by the processor 322, and a buffer area for communication data. The main storage device is a storage medium for the processor 322 to cache programs and data and expand a work area. The main storage device includes, for example, a RAM (RandomAccessMemory), a ROM (ReadOnlyMemory), and a flash memory. The auxiliary storage device is a storage medium that stores a program executed by the processor 322, data used for information processing, operation setting information, and the like. The auxiliary storage device is, for example, an HDD (Hard-disk Drive), an SSD (Solid State Drive), an EPROM (Erasable Programmable ROM), a flash memory, a USB memory, a memory card, or the like.

The input / output IF324 is an interface for inputting / outputting data to / from a device connected to the machine learning device 30. The input / output IF324 inputs / outputs data to / from a device such as a disk drive, an operation unit, or a display device that reads data from a storage medium such as a CD or DVD. The operation unit is an input unit such as a mouse, a keyboard, and a touch panel into which information for the machine learning device 30 is input by an operator's operation. The display device is an output unit that displays and outputs information such as processing results to the operator.

The communication IF35 is an interface (communication module) that communicates with other devices via the communication line N, and is also called a CCU (Communication Control Unit). A plurality of components of the machine learning device 30 shown in FIG. 7 may be provided, or some components may not be provided.

In the machine learning device 30 of the present embodiment, the processor 322 executes an application program, so that the machine learning device 30 functions as each processing unit such as a data acquisition unit 31 and a model generation unit 32. That is, the processor 322 can also be used as each processing unit depending on the software to be executed. However, some or all of the above processing units are dedicated LSIs (large scale integration) such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), logic circuits, and others. It may be formed by hardware such as a digital circuit of. Further, an analog circuit may be included in at least a part of each of the above processing units.

<< Diagnosis method >>
FIG. 8 is a diagram showing a process in which the diagnostic device 20 diagnoses a diseased tree with respect to an image of a tree to be diagnosed. The diagnostic apparatus 20 executes the process of FIG. 8 when the start of diagnosis is instructed.

In step S10, the diagnostic device 20 acquires an image of the tree to be diagnosed.

In step S20, the diagnostic device 20 diagnoses the tree including the affected leaf group as a diseased tree by inputting the image of the tree acquired in step S10 into the diagnostic model generated by the machine learning device 30. , Diagnose a tree that does not contain the affected leaf group as a healthy tree.

In step S30, the diagnostic device 20 identifies the position where the diseased tree exists based on the position in the image of the tree diagnosed as the diseased tree in the image to be diagnosed. For example, the diagnostic device 20 refers to the log data and is based on the position and attitude of the flying object 10 when the image to be diagnosed is taken, and the position of the tree diagnosed as the diseased tree in the image. , Identify the location of the diseased tree.

FIG. 9 is a diagram showing an example of log data, and FIG. 10 is an explanatory diagram of a process for identifying the position where the diseased tree exists based on the position in the image.

As shown in FIG. 9, the flying object 10 has the position information 45 of the flying object 10, the time information 46 indicating the time when the position information is acquired, and the attitude information 47 of the flying object 10 at the time. There is. The position information 45 is, for example, information indicating a position using latitude and longitude (longitude / latitude) and altitude in the world geodetic system. The attitude information 47 is based on, for example, a state in which the flying object 10 is horizontal and faces a predetermined direction (for example, north), and the inclination from this reference state is indicated by rotation angles in the pitch direction, the roll direction, and the yaw direction. The value.

The image 51 shown in FIG. 10 was taken by the photographing device 13, for example, the center of the image 51 is the origin 52, the X-axis passing through the origin 52 corresponds to the front-rear axis of the flying object 10, and the Y-axis is It corresponds to the left and right axes of the flying object 10. When the flying object 10 arranges the photographing lens directly downward and the flying object 10 flies in a horizontal state, the object reflected in the center of the image 51 exists at the same latitude and longitude as the flying object 10.

Further, in FIG. 10, the object 53 is shown in the image 51 at a position separated from the origin 52 by a distance Lx in the X-axis direction and a distance Ly in the Y-axis direction. On the ground surface where trees exist, the distances corresponding to the distances Lx and Ly on the image 51 are obtained, for example, by the angle of view of the photographing lens and the altitude of the flying object 10 (height from the ground to the flying object). Then, a position separated from the position of the flying object 10 by a distance corresponding to the distance Lx in the direction corresponding to the X-axis and a distance corresponding to the distance Lx in the direction corresponding to the X-axis is defined as the position where the object 53 exists. You can ask. When the attitude of the flying object 10 is tilted from the reference state, the positional relationship between the position on the image and the position on the ground changes according to the tilt. Therefore, the position on the screen is changed according to the attitude information 47. Find the position on the ground corresponding to the position.

In step S40, the diagnostic device 20 outputs the position of the diseased tree obtained in step S30 as the result of the diagnosis. FIG. 11 is a diagram showing an example in which the positions of affected trees are plotted on a map. In FIG. 11, white "D", "M", and "L" indicate the diseased tree. In addition, "D" is severe, "M" is moderate, and "L" is mildly affected tree.

《Single Shot MultiBox Detector》
The diagnostic unit 22 of the present embodiment has adopted the method proposed in SSD: Single Shot MultiBox Detector (2016), which is a paper by Wei Liu et al., As a method for detecting the characteristics of the disease from the photographed image. FIG. 12 is a diagram showing an SSD network structure. This technique is based on a feed-forward convolutional network, which produces a collection of fixed-length size bounding boxes, scores the existence of object class entities within those boxes, and finally. Non-maximum suppression (NMS, for example, comparing the values of neighboring pixels with the value of the pixel of interest and setting it to 0 if the pixel of interest is not the maximum value) is performed to generate a normal detection.

The first layer of the network is based on the standard architecture used for image classification, which is the base network. The VGG-16 network was used as the base network, but it is not limited to this.

Added a convolution feature layer at the end of the base network. These layers significantly reduce the size and allow prediction of detection on a multi-scale scale. The convolution model for predicting detection is different for each feature layer.

Each added feature layer (or an existing feature layer in the base network) can generate a fixed set of detection predictions using a set of convolution filters. These are shown at the top of the SSD network structure in FIG. For feature layers of size mxn with p-channels, the basic element for predicting potential detection parameters is a small kernel of 3x3xp, which is relative to the category score, or the coordinates of the default box. Generate all shape offsets. The kernel produces an output value at each position of m × n to which it applies. The output value of the bounding box offset is measured against the position of the default box with respect to the position of each feature map.

In this example, at the top of the network, each feature map cell and a set of default bounding boxes are associated for multiple feature maps. The default boxes are convolved to fill the feature map tightly so that the position of each default box is fixed relative to its corresponding cell. In each feature map cell, predict the offset to the shape of the default box in the cell, as well as the per-class score indicating that an instance of the class exists in each of those boxes. Specifically, for each of the k boxes at a certain position, c class scores and four offsets with respect to the shape of the original default box are calculated. As a result, the total number of filters applied around each position in the feature map is (c + 4) k, producing (c + 4) kmn outputs for the m × n feature map. The default box in this example applies to several feature maps with different resolutions. Using different default box shapes in some feature maps can efficiently discretize the output box shape. Also, in this example, the default box is matched with all correct boxes whose jaccard overlap is higher than the threshold (0.5). This simplifies the learning problem rather than having only one default box selected.

In the SSD training of this example, x ^p _ij = {1, 0} indicates the matching of the i-th default box and the j-th correct answer box in the category p. In the above matching strategy, Σ _i x ^p _ij ≧ 1 can be obtained. In this case, the overall objective error function (loss function) is a weighted sum of the misalignment error (loc) and the classification error (conf) as in Equation A of FIG.

Here, N is the number of matching default boxes. When N = 0, the error is set to 0. The misalignment error is the Smooth L1 error between the parameters of the predicted box (l) and the correct answer box (g) (Equation B). Then, the offset is regressed with respect to the center (cx, xy) of the default bounding box (d) and its width (w) and height (h). The classification error is the softmax error for the certainty (c) of multiple classes (Equation C). In the next evaluation, the weight term α was set to 1 by cross-validation.

<< Evaluation of diagnostic results >>
Multiple datasets were evaluated to confirm the effect of differences in clones and tree age on diagnostic results. FIG. 13 is a diagram showing a breakdown of a plurality of data sets. In the example of FIG. 13, data of clones P2 to P3 and young to old trees were used. Of these, in the data set (i), the data of the clone P2 and the block alone as the middle age of the tree were used, and in the data set (ii), the data of the clone P3 and the different ages of the tree from young to old were used. In addition, in the data set (iii), data of clones P2 to P4 and different clones of middle age were used. Then, in the dataset (iv), the entire data was used, and in the dataset (v), the data of the seriously affected trees were used.

FIG. 14 is a diagram showing an example of the diagnosis result. In FIG. 14, H1 to H6 and D1 to D6 are tree regions extracted from the captured image by the diagnostic unit 22. In the tree regions H1 to H6 and D1 to D6, the portion labeled with the rectangular frame is an image of the leaf group selected by the diagnostic unit 22. The diagnostic unit 22 of the present embodiment shows the selected leaf group having the characteristic of disease (hereinafter, also referred to as the diseased leaf group) in red, and in FIG. 14, the diseased leaf group is shown. The label is shaded with diagonal lines. In addition, the diagnostic unit 22 showed the frame and label of the selected leaf group of healthy ones having no characteristic of disease (hereinafter, also referred to as healthy leaf group) in blue. In FIG. 14, FIG. 14 shows the healthy leaf group of this healthy leaf group. The label is painted white. In the example of FIG. 14, the diagnostic unit 22 diagnoses the trees reflected in the tree regions H05, D1, D2, D4 to D6 having the diseased leaf group as the diseased tree, and the tree regions H01 to H4, H6, D3 having no diseased leaf group. The tree reflected in is diagnosed as a healthy tree.

In order to evaluate this diagnosis result, when the roots of the trees to be diagnosed were dug and actually confirmed, the trees in the tree areas H1 to H6 were healthy trees, and the trees in the tree areas D1 to D6 were diseased trees. .. That is, the tree region H5 mistakes a healthy tree for a diseased tree, the tree region D3 mistakes a diseased tree for a healthy tree, and the others are correct.

As shown in Table 1, the results of actually confirming whether the results of the diagnosis of "disease" or "healthy" by the diagnosis unit 22 are "disease" or "healthy" are classified as shown in Table 1, and the correct answer rate is used. (Accuracy), precision rate (Precision), and recall rate (Recall) were calculated and evaluated.

Here, TP is a case where the diagnosis result of the diagnosis unit 22 is illness and the confirmation result is illness. FP is a case where the diagnosis result of the diagnosis unit 22 is sick and the confirmation result is sound. FN is a case where the diagnosis result of the diagnosis unit 22 is sound and the confirmation result is illness. TN is a case where the diagnosis result of the diagnosis unit 22 is sound and the confirmation result is sound.

The correct answer rate is the ratio at which the diagnosis result that the diagnosis unit 22 has determined to be "healthy" or "affected" and the actual confirmation result match, and is obtained from the following formula 1.
Correct answer rate = (TP + TN) / (TP + FP + TN + FN) ... Equation 1

The precision rate is the ratio at which the diagnosis result of the diagnosis unit 22 as "affected" is actually "affected", and is obtained from the following formula 2.
Conformity rate = TP / (TP + FP) ・・・ Equation 2

The recall rate is the rate at which what is actually "affected" is diagnosed as "affected" by the diagnostic unit 22, and is calculated from the following formula 3.
Recall rate = TP / (TP + FN) ... Equation 3

Table 2 shows the results of evaluating the accuracy rate (Accuracy), precision rate (Precision), and recall rate (Recall) of the diagnosis results of the diagnosis unit 22 in comparison with the actual confirmation results.

As shown in Table 2, the correct answer rate and recall rate were low in the datasets (ii) of different ages. It is presumed that this is because the morphology on the image is different due to the difference in tree age. For this reason, it is presumed that the correct answer rate also decreases for the dataset (iv) that contains data of different ages.

For the data set (v) with a heavy degree of illness, the accuracy was relatively high, and the recall rate was particularly high. It is presumed that this is because the characteristics of the disease are clearly shown on the image in the case of trees with a heavy degree of disease. However, since the aim is to detect the disease at an early stage, it is desired that such high accuracy can be obtained even for trees with a mild degree of disease.

In the case of a properly managed rubber tree plantation, the morbidity rate is about 1%, so if all the trees are judged to be healthy, the correct answer rate will be 99%, but in this case, all the sick trees will be overlooked. Therefore, there is no point in increasing only the correct answer rate. On the contrary, if all the trees are diagnosed as diseased, the recall rate will be 100% and the diseased tree will not be overlooked, but the correct answer rate and the precision rate will be as low as 1%, and of course there is no point in diagnosing. In particular, if the precision rate is low, when treating or cutting roots of a tree diagnosed as having a disease, it is confirmed that the root is actually healthy when the root is dug up, and the work such as digging up is wasted. Will often be. Therefore, it is desirable that the precision rate and the recall rate can be appropriately increased.

In order to improve the diagnostic accuracy of the diagnostic unit 22, the inventors of the present application repeatedly examined the SSD parameters shown in FIG. 12 to obtain the diagnostic result, and the loss function L (x, c, l, Focused on g). In this loss function, the term α is a parameter for changing the specific weights of the classification error and the misalignment error, and in the diagnosis in Table 2, α = 1. As a result of diligent studies, the inventors of this loss function L (x, c, l, g) under the condition that a plurality of trees are photographed from the sky and the characteristics of the leaf group are detected as in the present embodiment. It was found that the diagnostic accuracy can be improved by reducing the parameter α and increasing the weight of the classification error with respect to the misalignment error.

Table 3 evaluates the diagnostic results when the parameter α is 0.5.

As shown in Table 3, by setting α = 0.5, the accuracy was improved as a whole, and in particular, the reproducibility of the entire data (iv) was improved, and 88% could be achieved. The parameter α is not limited to 0.5, and may be, for example, in the range of 0.25 to 0.75.

As described above, according to the present embodiment, the diseased tree can be diagnosed with high accuracy. Conventionally, when a skilled person visually inspects the above-ground part of Hevea brasiliensis to make a diagnosis, the accuracy is 20% or less. Therefore, the diagnostic device of the present embodiment greatly improves the diagnostic accuracy.
In this embodiment, the disease of Hevea brasiliensis has been described, but the diseased tree diagnosis system of this embodiment is not limited to Hevea brasiliensis, and if it is a disease characterized by disease in the leaf group, other diseases Can also be diagnosed.

Note that the above-described embodiment and modification are merely examples, and are not limited to the above-mentioned configuration, and may be changed within a range not contrary to the gist of the invention. For example, the elements of the above embodiments and modifications may be combined, or some elements may be omitted.

10: Air vehicle 11: Drive unit 12: Control unit 13: Imaging device 13A: Imaging lens 13B: Imaging element 14: Position information acquisition unit 20: Diagnostic device 21: Target acquisition unit 22: Diagnostic unit 23: Diagnosis result output unit 24 : Position identification unit 30: Machine learning device 31: Data acquisition unit 32: Model generation unit 100: Diseased tree diagnosis system

Claims

The target acquisition unit that acquires the image of the tree to be diagnosed,
The image of the tree, the information indicating the range in which the leaf group is shown in the image, and the information indicating whether or not the diseased characteristic is present in the range are used as teacher data, and the affected leaf group is included from the image of the tree. A diagnostic unit that diagnoses a tree including a diseased leaf group as a diseased tree by inputting an image of the tree to be diagnosed into a machine-learned diagnostic model for diagnosing a tree as a diseased tree.
A diagnosis result output unit that outputs the result of the diagnosis and
A diseased tree diagnostic system equipped with.
The diseased tree diagnosis system according to claim 1, wherein the image of the tree is a bird's-eye view image taken from above the tree.
Claim 1 in which the leaf group exists between a first height set a predetermined distance below the upper end of the tree and a second height set a predetermined distance below the first height. Or the diseased tree diagnostic system according to 2.
The diagnostic model performs at least one image processing of rotation, movement, enlargement, reduction, inversion, color change, noise addition, blurring, high frequency component enhancement, and distortion on the image of the tree, and after the image processing. The diseased tree diagnosis system according to any one of claims 1 to 3, wherein machine learning is performed using the teacher data increased by using the image of 1 as different teacher data from the image before image processing. ..
Any of claims 1 to 4 further provided with a position specifying portion for specifying the position where the diseased tree exists based on the position in the image of the tree diagnosed as the diseased tree in the image to be diagnosed. The diseased tree diagnostic system according to item 1.
When the image to be diagnosed is photographed by an air vehicle flying over the tree, the positioning unit determines the position of the air vehicle when the image to be diagnosed is photographed and the diseased tree. The diseased tree diagnosis system according to claim 5, wherein the position where the diseased tree exists is specified based on the position of the tree diagnosed as having the diseased tree in the image.
Steps to acquire an image of the tree to be diagnosed,
The image of the tree, the information indicating the range in which the leaf group is shown in the image, and the information indicating whether or not the diseased characteristic is present in the range are used as teacher data, and the affected leaf group is included from the image of the tree. A step of diagnosing a tree including a diseased leaf group as a diseased tree by inputting an image of the tree to be diagnosed into a machine-learned diagnostic model for diagnosing a tree as a diseased tree.
The step of outputting the result of the diagnosis and
A computer-operated method of diagnosing diseased trees.
Steps to acquire an image of the tree to be diagnosed,
The image of the tree, the information indicating the range in which the leaf group is shown in the image, and the information indicating whether or not the diseased characteristic is present in the range are used as teacher data, and the affected leaf group is included from the image of the tree. A step of diagnosing a tree including a diseased leaf group as a diseased tree by inputting an image of the tree to be diagnosed into a machine-learned diagnostic model for diagnosing a tree as a diseased tree.
The step of outputting the result of the diagnosis and
A diseased tree diagnostic program that allows a computer to run.
A data acquisition unit that acquires an image of a tree, information indicating a range in which a group of leaves is shown in the image, and information indicating whether or not the area has a disease characteristic as teacher data.
A model generation unit that generates a diagnostic model for diagnosing a tree including a diseased leaf group as a diseased tree from an image of the tree by machine learning using the teacher data.
A machine learning device equipped with.