RU2805670C2 - Detection of plants diseases by classification of plants photography - Google Patents

Abstract

FIELD: agriculture.

SUBSTANCE: invention relates to a method, system, and machine-readable long-term storage medium for recognizing plant diseases from image data using a convolutional neural network. In the method, the processor receives a set of photographs of plants infected with a plurality of diseases, wherein the set of photographs shows leaves with a plurality of marked areas having several aspect ratios, each marked area being associated with a disease label from a plurality of diseases and showing at least one damage caused by the disease, a specific photograph of the set of photographs shows a specific sheet having a specific marked area with multiple damage, the total size of the multiple damage exceeds a predetermined percentage of the size of the specific marked area; determination by the processor of a group of anchor frames from a set of marked areas for each of a number of convolutional layers of a single-stage multibox detector (SSD), moreover: The SSD is configured to receive an image and assign to each of one or more image areas at least one class from a plurality of classes corresponding to a plurality of diseases, a group of anchor frames having different aspect ratios and corresponding to different features of the plurality of classes; moreover, the definition of the group of anchor frames further includes: splitting, combining or deleting one or more of the plurality of marked areas to create a new set of marked areas in accordance with the restriction on the size of the marked area, on the ratio of the size of the cluster of damage within the marked area and the sheet, or on the density of damage within the marked area based on a predetermined percentage, wherein the determination is further carried out on a new set of marked areas; the method further comprising linking each region of the plurality of marked regions to one of the anchor frame groups; and training the SSD using the anchor frame group, the set of photographs having the marked area set, the corresponding mark set, and the corresponding anchor set.

EFFECT: improved accuracy of image recognition.

13 cl, 10 dwg

Description

Copyright notice

Portions of the specification of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of a patent document or patent specification as it appears in the patent file or the records of the Patent and Trademark Office, but otherwise retains all copyrights or rights whatsoever. were. © 2015-2019 Climate Protection Corporation.

Application area

The present invention relates to the technical fields of plant disease detection and machine learning. The present invention also relates to the technical field of improving the configuration and training of machine learning models for plant disease recognition.

State of the art

The methods described in this section represent methods that could be applied, but not necessarily methods that have been previously developed or applied. Therefore, unless otherwise noted, it should not be assumed that any method described in this section qualifies as prior art simply by virtue of its inclusion in this section.

Detection of plant diseases is important in agriculture. Today, an automated approach often involves classifying plant photographs, which can be implemented by using a convolutional neural network (CNN) with many convolutional layers. Some CNNs work together through a two-step approach, where the first CNN typically suggests regions of interest within the image data, and the second CNN then classifies each proposed region. Some CNNs require all images to be a fixed size. There are CNNs that can take pictures of different sizes and suggest and classify regions of interest in a single photo with high quality. Considering the various symptoms of plant diseases, it would be useful to specifically configure and train such a CNN to classify plant photographs and identify plant disease infection to promote plant health and growth.

Brief description of the invention

The attached claims may serve as a brief description of the invention.

Brief description of drawings

The graphic materials show the following.

Figure 1 shows an example of an automated system designed to perform the functions described herein, shown surrounded by other devices with which the system can interact.

Figure 2 shows two examples of the logical organization of sets of instructions in RAM when loading to execute an example mobile application.

Figure 3 illustrates a programmable process by which an automated agrointelligence system creates one or more pre-configured agronomic models using agronomic data provided by one or more data sources.

FIG. 4 is a block diagram illustrating an automated system on which an embodiment of the invention may be implemented.

Figure 5 shows an example of displaying a sequential work schedule for data entry.

Figure 6 shows an example of a spreadsheet for data entry.

Figure 7A shows a photograph of corn leaves, each of which has symptoms of one disease.

Figure 7B shows an example of a photograph of a corn leaf showing symptoms of several diseases.

FIG. 8 shows an exemplary process for applying non-maximum suppression to refine an initial classification result to a final classification result.

Figure 9 shows an example of a method performed by a server that is programmed to configure and apply a convolutional neural network for plant disease detection.

Detailed description

In the following, for purposes of explanation, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it should be obvious that embodiments can be implemented without these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention. Embodiments are disclosed in sections according to the following outline.

1. General overview

2. An example of an automated agrointelligence system

2.1. Structure overview

2.2. Software Application Overview

2.3. Reception of data into an automated system

2.4. Process Overview - Agronomic Model Training

2.5. Implementation example - hardware

3. Descriptions of functions

3.1. Detection of plant diseases

3.2. Digital model configuration

3.3. Sequence of training and construction of a digital model

3.4. Execution of the digital model

3.5. Examples of processes

4. Extensions and alternatives

1. General overview

A system and technological methods for configuring and using a convolutional neural network (CNN) for recognizing plant diseases are disclosed. In some embodiments, the system is programmed to collect photographs of infested plants or leaves that indicate areas of infestation symptoms. Each photo can have multiple marked areas. Depending on the size or grouping of symptomatic areas, one marked area may include only one lesion caused by a single disease, while another may include multiple, closely spaced lesions caused by a single disease. The system is programmed to detect anchor boxes (frames by default) having different aspect ratios, depending on these marked regions, for each convolutional layer of a Single Shot Multibox Detector (SSD). For some plant types, common diseases result in generally many aspect ratios, some of which are quite extreme. The system is programmed to learn SSD using marked areas and anchor boxes and to apply SSD to new photographs to identify diseased plants.

In some embodiments, the system is programmed to collect tagged photographs of corn leaves. Each photo can have one or more marked areas (reference frames). Each marked area may have one or more lesions caused by one of a variety of corn diseases and be labeled with one corn disease. The system can be programmed to further process the marked areas. Specifically, the system can be programmed to split a marked area into multiple marked areas or combine multiple marked areas into one based on how areas corresponding to damage are marked or grouped in the photograph. For example, an expert might note two parts of a large cluster of damage caused by Southern Rust in a photograph. The system can be programmed to merge and expand two parts into one marked region covering the entire cluster because the regions corresponding to dissociated lesions in the cluster are located close to each other and the cluster covers most of the leaf.

In some embodiments, the system is programmed to determine anchor boxes for each of a number of SSD convolutional layers from the resulting labeled regions. See Liu W. et al. (2016) SSD: Single Shot MultiBox Detector in: Leibe V., Matas J., Sebe N., Welling M. (eds.) Computer Vision - ECCV 2016, pp. 21-37. Lecture Notes in Computer Science, volume 9905. Springer, Cham. The system can be programmed to normalize and cluster the marked regions and to compute an aggregate for each cluster to determine the anchor frame. Each anchor box may thus have a different scale and aspect ratio representative of a subset of the marked areas showing symptoms of a common corn disease. For some corn diseases, the aspect ratio may vary from 1/7 to 7.

In some embodiments, the system is programmed to anchor each marked area to an anchor frame based on the shape of the marked area. In particular, the system can be configured to complete a successful snap when the anchor frame alignment can cover more than a predetermined percentage of the marked area. The system is then programmed to train the SSD with images containing marked areas, corresponding labels, anchor boxes and anchors.

In some embodiments, the system is programmed to receive a new image of a corn leaf from a client device and apply an SSD to the new image to obtain an initial identification of multiple regions in the image and a classification into one of the corn diseases for each of the multiple regions with a corresponding confidence score. When the same area is classified into multiple maize diseases, some classifications associated with relatively low confidence scores may be filtered out. The system is programmed to subsequently transmit classification results to the client device.

The system provides all sorts of technical advantages. SSDs have been shown to achieve better performance than similar CNNs, being faster than previous single-stage detectors and also more accurate, in fact as accurate as slower techniques generating accurate region proposals and performing pooling. The system provides an approach to tuning and training SSDs that is particularly suitable for classifying certain types of tasks, such as plant disease symptoms. These symptoms include lesions that are sized and located in a specific pattern, and the approach provided by the system captures this specificity and thus further improves the performance of SSD in classifying plant disease symptoms. This improvement, in turn, leads to improved plant health and growth.

Other aspects and features of the embodiments will become apparent from other sections of the description.

2. An example of an automated agrointelligence system

2.1. Structure overview

Figure 1 is an example of an automated system designed to perform the functions described below, shown in the field with other devices with which the system can interface. In one embodiment, a user 102 owns, operates, or possesses a computing device 104 for managing field operations in a field environment or environment associated with a field environment, such as a field dedicated to agricultural activity or a management location for one or more agricultural fields. The field management computer device 104 is programmed or configured to provide field data 106 to the agrointelligence computer system 130 through one or more networks 109.

Examples of 106 field data include:

a) identification data (e.g. land area, field name, field identifiers, geographical identifiers, boundary identifiers, crop identifiers and any other suitable data that can be used to identify agricultural land, such as community land unit (CLU), number batch and block, land plot number, geographical coordinates and boundaries, farm serial number (FSN), farm number, field number, plot number, settlement and/or pasture),

(b) crop data (e.g. crop type, crop variety, crop rotation, whether the crop is grown organically, harvest date, actual production history (AH), expected yield, yield, product price, crop revenue, grain moisture content , tillage practices and information about previous growing seasons),

(c) soil data (e.g. type, composition, pH, organic matter (OM), cation exchange potential (CEC)),

(d) planting data (e.g. planting date, seed type, relative maturity (RM) of planting material, seed population),

(e) fertilizer data (e.g. type of nutrients (nitrogen, phosphorus, potassium), type of application, date of application, quantity, source, method),

f) chemical application data (e.g. pesticide, herbicide, fungicide, other substance or mixture of substances intended for use as a plant regulator, defoliant or desiccant, date of application, quantity, source, method),

g) irrigation data (e.g. date of application, quantity, source, method),

h) weather data (e.g. precipitation amount, precipitation rate, forecast precipitation, drainage region, temperature, wind, weather forecast, pressure, visibility, cloudiness, heat index, dew point, humidity, snow depth, air quality, sunrise , sunset),

i) survey data (for example, images and light spectrum information obtained from an agricultural equipment sensor, camera, computer, smartphone, tablet, drone, aircraft or satellite),

j) surveillance data for reconnaissance purposes (photographs, videos, free-form recordings, voice recordings), voice transcriptions, weather conditions (temperature, precipitation (current and over time), soil moisture, crop growth stage, wind speed, relative humidity, dew point, black layer) and

k) soil, seeds, crop phenology, pest and disease reports, and forecast sources and databases.

The data server 108 is coupled to the agrointelligence computer system 130 and is programmed or configured to transmit external data 110 to the agrointelligence computer system 130 via a network 109. The external data server 108 may be owned or operated by the same individual or entity as the automated system 130 agrointelligence, or other natural or legal entity, such as a government agency, non-governmental organization (NGO) and/or private data service provider. Examples of external data include, but are not limited to, weather data, imagery, soil data or statistics relating to crop yields. External data 110 may contain the same information as field data 106. In some embodiments, external data 110 is provided by an external data server 108 owned by the same organization that owns and/or operates the automated agrointelligence system 130. For example, automated agricultural intelligence system 130 may include a data server focused solely on the type of data that might otherwise be obtained from third-party sources, such as weather data. In some embodiments, the external data server 108 may actually be included in the system 130.

The farm equipment 111 may include one or more remote sensors 112 that are directly or indirectly connected through the farm equipment 111 to the automated agrointelligence system 130 and are programmed or configured to transmit sensor data to the automated agrointelligence system 130 . Examples of 111 agricultural equipment include tractors, combines, harvesters, seeders, trucks, fertilizer application equipment, aircraft, including unmanned aerial vehicles, and any other piece of physical machinery or equipment, typically mobile machines, that can be used to perform tasks related to agriculture. In some embodiments, one piece of equipment 111 may include a plurality of sensors 112 that are locally networked on that piece of equipment; Controller Area Network (CAN) is an example of such a network that can be installed on combines, harvesters, sprayers and cultivators. Application controller 114 is connected to the automated agricultural intelligence system 130 via network(s) 109 and is programmed or configured to receive one or more scripts that are used to control the operating parameters of the agricultural vehicle or implement from the automated agricultural intelligence system 130. For example, a controller network (CAN) bus interface can be used to provide communication from the automated agrointelligence system 130 to the farm equipment 111 such as the CLIMATE FIELDVIEW DRIVE available from Climate Corporation, San Francisco, California. The sensor data may contain the same information as the field data 106. In some embodiments, the remote sensors 112 may not be attached to the farm equipment 111, but may be remotely located in the field and may communicate over the network 109.

Appliance 111 may include an on-board computer 115 that is programmed with a cockpit application, which may consist of a version or variant of a mobile application for apparatus 104 described in more detail elsewhere herein. In one embodiment, cockpit computer 115 includes a compact computer, often a tablet computer or smartphone, with a graphical display, such as a color display, that is mounted in the operator's compartment of equipment 111. Cockpit computer 115 may perform some or all of the operations and functions described later herein. for mobile computing device 104.

Network(s) 109 is broadly defined as any combination of one or more data networks, including local area networks, broadband networks, or wide area networks, using any of wired or wireless communications links, including terrestrial or satellite links. The network(s) can be implemented by any means or mechanism that allows data exchange between the various elements of Fig. 1. The various elements of FIG. 1 may also have direct (wired or wireless) communication links. Sensors 112, controller 114, computer 108 with external data server, and other system elements, each of which includes an interface compatible with the network(s) 109 and is programmed or configured to use standardized protocols for communication between networks, such as TCP/IP, Bluetooth, CAN protocol and higher level protocols such as HTTP, TLS, etc.

The automated agrointelligence system 130 is programmed or configured to receive field data 106 from the field manager computing device 104, external data 110 from an external computer data server 108, and sensor data from a remote sensor 112. The automated agrointelligence system 130 may be further configured to host, use, or execute one or more computer programs, other software elements, digital programming logic such as an FPGA or ASIC, or any combination thereof to perform translation and storage of data values, build digital models of one or more crops in one or more fields, generate recommendations and notifications, and generating and sending scripts to the fertilizer controller 114, in the manner described further in other sections of the present invention.

In one embodiment, the automated agrointelligence system 130 is programmed with or includes a communication layer 132, a presentation layer 134, a data management layer 140, a hardware/virtualization layer 150, and a model and field data store 160. "Layer" in this context means any combination of electronic digital interface circuits, microcontrollers, embedded software such as drivers and/or computer programs or other software elements.

The communications layer 132 may be programmed or configured to perform input/output interface functions, including sending requests to the field manager computing device 104, the external data server computer 108, and the remote sensor 112 for field data, external data, and sensor data, respectively. Layer 132 may be programmed or configured to transfer the received data to model and field data storage 160 for storage as field data 106.

The presentation layer 134 may be programmed or configured to create a graphical user interface (GUI) that is to be displayed on the field manager computing device 104, on-board computer 115, or other computers that are connected to the system 130 via the network 109. The GUI may include controls entering data to be sent to the automated agrointelligence system 130, generating requests for models and/or recommendations and/or displaying recommendations, notifications, models and other field data.

The data management layer 140 may be programmed or configured to control read and write operations using the storage 160 and other system functional elements, including queries and result sets passed between the system functional elements and the store. Examples of data management layer 140 include, but are not limited to, JDBC, SQL Server interface code, and/or HADOOP interface code. Repository 160 may be a database. In the context of this document, the term "database" can refer to either a body of data, a relational database management system (DBMS), or both types of data. For the purposes of this document, a database can include any set of data, including hierarchical databases, relational databases, flat file databases, object-relational databases, object-oriented databases, distributed databases, and any other structured collections. records or data stored on a computer system. Examples of DBMS include, but are not limited to, ORACLE®, MYSQL, IBM® DB2, MICROSOFT® SQL SERVER, SYBASE®, and POSTGRESQL databases. However, any database that provides the ability to use the systems and methods described in this document may be used.

When field data 106 is not directly fed into the automated agrointelligence system through one or more farm machines or farm equipment devices that interact with the automated agrointelligence system, the user may be prompted through one or more user interfaces on the user device (served by the automated agrointelligence system) to enter such information. In an exemplary embodiment, the user can provide identification by accessing the map on the user's device (served by the automated agri-intelligence system) and selecting specific CLUs that have been graphically displayed on the map. In an alternative embodiment, the user may provide identification by accessing the map on the user's device (served by the automated agri-intelligence system 130) and drawing field boundaries on the map. These CLU selections or map drawings represent geographic identifiers. In alternative embodiments, the user may provide identification data by accessing field identification data (provided as form files or similar format) from the USDA Farm Service Agency or other source through the user's device and providing such field identification data to automated agrointelligence system.

In one embodiment, the automated agrointelligence system 130 is programmed to generate and display a graphical user interface including a data manager for data input. After identifying one or more fields through the methods described above, the data management device may provide one or more graphical user interface widgets that, when selected, can identify changes in the field, soil, crops, tillage practices, or nutrients. The data management device may include a sequential schedule view, a spreadsheet view, and/or one or more editable programs.

Figure 5 shows an example of a sequential work schedule for data entry. Using the display shown in FIG. 5, a selection of a specific field and a specific date for adding an event can be entered into the user's computer. Activities displayed at the top of the sequential work schedule may include Nitrogen, Planting, Practices, and Soil. To add a nitrogen application event, the user's computer can provide information to select the nitrogen tab. The user's computer on the job schedule can then highlight an area for a specific field to indicate nitrogen application to the selected field. In response to highlighting an area of a particular field on a work schedule, the data management device may display a data input overlay allowing data regarding nitrogen application, planting procedures, soil amendments, tillage procedures, irrigation methods, or other information to be entered into the user's computer , related to a specific field. For example, if an area on a work schedule is selected on a user's computer and a nitrogen application is specified, the data entry overlay may include fields for entering the amount of nitrogen applied, the date of application, the type of fertilizer used, and any other information related to the nitrogen application.

In one embodiment, the data management device provides an interface for creating one or more programs. "Program" in this context means a set of data relating to nitrogen applications, planting procedures, soil amendments, tillage procedures, irrigation methods or other information that may be associated with one or more fields and which may be stored in a digital data warehouse for reuse as a data set in other operations. Once a program is created, it can be conceptually applied to one or more fields, and references to the program can be stored in digital storage in conjunction with data identifying the fields. Thus, instead of manually entering identical data related to the same nitrogen application for several different fields, the user's computer can create a program that specifies a specific nitrogen fertilizer application and then apply the program to several different fields. For example, in the presentation of the work schedule in Fig. 5, the top two timelines select the Preplant program, which involves applying 150 lbs N/acre (170 kg/ha) in early April. The data management device may provide an interface for program editing. In one embodiment, when a particular program is edited, each field that selected that particular program is edited. For example, in Fig. 5, if the Preplant program is edited to reduce N application to 130 lb N/acre (146 kg/ha), the top two fields can be updated with reduced N application based on the edited program.

In one embodiment, in response to receiving edits for a field tagged with a program, the data management device removes the field's mapping to the selected program. For example, if a nitrogen application is added to the top field in FIG. 5, the interface may be updated to indicate that the Spring Application program is no longer applied to the top field. While early April nitrogen applications may remain, updating the Preplant program will not change April nitrogen applications.

Figure 6 shows an example of a spreadsheet view for data entry. Using the display shown in FIG. 6, the user can create and edit information for one or more fields. The data management device may include spreadsheets for entering nitrogen, planting, technology, and soil information, as shown in FIG. 6. To edit a specific record on a user's computer, you can select a specific record in the spreadsheet and update the values. For example, FIG. 6 shows an update to the yield target for a second field. In addition, on the user's computer, you can select one or more fields to apply one or more programs. In response to the selection of a program for a particular field, the data management device may automatically populate records for the particular field based on the selected program. As with the work schedule, the data manager can update records for each field associated with a particular program in response to receiving a program update. In addition, the data management device may remove a selected program's mapping to a field in response to receiving an editorial change for one of the entries in that field.

In one embodiment, model and field data are stored in model and field data store 160. Model information includes data models created for one or more fields. For example, a crop model may include a digital model of the yield of one or more fields. "Model" in this context means an electronic digital set of executable instructions and associated data values that is capable of receiving and responding to a program or other digital challenge, activation, or permission request based on specified input values to produce one or more stored or calculated output values, which can serve, among other things, as the basis for computerized recommendations, output display or machine control. It is convenient for those skilled in the art to express models using mathematical equations, but such form of expression does not limit the models disclosed herein to abstract concepts; instead, each model herein has a practical application on a computer in the form of stored executable instructions and data that implement the model using the computer. The model may include a model of past activities in one or more fields, a model of the current state of one or more fields, and/or a model of predicted activities in one or more fields. Model and field data can be stored in in-memory data structures, rows in a database table, flat files or spreadsheets, and other forms of stored digital data.

In one embodiment, the automated agrointelligence system 130 is programmed as a server computer (server) 170 for managing the classification model. The server 170 is further configured to include a model configuration instruction 172, a model construction instruction 174, a model execution instruction 176, and a user interface instruction 178.

In some embodiments, model configuration instruction 172 provides computer-executable instructions for collecting raw image data and determining the values of certain digital model parameters for recognizing plant diseases from the raw image data. If the digital model is an SSD, the original image data may be photographs of plants, including marked areas, each designated by a plant disease identifier. The marked regions can be processed taking into account the full images when constructing the training set for the SSD. The original image data can also simply be labeled areas with appropriate labels. Relevant SSD options include a group of anchor frames. Model Configuration Instruction 172 provides computer-executable instructions for creating anchor frames representing plant disease symptoms.

In some embodiments, model building instructions 174 provide computer-executable instructions for building a training set and training a digital model via the training set. The training set may include images of a specific size with labeled regions and corresponding labels, or cropped, trimmed, or scaled versions of images with equivalent labeled regions and corresponding labels. In the case where the digital model is an SSD, the training set also includes bindings of the marked areas to anchor frames.

In some embodiments, model execution instructions 176 provide computer-executable instructions for applying the digital model to new images for classification. The new image may be a new photograph of the plant showing symptoms of one or more plant diseases in one or more areas. The new image may require similar cropping, trimming, or scaling before being fed into the digital model. When the digital model is SSD, application of the digital model is expected to produce at least one classification in a particular plant disease for each of one or more regions.

In some embodiments, user interface instructions 178 offer computer-executable instructions for controlling communication with other devices. The data exchange may include receiving the original image data, including image source labels, receiving a new photo for classification from the client device, sending classification results for the new photo to the client device, or sending digital data representing the SSD to another client device.

Each component of the server 170 contains a set of one or more pages of random access memory, such as RAM, in the automated agrointelligence system 130, into which executable instructions are loaded and which, when executed, cause the automated agrointelligence system to perform the functions or operations described herein with reference to these modules. For example, the Model 172 configuration module may include a set of pages in RAM that contain instructions that, when executed, cause the region selection functions described herein to be performed. The instructions may be in machine-executable code in the CPU instruction set and may be compiled from source code written in JAVA, C, C, OBJECTIVE-C, or any other human-readable programming language or computing environment, alone or in combination with scripts in JAVASCRIPT, other scripting languages and other programming source codes. The term "pages" is intended to refer broadly to any area of RAM, and the specific terminology used in a system may vary depending on the memory architecture or processor architecture. In another embodiment, each component of server 170 may also be one or more source code files or projects that are digitally stored in a storage device such as non-volatile RAM or disk storage, in an automated agri-intelligence system 130, or in a separate storage system. which, when compiled or interpreted, cause the generation of executable instructions that, when executed, cause the automated agricultural intelligence system to perform the functions or operations described in this document with reference to these modules. In other words, a drawing may be a way in which programmers or software developers organize and arrange source code for subsequent compilation into an executable file or interpretation into bytecode or its equivalent for execution by automated agrointelligence system 130.

The hardware/virtualization layer 150 includes one or more central processing units (CPUs), memory controllers, and other devices, components, or automation system elements, such as volatile or nonvolatile memory, nonvolatile storage such as disk, and input/output devices or interfaces. as shown and described, for example, in Fig.4. Layer 150 may also include programmed instructions that are configured to support virtualization, containerization, or other technologies.

To illustrate an illustrative example, FIG. 1 shows a limited number of instances of certain functional elements. However, in other embodiments, there may be any number of such elements. For example, in embodiments there may be thousands or millions of different mobile computing devices 104 associated with different users. In addition, the system 130 and/or the external data server 108 may be implemented using two or more processors, cores, clusters, or instances of physical machines or virtual machines configured in a separate location or co-located with other elements in a data center, shared computing complex or cloud computing complex.

2.2. Application overview

In some embodiments, implementing the functions described herein using one or more computer programs or other software elements that are loaded and executed by one or more general purpose computers will result in the general purpose computers being configured as a specific a machine or computer specifically adapted to perform the functions described in this document. In addition, each of the flowcharts described later herein may serve, alone or in combination with prose descriptions of processes and functions, as algorithms, plans, or directions that can be used to program a computer or logic to implement the described functions. . In other words, all prose text herein and all illustrations taken together are intended to disclose algorithms, plans, or directions sufficient to enable a skilled person to program a computer to perform the functions described herein when combined with the skill and knowledge of that level of skilled person. techniques, taking into account the level of skill that is appropriate for inventions and disclosures of this type.

In some embodiment, the user 102 interacts with the automated agrointelligence system 130 using a field manager computing device 104 configured with an operating system and one or more application programs or applications; The field manager's computing device 104 can also interact with the automated agrointelligence system independently and automatically under program or logic control, although direct user interaction is not always necessary. Field manager computing device 104 is broadly defined as one or more smartphones, PDAs, tablet computing devices, laptops, desktop computers, workstations, or any other computing devices capable of transmitting and receiving information and performing the functions described herein. Computing device 104 may communicate over a network using a mobile application stored on computing device 104, and in some embodiments, the device may be connected via cable 113 or connector to sensor 112 and/or controller 114. A specific user 102 may own, control or possess and use in connection with the system 130 more than one field management computing device 104 at a time.

A mobile application may provide client-side functionality over a network with one or more mobile computing devices. In one embodiment, the field manager computing device 104 may access the mobile application through a web browser or through a local client application. The field manager computing device 104 may transmit data to and receive data from one or more external servers using web protocols or formats such as HTTP, XML and/or JSON, or application specific protocols. In one embodiment, the data may take the form of queries and input of user information, such as field data, into the mobile computing device. In some embodiments, the mobile application interacts with location tracking hardware and software on the field manager computing device 104, which determines the location of the field manager computing device 104 using standard tracking methods such as global positioning system (GPS) multilateration radio signals, WiFi systems positioning or other mobile positioning methods. In some cases, location data or other data associated with device 104, user 102, and/or user account(s) may be obtained by querying the device's operating system or by querying an application on the device to obtain data from the operating system.

In one embodiment, field manager computing device 104 sends field data 106 to automated agrointelligence system 130 including, but not limited to, data values representing one or more of: the geographic location of one or more fields, tillage information for one or more fields, crops planted in one or more fields, and soil data extracted from one or more fields. Field manager computing device 104 may send field data 106 in response to input from user 102 indicating data values for one or more fields. In addition, field manager computing device 104 may automatically send field data 106 when one or more data values become available to field manager computing device 104. For example, field manager computing device 104 may be communicatively coupled to remote sensor 112 and/or application controller 114, which includes an irrigation sensor and/or an irrigation controller. In response to receiving data indicating that the fertilizer controller 114 has released water in one or more fields, the field manager computing device 104 may transmit field data 106 to the automated agri-intelligence system 130 indicating that water has been released in one or more fields. . The data from field 106 indicated in this message may be entered and transmitted via electronic digital data that is transferred between computing devices using parametric URLs over HTTP or other suitable communication or messaging protocol.

A commercial example of a mobile application is CLIMATE FIELDVIEW, commercially available from The Climate Corporation, San Francisco, California. The CLIMATE FIELDVIEW application or other applications may be modified, extended, or adapted to include features, functionality, and programming that were not disclosed prior to the filing date of such invention. In one embodiment, the mobile application includes an integrated software platform that allows the producer to make fact-based decisions about their operation as it combines the producer's historical field data with any other data the producer wants to compare. Combinations and comparisons can be made in real time and are based on scientific models that provide potential scenarios that allow the manufacturer to make better, more informed decisions.

Figure 2 illustrates two views of an example of the logical organization of sets of instructions in RAM when loading for execution of an example mobile application. In FIG. 2, each named element represents an area of one or more pages of RAM or other random access memory, or one or more blocks of disk storage or other non-volatile storage, as well as programmed instructions in these areas. In one embodiment, in terms of (a), the mobile computer application 200 includes instructions 202 for sharing account-field data, review and alert instructions 204, digital map book instructions 206, seeds and planting instructions 208, instructions 210 for nitrogen, 212 for weather, 214 for field health, and 216 for productivity.

In one embodiment, the mobile computer application 200 includes an account, fields, data entry, sharing instructions 202 that are programmed to receive, translate, and ingest field data from third-party systems via manual upload or API. Data types may include, but are not limited to, field boundaries, yield maps, planted status maps, soil test results, application status maps, and/or management zones. Data formats may include, but are not limited to, form files, third party native data formats, and/or exported agricultural management information systems (FMIS) data. Retrieving data can occur through manual downloads, email with attachments, external APIs that push data into the mobile app, or instructions that call APIs of external systems to push data into the mobile app. In one embodiment, mobile computer application 200 includes data entry. In response to receiving the data input option, the mobile computer application 200 may display a graphical user interface for manually downloading data files and importing the downloaded files into the data management device.

In one embodiment, digital map book instructions 206 include field map data layers stored in device memory and programmed with data visualization and geospatial field data. This provides farmers with easy-to-use information at their fingertips for reference, harvesting, and visibility into field performance. In one embodiment, the overview and alert instructions 204 are programmed to provide insight into what is important to the manufacturer as a whole and timely recommendations to take action or focus on specific issues. This allows the farmer to concentrate on what needs attention, save time and maintain yields throughout the season. In one embodiment, the seeds and planting instructions 208 are programmed to provide tools for seed selection, hybrid placement, and scenario creation, including variable rate (VR) scenario creation based on scientific models and empirical data. This allows farmers to maximize yield or return on investment through optimized seed purchase, placement and seeding.

In one embodiment, the scenario creation instructions 205 are programmed to provide an interface for creating scenarios, including variable rate (VR) performance scenarios. The interface allows farmers to create scenarios for field implements such as soil fertilization, planting and irrigation. For example, the planting script interface may include tools to determine the type of planting material. Upon receiving the seed type option, the mobile computer application 200 may display one or more fields divided into management zones, such as field map data layers created within the instructions of the digital map book 206.

In one embodiment, management zones include soil zones along with a panel identifying each soil zone and soil name, texture, drainage for each zone, or other field data. Mobile computer application 200 may also display editing or creating tools on a map of one or more fields, such as graphical tools for drawing management zones, such as soil zones. Planting procedures may be applied to all management zones, or different planting procedures may be applied to different subsets of management zones. When a script is created, the mobile computer application 200 may make it available for download in a format readable by the application controller, such as a zipped or compressed form. Additionally and/or alternatively, the script may be sent directly to the on-board computer 115 from the mobile computer application 200 and/or downloaded to one or more data servers and stored for future use.

In one embodiment, nitrogen instruction 210 is programmed to provide a means for making nitrogen decisions by visualizing nitrogen availability for crops. This allows farmers to maximize yield or return on investment through optimized nitrogen application during the season. Examples of programmed functions include displaying images such as SSURGO images, which allow you to draw fertilizer application zones, and/or images derived from sub-area soil data, such as sensor data, at high spatial resolution (down to millimeters or less depending on the proximity of the sensors and resolution); loading of existing zones determined by the farmer; providing a graph of plant nutrient availability and/or a map that allows you to customize nitrogen application to multiple zones; output of scripts for controlling machines; tools for entering and configuring mass data; and/or maps for data visualization, among others. "Bulk data entry" in this context may mean entering data once and then applying the same data to multiple fields and/or zones that have been defined in the system; for example, the data may include the same nitrogen application data to many fields and/or zones of the same farmer, but such bulk data entry applies to the entry of any type of field data into the mobile computer application 200. For example, nitrogen instructions 210 may be are programmed to accept descriptions of nitrogen application and practice programs and to accept user input indicating the application of these programs to multiple fields. “Nitrogen application programs” in this context refer to stored, named sets of data that are associated with: a name, color code or other identifier, one or more application dates, material or product types for each date and quantity, method of application or incorporation, such as injection or broadcast, and/or the amount or rates of application for each of the dates, crops or hybrids that are the subject of application, among other things. In this context, "nitrogen programs" refer to stored, named sets of data that are associated with the name of the practice; previous culture; soil cultivation system; date of primary tillage; one or more previously used tillage systems; one or more indicators of the type of tillage, such as manure, that was used. Nitrogen instructions 210 may also be programmed to produce, resulting in the display of a nitrogen graph that shows a prediction of plant use of said nitrogen, as well as a prediction of excess or deficiency; in some embodiments, various color indicators may indicate the amount of excess or deficiency. In one embodiment, the nitrogen graph is a graphical representation on a computer display device comprising a plurality of rows, each of which is associated with and identifies a field; data indicating what crop is planted on the field, its size, location on the field and a graphic representation of the field perimeter; in each row - a time scale by month with graphic indicators indicating each application and the amount of nitrogen at points associated with the names of the months; and digital and/or color indicators of surplus or deficiency, in which the amount of surplus or deficiency is indicated by color indicators.

In one embodiment, the nitrogen schedule may include one or more graphical elements for user input, such as dials or sliders, to dynamically change nitrogen application programs so that the user can optimize their nitrogen schedule. The user can then use their optimized nitrogen schedule and associated planting and nitrogen programs to implement one or more scenarios, including variable rate (VR) performance scenarios. Nitrogen instructions 210 may also be programmed to generate, resulting in the display of a nitrogen map that shows predictions of plant use of said nitrogen, and whether an excess or deficiency is predicted; in some embodiments, various color indicators may indicate the amount of excess or deficiency. A nitrogen map can show projections of plant use of specified nitrogen and whether excess or deficiency is predicted for various time periods in the past and future (e.g. daily, weekly, monthly or yearly) using numerical and/or color indicators of excess or deficit, in which the colors indicate the value. In one embodiment, the nitrogen map may include one or more graphical elements for user input, such as dials or sliders, to dynamically change planting and nitrogen management programs so that the user can optimize his nitrogen map, for example, to obtain a preferred amount of excess or lack. The user can then use their optimized nitrogen map and associated planting and nitrogen management programs to implement one or more scenarios, including variable rate (VR) performance scenarios. In other embodiments, similar nitrogen instructions 210 may be used for other nutrient applications (such as phosphorus and potassium), pesticide applications, and irrigation programs.

In one embodiment, weather instructions 212 are programmed to provide the latest weather data and forecast weather information for a particular field. This allows farmers to save time and have an efficient integrated display for making daily operational decisions.

In one embodiment, field hygiene instructions 214 are programmed to provide timely remote sensing images highlighting seasonal crop variations and potential problems. Examples of programmed functions include cloud checking to identify possible clouds or cloud shadows; determination of nitrogen indices based on field images; graphical visualization of scouting layers, including, for example, those related to field health, and viewing and/or sharing of scouting notes; and/or downloading satellite imagery from multiple sources and prioritizing the images, particularly for the manufacturer.

In one embodiment, operating instructions 216 are programmed to provide reports, analysis, and insights using on-farm data for assessment, understanding, and decision making. This allows the farmer to achieve better results for the next year through evidence-based conclusions about why returns on investment have been stagnant and an understanding of the factors limiting returns. Operating instructions 216 may be programmed to communicate over network 109 with internal analytical programs running on automated agrointelligence system 130 and/or external data server 108 and configured to analyze metrics such as yield, payback differential, hybrid, etc., among others. population, SSURGO zone, soil properties or altitude. Program reports and analyzes may include yield variability analysis, treatment effect estimates, yield benchmarking, and other metrics relative to other farmers, based, among other things, on anonymized data collected from many farmers or seed and planting data.

Applications configured in this way can be implemented across multiple computing device platforms while maintaining the same general user interface appearance. For example, a mobile application can be programmed to run on tablet computers, smartphones, or servers that are accessed through browsers on client computers. In addition, a mobile application configured on tablet computers or smartphones may provide a complete operation of an application or on-board computer application suitable for displaying and processing capabilities of the on-board computer 115. For example, referring now to representation (b) in FIG. 2, in one of In embodiments, the on-board computer application 220 may include on-board map instructions 222, remote viewing instructions 224, data collection and transmission instructions 226, machine alert instructions 228, script transmission instructions 230, and on-board scouting instructions 232. The code base for instructions of type (b) can be the same as for type (a), and the executable files implementing the code can be programmed to detect the type of platform on which they are executed and to demonstrate using a graphical user interface only those functions that are suitable for a flatbed platform or a complete platform. This approach allows the system to recognize distinctly different user functions suitable for the on-board environment and for different on-board technological environments. The on-board instruction cards 222 may be programmed to present maps of fields, farms, or regions that may be useful for controlling machine operation. Remote viewing instructions 224 may be programmed to enable, control, and provide real-time or near real-time snapshots of the machine's operation to other computing devices connected to the system 130 via wireless networks, wired connectors or adapters, and the like. Data collection and transmission instructions 226 may be programmed to enable, control, and enable transmission of data collected on sensors and controllers to system 130 via wireless networks, wired connectors or adapters, and the like. The automatic alert instructions 228 may be programmed to detect machine or tooling problems associated with the board and generate alerts to the operator. Instruction transmission script 230 may be configured to transmit instructions in scripts that are configured to control machine operation or collect data. Scout on-board computer instructions 232 may be programmed to display location-based alerts and information received from system 130 based on the location of computing devices 104, farm equipment 111, or sensors 112 in the field, as well as to receive, control, and provide transmission to the system 130 of search and reconnaissance information based on the location of agricultural machinery 111 or sensors 112 in the field.

2.3. Data entry into the computer system

In one embodiment, external data server 108 stores external data 110, including soil composition data for one or more fields and weather data representing temperature and precipitation for one or more fields. Weather data may include past and present weather data, as well as future weather forecast data. In one embodiment, the front-end data server 108 includes a plurality of servers located in different entities. For example, the first server may contain soil composition data, while the second server may contain weather data. In addition, soil composition data can be stored on multiple servers. For example, one server may store data representing the percentage of sand, silt, and clay in the soil, while a second server may store data representing the percentage of organic matter (OM) in the soil.

In one embodiment, remote sensor 112 includes one or more sensors that are programmed or configured to make one or more observations. The remote sensor 112 may be an airborne sensor such as a satellite, a vehicle sensor, a planting equipment sensor, a tillage sensor, a fertilizer or insecticide application sensor, a harvest sensor, or any other equipment capable of receiving data from one or more fields. In one embodiment, the fertilizer application controller 114 is programmed or configured to receive instructions from the automated agrointelligence system 130 . The fertilizer controller 114 may also be programmed or configured to control the operating parameters of the agricultural vehicle or implement. For example, a fertilizer controller may be programmed or configured to control operating parameters of a vehicle such as a tractor, planting equipment, tillage equipment, fertilizer or insecticide application equipment, harvesting equipment, or other agricultural equipment such as a water valve. In other embodiments, any combination of sensors and controllers may be used, some of which are shown below by way of example only.

System 130 may receive data, under the control of user 102, in bulk from a large number of farmers who have contributed data to a common database system. This form of data acquisition may be referred to as "manual data entry" because one or more user-controlled computer operations are requested or initiated by system 130 to obtain data. An example is the CLIMATE FIELDVIEW application, commercially available from Climate Corporation, San Francisco, California, for exporting data to system 130 for storage in storage 160.

For example, seed monitoring systems can both control planter components and receive planting data, including signals from seed sensors through a signal harness consisting of a CAN backbone and point-to-point connections for logging and/or diagnostics. Seed monitoring systems may be programmed or configured to display seed spacing, seed quantity, and other information through on-board computer 115 or other devices in system 130. Examples are provided in US Pat. No. 8,738,243 and US Pat. Pub. 20150094916, and the present disclosure assumes knowledge of those other patent inventions.

Likewise, yield monitoring systems may include yield sensors for harvesters that send yield measurements to on-board computer 115 or other devices in system 130. Yield monitoring systems may use one or more remote sensors 112 to provide grain moisture measurements at combine harvester or other combine harvester and transmitting these results to the user through the on-board computer 115 or other devices in the system 130.

In one embodiment, exemplary sensors 112 that can be used with any moving vehicle or device of the type described herein include kinematic sensors and position sensors. Kinematic sensors can include any speed sensors such as radar or wheel speed sensors, accelerometers or gyroscopes. Position sensors may include GPS receivers or transceivers, as well as WiFi location or mapping applications that are programmed to determine location based, in part, on nearby WiFi access points.

In one embodiment, exemplary sensors 112 that may be used in conjunction with tractors or other moving vehicles include engine speed sensors, fuel consumption sensors, area or distance meters that interface with GPS or radar signals, sensors power take-off speeds, tractor hydraulic sensors configured to detect hydraulic parameters such as pressure or flow and/or hydraulic pump speed sensors, wheel speed or wheel slip sensors. In one embodiment, exemplary controllers 114 that may be used in conjunction with tractors include: hydraulic directional controllers; pressure controllers and/or flow controllers; hydraulic pump rotation speed controllers; controllers or speed regulators; hitch position controllers; or wheel position controllers that provide automatic steering.

In one embodiment, exemplary sensors 112 that may be used in conjunction with planting equipment such as a planter, seeders, or air seeders include: seed sensors, which may be optical, electromagnetic, or impact; downforce sensors such as load cells, strain gauges, pressure sensors; soil sensors such as reflectance sensors, moisture sensors, conductivity sensors, optical residue sensors or temperature sensors; component performance sensors, such as planting depth sensors, cylinder pressure sensors, disc speed sensors, seed drive motor sensors, seed transport system speed sensors, or vacuum level sensors; or pesticide application sensors, such as optical or other electromagnetic or impact sensors. In one embodiment, exemplary controllers 114 that may be used in conjunction with such seed planting equipment include: instrument panel folding controllers, such as controllers for valves associated with hydraulic cylinders; downforce controllers, such as controllers for valves associated with pneumatic cylinders, airbags or hydraulic cylinders and programmed to apply downforce to individual rows or the entire planter frame; planting depth controllers such as linear actuators; metering controllers such as electric seed counter drive motors, hydraulic seed counter drive motors or roller clutches; hybrid breeding controllers, such as seed counter drive motors, or other drives programmed to selectively feed or prevent seed or seed mixture from being fed to seed counters or central hoppers; metering controllers such as seed counter drive motors or hydraulic seed counter drive motors; seed transport system controllers, such as conveyor belt motor controllers; marker controllers, such as pneumatic or hydraulic actuator controllers; or pesticide application rate controllers, such as metering drive controllers, orifice size or position controllers.

In one embodiment, exemplary sensors 112 that may be used in conjunction with tillage equipment include position sensors for tools such as shanks or discs; tool position sensors for such tools that are configured to detect depth, inclination or lateral distance; downforce sensors; or traction sensors. In one embodiment, exemplary controllers 114 that may be used in conjunction with tillage equipment include downforce or tool position controllers, such as controllers configured to control depth, block angle, or lateral distance between blocks.

In one embodiment, exemplary sensors 112 that may be used in conjunction with equipment for applying fertilizers, insecticides, fungicides, and the like, such as plantation fertilizer systems, soil fertilizer applicators, or fertilizer sprayers , include: fluid system criteria sensors, such as flow or pressure sensors; sensors indicating which pressure valves or fluid supply valves are open; sensors associated with tanks, such as fill level sensors; sectional or system-wide supply line sensors or row-specific supply line sensors; or kinematic sensors such as accelerometers located on the sprayer booms. In one embodiment, exemplary controllers 114 that may be used in conjunction with such devices include pump speed controllers; valve controllers, which are programmed to control pressure, flow, direction, PWM, etc.; or position actuators, for example for boom height, sole depth or boom position.

In one embodiment, exemplary sensors 112 that may be used in conjunction with harvesters include: yield monitors such as impact plate strain sensors or position sensors, capacitive flow sensors, load sensors, weight sensors, or torque sensors, associated with elevators or augers, or optical or other electromagnetic grain height sensors; grain moisture sensors such as capacitive sensors; grain loss sensors, including impact, optical or capacitive sensors; header performance sensors such as header height, header type, platen gap, feed speed, and reel speed; separator performance sensors such as concave clearance, rotor speed, block clearance or chaff sieve clearance; sensors for position, operation or speed of augers; or engine speed sensors. In one embodiment, exemplary controllers 114 that may be used in conjunction with combines include: controllers for header performance, such as header height, header type, platen gap, feed speed, or reel speed; controllers for separator operating characteristics such as concave clearance, rotor speed, block clearance or chaff sieve clearance; or controllers for position, operation or speed of the auger.

In one embodiment, exemplary sensors 112 that may be used in conjunction with grain carts include weight sensors or sensors for auger position, operation, or speed. In one embodiment, exemplary controllers 114 that may be used in conjunction with grain carts include auger position, operation, or speed controllers.

In one embodiment, the exemplary sensors 112 and controllers 114 may be installed in unmanned aerial vehicles (UAVs) or "drones". Such sensors may include: cameras with detectors that operate effectively in any range of the electromagnetic spectrum, including visible light, infrared, ultraviolet, near-infrared (NIR), etc.; accelerometers; altimeters; temperature sensors; humidity sensors; pit tube sensors or other air or wind speed sensors; battery life sensors; or radar emitters and devices for detecting reflected radar energy; other emitters of electromagnetic radiation and devices for detecting reflected electromagnetic radiation. Such controllers may include guidance or motor control devices, control surface controllers, camera controllers, or controllers programmed to enable, operate, receive data from any of the above sensors, control, and configure. Examples are given in US Pat. Arr. No. 14/831,165, and the present description assumes knowledge of this other patent specification.

In one embodiment, sensors 112 and controllers 114 may be attached to a sampler and measurement device that is configured or programmed to collect soil samples and perform soil chemical tests, soil moisture tests, and other soil-related tests. For example, the device disclosed in patent application US 15/551,582, filed August 16, 2017, may be used, and the present disclosure assumes knowledge of the specification of that patent.

In one embodiment, sensors 112 and controllers 114 may include weather devices for monitoring field weather conditions. For example, the device disclosed in US Provisional Patent Application US 62/154,207, filed April 29, 2015, US Provisional Patent Application US 62/175,160, filed June 12, 2015, US Provisional Patent Application US 62/198,060, filed July 28, 2015, and U.S. Provisional Patent Application US 62/220,852, filed September 18, 2015, may be relied upon, and the disclosure assumes knowledge of the disclosures of those patents.

2.4. Process Overview - Agronomic Model Training

In one embodiment, the automated agrointelligence system 130 is programmed or configured to create an agronomic model. In this context, an agronomic model is a data structure in the memory of an automated agrointelligence system 130 that includes field data 106, such as identification and yield data for one or more fields. An agronomic model may also include calculated agronomic properties that describe either the conditions that may affect the growth of one or more crops in a field, or the properties of one or more crops, or both. Additionally, an agronomic model may include recommendations based on agronomic factors such as crop recommendations, irrigation recommendations, planting recommendations, fertilizer recommendations, fungicide recommendations, pesticide recommendations, harvest recommendations, and other agricultural management recommendations. cultures. Agronomic factors can also be used to evaluate one or more crop-related outcomes, such as yield. Agronomic crop yield is an estimate of the amount of a crop produced or, in some examples, the revenue or profit generated by that crop.

In one embodiment, the automated agrointelligence system 130 may use a preconfigured agronomic model to calculate agronomic properties associated with received current location and yield data for one or more fields. The pre-configured agronomic model is based on previously processed field data, including but not limited to identification data, yield data, fertilizer data and weather data. A pre-configured agronomic model can be cross-validated to ensure model accuracy. Cross-validation may include a comparison with a ground edit that compares predicted results with actual results in a field, such as comparing a rainfall estimate with a rain gauge or sensor providing weather data at the same or nearby location, or a nitrogen estimate with a soil sample measurement .

Figure 3 illustrates a programmable process by which an automated agrointelligence system generates one or more preconfigured agronomic models using field data provided by one or more data sources. Fig. 3 may represent an algorithm or instructions for programming functional elements of an automated agrointelligence system 130 to perform the described operations.

At block 305, the automated agrointelligence system 130 is configured or programmed to perform pre-processing of agronomic data obtained from one or more data sources. Field data obtained from one or more data sources may be preprocessed to remove noise, confounding effects, and bias factors within the agronomic data, including anomalies that may negatively impact the resulting field data values. In one embodiment, pre-processing of agronomic data may include, but is not limited to, removal of data values typically associated with outliers, specific measurement data points that are known to unduly bias other data values, data smoothing, aggregation, or sampling methods used to remove or reduce the additive or multiplicative effects of noise, and other filtering or data acquisition techniques used to ensure a clear distinction between positive and negative input data.

At block 310, the automated agrointelligence system 130 is configured or programmed to select a subset of data using the preprocessed field data to determine a set of data useful for creating an initial agronomic model. The automated agrointelligence system 130 may provide data subset selection methods, such as a genetic algorithm method, an all subset modeling method, a sequential search method, a stepwise regression method, a particle swarm optimization method, and an ant colony optimization method. For example, the genetic algorithm selection method uses an adaptive heuristic search algorithm based on evolutionary principles of natural selection and genetics to identify and evaluate data sets within preprocessed agronomic data.

At block 315, the automated agrointelligence system 130 is configured or programmed to implement an evaluation of a set of field data. In one embodiment, a specific set of field data is assessed by creating an agronomic model and using specific quality thresholds for the created agronomic model. Agronomic models can be compared and/or validated through one or more comparison methods, such as, for example, root mean square error with cross validation (RMSECV), mean absolute error, and mean percentage error. For example, the RMSECV system can cross-validate agronomic models by comparing the predicted values of agronomic indicators generated by the agronomic model with the historical values of collected and analyzed agronomic indicators. In one embodiment, the agronomic data set evaluation logic is used as a feedback loop in which agronomic data sets that do not meet configured quality thresholds are used in future data subset selection steps (block 310).

At block 320, the automated agrointelligence system 130 is configured or programmed to create an agronomic model based on cross-validated agronomic data sets. In one embodiment, an agronomic model developer may implement multivariate regression to create pre-configured models of agronomic data.

At block 325, the automated agrointelligence system 130 is configured or programmed to store preconfigured agronomic data models for future evaluation of field data.

2.5. Case Study - Hardware Overview

In accordance with one embodiment, the techniques described herein are implemented by one or more specialized computing devices. Application-specific computing devices may be hard-wired to implement technical solutions, or may include digital electronic devices, such as one or more application-specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs), that are hard-wired to implement these technical solutions, or may include one or more general purpose hardware processors programmed to execute technical solutions in accordance with software instructions in hardware-implemented software, memory, other storage devices, or a combination thereof. Such application-specific computing devices may also combine custom hardware logic, application-specific integrated circuits (ASICs), or programmable logic integrated circuits (FPGAs) with custom programming to implement technical solutions. Special purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, network devices, or any other device that includes hardware and/or software logic to implement technical solutions.

For example, FIG. 4 is a block diagram illustrating a computer system 400 on which an embodiment of the invention may be implemented. The computer system 400 includes a bus 402 or other communication mechanism for transmitting information, as well as a hardware processor 404 coupled to the bus 402 for processing the information. The hardware processor 404 may be, for example, a general purpose microprocessor.

Computer system 400 also includes random access memory 406, such as random access memory (RAM) or other dynamic storage device, coupled to bus 402 to store information and instructions executed by processor 404. Memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 404. Such instructions, when stored on non-transitory storage media accessible to the processor 404, transform the computing system 400 into a specialized machine configured to perform the operations specified in the instructions.

The computer system 400 further includes a read-only memory (ROM) 408 or other static storage device connected to the bus 402 for storing static information and instructions for the processor 404. A storage device 410, such as a magnetic device, is provided and connected to the bus 402 to store the information and instructions. disk, optical disk or semiconductor disk.

The computer system 400 may be coupled via bus 402 to a display 412, such as a cathode ray tube (CRT), to display information to a computer user. A data input device 414, including alphanumeric and other keys, is connected to the bus 402 to transmit information and select commands to the processor 404. Another type of user input device is a cursor control 416, such as a mouse, trackball, or cursor direction keys to transmit information about direction and selection of commands to the processor 404, and to control the movement of the cursor on the display 412. This input device typically has two degrees of freedom along two axes, a first axis (eg x) and a second axis (eg y), which allows the device to specify positions in the plane .

The computer system 400 may implement the techniques described herein using custom hardware-implemented logic, one or more SIS or FPGAs, firmware, and/or program logic that, in combination with the computer system, causes or is programmed to cause the computer system 400 to become into a specialized machine. In accordance with one embodiment, the technical solutions described herein are executed by the computer system 400 as a result of the processor 404 executing one or more sequences of one or more instructions contained in the main memory 406. Such instructions may be transferred to the main memory 406 from another storage medium, for example from storage device 410. Execution of sequences of instructions contained in main memory 406 causes processor 404 to perform the processing steps described herein. In alternative embodiments, hardware-based circuitry may be used instead of or in combination with software instructions.

As used herein, the term "storage medium" refers to any non-transitory storage media on which data and/or instructions leading to the specific operation of a machine are stored. Such storage media may be non-volatile and/or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or semiconductor storage, such as storage device 410. Non-volatile media includes dynamic memory, such as random access memory 406. Common forms of storage media include, for example, floppy disk, floppy disk, hard disk , solid state drive, magnetic tape or any other magnetic storage media, CD-ROM, any other optical storage media, any physical media with hole patterns, RAM, ROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

The storage medium is different from the transmission medium, but can be used in conjunction with it. The transmission medium is involved in the transfer of information between storage media. For example, the transmission medium includes coaxial cables, copper wire, and fiber optics, including the wires that constitute buses 402. The transmission medium may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data transmission.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, instructions may initially be transferred to a magnetic disk or semiconductor disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send them over the telephone line using a modem. The local computer system modem 400 may receive data over a telephone line and use an infrared transmitter to convert the data into an infrared signal. The infrared detector may receive data contained in the infrared signal, and associated circuitry may provide the data to bus 402. Bus 402 transfers the data to main memory 406, from which processor 404 retrieves and executes instructions. Instructions received by main memory 406 may optionally be stored in memory 410 either before or after execution by processor 404.

Computer system 400 also includes a communications interface 418 coupled to bus 402. Communications interface 418 provides two-way data transfer with a network link 420 that is connected to a local area network 422. For example, communications interface 418 may be an integrated digital services network (ISDN) card. , cable modem, satellite modem, or modem to provide data transmission with the appropriate type of telephone line. As another example, communications interface 418 may be a local area network (LAN) card for providing data transfer from a compatible LAN. Wireless connections can also be implemented. In any such implementation, communications interface 418 sends and receives electrical, electromagnetic, or optical signals that convey digital data streams representing various types of information.

Network link 420 typically provides data transmission over one or more networks to other data communication devices. For example, communication link 420 may provide connectivity over a local area network 422 to a host computer 424 or to a data device operated by an Internet Service Provider (ISP) 426. The Internet Service Provider 426, in turn, provides data services via the worldwide packet data network, which is now commonly referred to as the "Internet" 428. Local area network 422 and the Internet 428 use electrical, electromagnetic or optical signals that transmit digital data streams. Signals over various networks and signals over network link 420 and through communications interface 418 that transfer digital data to and from computer system 400 are examples of forms of data transfer.

Computer system 400 may send messages and receive data, including program code, over network(s), network link 420, and communications interface 418. Using the Internet as an example, server 430 may transmit requested application code over Internet 428, ISP 426, local area network 422, and communication interface 418. The received code may be executed by processor 404 as it is received and/or stored in memory 410 or other non-volatile storage device for later execution.

3. Descriptions of functions

3.1. Detection of plant diseases

Various classification methods based on image analysis are available today. These classification methods can be used to analyze photographs of plants and classify the photographs into specified disease classes or disease-free classes, thereby identifying potential plant infections with the corresponding diseases. Some of these classification methods include CNNs, including SSD.

Typically, SSD starts with a basic CNN containing a number of convolutional layers that correspond to multi-scale receptive fields. SSD then performs classification through feature maps generated not only by the last convolutional layer, but also by other convolutional layers. More precisely, SSD uses a set of user-defined anchor boxes, which typically have different sizes and aspect ratios and which correspond to various features of given classes, for each of the specified convolutional layers in the underlying CNN. The SSD then includes a set of 4+c small (e.g. 3x3) filters for each of the anchor boxes (in what is called a convolutional feature layer), with 4 of them corresponding to the four sides of the anchor box, and 'c' being the number of classes for each of the convolutional boxes. layers. These small filters are trained with reference images of known features for each of the classes. These images can come in all sorts of sizes and aspect ratios, and each image is associated with a class and an anchor box. These small filters can be used to determine whether the region outlined by one of the anchor boxes in the feature map produced by the corresponding convolutional layer corresponds to features of one of the classes.

As described above, SSD can recognize all sorts of class features having all sorts of sizes and aspect ratios in one or more images, and classify parts of the images into classes with corresponding confidence scores in one stage, without requiring an initial, separate stage of generating a candidate region to locate the features on the images. It is possible that the SSD initially classifies the image area into several classes. Non-maximal suppression (NMS) can be applied to select one of several classes as the final classification. SSD has been shown to be faster than previous single-stage detectors, as well as more accurate, in fact, than slower methods that generate accurate region proposals and perform pooling. There are third-party libraries available on the GitHub platform that implement SSD functions using Keras and Python libraries, which can use VGG16 as a basic CNN as an example.

3.2. Digital model configuration

In some embodiments, the server 170 is programmed to build a digital model for identifying plant disease infestations from plant images via the SSD. For corn, common plant diseases are anthracose leaf blight (ALB), common rust (CR), eye blight (EYE), gray leaf blight (GLS), Goss's wilt (GW), northern leaf blight (NLB), northern leaf spot (NLS). ), southern leaf blight (SLB) and southern rust (SR). The digital model may be designed to classify images into a certain number of classes corresponding to a certain number of such plant diseases. The server 170 is programmed to first receive a set of images, such as photographs of corn leaves, that may have areas of disease symptoms marked or directly characterize such disease symptoms.

Figure 7A shows examples of photographs of corn leaves, each of which has symptoms of one disease. Figure 720 shows GLS symptoms in box (defining the marked area) 704 and box 706. Box 704, labeled “GLS,” includes a single lesion, and box 706, also labeled “GLS,” includes a disconnected but closely located lesion. , which is not uncommon. Frame 704 and frame 706 are different sizes and similar, but differ in aspect ratio. Having a single frame 706 instead of multiple frames for each of the disconnected faults can improve the efficiency of training and execution of the digital model. Figure 722 shows SR symptoms in box 708, which includes a large group of individual lesions, as is often the case. The 708 frame also has a clear size and aspect ratio.

Figure 7B is an example of a photograph of a corn leaf showing symptoms of multiple diseases. Figure 724 shows the symptoms of GLS in box 710. Figure 724 also shows the symptoms of CR in box 712, which includes many individual lesions. Frame 710, labeled "GLS", and frame 712, labeled "CR", have different sizes and aspect ratios. Although frame 712 includes multiple lesions, since damage from CR predominates, this marked area corresponding to frame 712 can still serve as a good sample for CR.

In some embodiments, server 170 is programmed to process each image having marked areas following certain rules that may complete or improve the marking. The server 170 may be configured to split each marked area into multiple areas or to combine multiple marked areas into one according to a constraint on the size of the marked area, the density of the damage cluster, or the proportion between cluster and leaf. For example, instead of frame mark 712, photograph 724 could only have a small marked area for individual CR damage. Server 170 may be programmed to automatically expand this marked area to frame 712 based on detection of nearby damage and determination of the overall size of the damage cluster relative to the sheet size. Additionally, server 170 may be configured to limit the number of marked regions in each image, such as no more than six, to simplify the SSD training process. For example, server 170 may be programmed to automatically reduce the number of labeled regions in image 724 from the current seven to six by unlabeling the labeled region that has the smallest size, or the region with the same aspect ratio as another labeled region.

In some embodiments, server 170 is programmed to determine anchor frames for convolutional layers in the SSD. Each anchor frame can be expressed in terms of length, aspect ratio, and scalability factor. For example, the length unit could be ten pixels, the width-to-height ratio could be 1.0, and the scalability factor could be 1, resulting in an anchor frame that is ten pixels wide by ten pixels long. The aspect ratio can be 2.0, which results in an anchor frame that is twenty pixels wide by ten pixels long. The scaling factor can be set to 2.0, resulting in an anchor frame that is twenty pixels wide by twenty pixels long. Server 170 may be programmed to use marked areas, such as those corresponding to frames 704, 706, 708, 710, or 712, to define anchor frames. The server 170 can be programmed to standardize the marked areas (eg, a fixed distance between the camera and the plant and a fixed camera resolution), group similar areas into one cluster, and calculate the total size and aspect ratio for each cluster to determine scale factors and aspect ratios. Since a number of convolutional layers in a basic CNN typically correspond to increasingly larger receptive fields, the server can be programmed to assign increasingly larger scaling factors to a number of convolutional layers. The server can also be programmed to apply a scaling factor, such as 0.1, which is less than the scaling factor used in a typical SSD implementation to help identify very small lesions caused by certain corn diseases or other very small disease symptoms. For example, scaling factors could be 0.1, 0.2, 0.37, 0.54, 0.71, 0.88, 1.05 for a series of six convolutional layers, with one convolutional layer assigned one scaling factor for all anchor frames. For corn diseases, common aspect ratios include 1.0/7.0, 1.0/5.0, 1.0/3.0, 0.5, 1.0, 2.0, 3.0, 5.0 or 7.0.

3.3. Training kit and digital model construction

In some embodiments, the server 170 is programmed to scale, fill, or otherwise process the image to produce the final images of the training set. For example, the max_crop_and_resize and random_pad_and_resize functions available on the GitHub platform can be adapted to generate variations of source images. The server 170 is programmed to associate each option with marked areas and corresponding labels, as in the original image. To identify plant diseases, classes correspond to plant diseases, and each label identifies one of the plant diseases. Images do not need to be rotated to provide additional images for the training set when symmetrical aspect ratios are used for anchor frames. To identify corn disease infestation, the number of marked areas can be at least 100 for each corn disease.

In some embodiments, server 170 is programmed to match a training set of images to anchor frames, as required to build the SSD. For example, SSDBoxEncoder, available on the GitHub platform, can be adapted to also link to variants of the original images for such a match with a pos_iou_threshold set of 0.5 and a neg_iou_threshold set of 0.2. The server 170 is programmed to subsequently build a digital model for corn disease recognition from the anchor boxes and training set, including images with labeled regions or variants thereof, corresponding class labels, and associated anchor box matches. For example, the model.fit_generator function in the Keras library can be used with lr_schedule set to 0.001.

3.4. Implementation of a digital model

In some embodiments, server 170 is programmed to receive a new image, such as a photograph of a corn plant, and apply a digital model to the new image. Server 170 is programmed to convert the new image to a square image, if necessary. Instead of cropping the new image causing loss of information, the server 170 may be configured to perform padding to create an updated image where each edge has a length equal to the long edge of the new image. Next, the server 170 may be configured to center the new image in the updated image and scale the result to obtain the final input image.

In some embodiments, server 170 is programmed to execute a digital model on the final input image. For example, the decode_y function available on the GitHub platform can be used to implement execution, with confidence_thresh set to 0.8 and iou_threshold to 0.5 for NMS.

FIG. 8 shows an example of the process of applying NMS to refine an initial classification result to a final classification result. Figure 802 shows part of a corn leaf. As discussed above, the SSD can initially classify the image area into several classes. In this example, each of the frames, including frame 804 and frame 808, delineates an area of the image that has been assigned to one of the corn disease classes. In particular, frame 808 refers to a set of frames that cover pixel 806 or a surrounding area. Before applying the NMS, those frames where the corresponding confidence scores are below a certain threshold, such as confidence_thresh, can be filtered out. In this example, the frame classification 804 is associated with a low confidence level and thus can be removed. The frame with the highest score is then selected via NMS, then all other frames that intersect with that frame with a value greater than a certain threshold, such as iou_thresh, are removed, and the process continues until the frame is no longer filtered. In this example, only frame 808 remains, and thus pixel 806 will be classified based on frame 808.

3.5. Examples of processes

Figure 9 shows an example of a method performed by a server that is programmed to configure and use a CNN for plant disease detection. FIG. 9 is a disclosure of an algorithm, plan, or diagram that may be used to implement one or more computer programs or other software elements that, when executed, result in the functional refinements and technical improvements described herein. In addition, the flowcharts presented here are described in the same level of detail that experts in the art typically use to communicate with each other about the algorithms, plans, or specifications that form the basis of the software they plan to code or implement using accumulated knowledge. their skills and knowledge.

In some embodiments, at step 902, the server 170 is programmed or configured to receive a set of photographs of plants infected with a variety of diseases. In particular, the photo set shows leaves with multiple marked areas of multiple aspect ratios. Each marked area is associated with a label for one of many diseases and shows at least one damage caused by one disease. The photo set includes a specific photograph showing a specific sheet having a marked area of multiple damage. For any such particular photograph, the total size of the multiple lesions relative to the size of the specific marked area must be greater than the first predetermined percentage value. In addition, the size of the specific marked area relative to the size of the specific sheet must be greater than the second predefined percentage value. For example, a set of photographs should show diseased leaves of corn, and one of the photographs should show a leaf containing a cluster of lesions from one of the corn diseases, where the lesions are located close to each other and occupy a large portion of the leaf.

In some embodiments, at step 904, server 170 is programmed or configured to determine a group of anchor boxes from a plurality of marked regions for each of a number of SSD convolutional layers. Each group of anchor boxes typically has a different aspect ratio, representative of at least a subset of the marked areas that may correspond to similar symptoms of the same disease. For corn, the aspect ratio can be as high as 7:1. A series of convolutional layers have increasingly larger receptive fields, so server 170 can be programmed to assign larger anchor boxes to deeper convolutional layers.

In some embodiments, at step 906, after determining a group of anchor frames, the server 170 may be programmed to subsequently associate each of the plurality of marked areas with one of the groups of anchor frames. The server 170 may be configured to match the marked region to the anchor frame when the percentage of the area where the marked region and the anchor frame intersect to the area of their union is greater than a certain threshold.

In some embodiments, at step 908, the server 170 is programmed or configured to construct an SSD based on a group of anchor frames, a set of photographs with a plurality of marked areas, a plurality of disease labels attached, and a plurality of anchor frame mappings attached. For educational purposes, a set of original photographs can be supplemented with modifications resulting from cropping, tamping, resizing, or other image processing operations performed on the original photographs. Modifications are associated with equivalent marked areas and corresponding markings as in the original images. Even when these image processing operations are performed, the aspect ratios of the marked areas are preserved and linked by the same anchor frames as in the original photographs.

In some embodiments, at step 910, the server 170 is programmed or configured to receive a new image from the client device. The new image may be a photograph of a plant that depicts symptoms of one or more diseases in one or more areas. At step 912, server 170 is programmed or configured to apply SSD to the new image to detect symptoms of one or more diseases in one or more areas of the new image.

In some embodiments, at step 914, the server 170 is programmed or configured to transmit data associated with one or more diseases or one or more areas to the client device. The data may identify each of one or more domains and one or more domain classifications. The data may also include a confidence estimate for each of one or more plant disease classifications.

4. Extensions and alternatives

In the above specification, embodiments of the invention are described with reference to numerous specific details, which may vary from implementation to implementation. Accordingly, the description and drawings are to be considered in an illustrative and not a restrictive sense. The sole and exclusive indication of the scope of the invention, and of what applicants believe to be the scope of the invention, is the literal and equivalent scope of the claims that appear from this application in the specific form in which such claims are provided, including any subsequent amendments.

Claims (37)

1. A computerized method for recognizing plant diseases from image data using a convolutional neural network, including: reception by the processor of a set of photographs of plants infected with a variety of diseases, and: the set of photographs shows leaves with a plurality of marked areas having multiple aspect ratios, each marked area associated with a disease tag of a plurality of diseases and showing at least one lesion caused by the disease, a specific photograph from a set of photographs shows a specific sheet having a specific marked area with multiple damage, the total size of the multiple lesions exceeds a predetermined percentage of the size of the specific marked area; determination by the processor of a group of anchor frames from a set of marked areas for each of a number of convolutional layers of a single-shot multibox detector (SSD), and: The SSD is configured to receive an image and assign to each of one or more image regions at least one class of a plurality of classes corresponding to a plurality of diseases, a group of anchor frames has different aspect ratios and corresponds to different features of multiple classes; wherein the definition of a group of anchor frames further includes: dividing, combining, or removing one or more of a plurality of marked areas to create a new set of marked areas subject to a constraint on the size of the marked area, on the ratio of the size of a damage cluster within a marked area to a sheet, or on the density of damage within a marked area based on a predefined percentage meanings, wherein the determination is carried out additionally according to a new set of marked areas; wherein the method further includes associating each area of the plurality of marked areas with one of the groups of anchor frames; And training an SSD using a group of anchor frames, a set of photographs having a plurality of labeled regions, a corresponding plurality of labels, and a corresponding plurality of anchors. 2. The computerized method of claim 1, wherein the determination includes marking a first marked area of the plurality of marked areas that is the smallest or comparable in size to a second marked area of the plurality of marked areas. 3. Computerized method according to claim 1, the definition including: combining multiple labeled areas into composite clusters; calculating the composite area for a cluster of composite clusters; specifying an anchor frame from a group of cluster-based anchor frames. 4. Computerized method according to claim 1, wherein the definition includes: identifying each frame from the groups of anchor frames by length unit, aspect ratio, and scaling factor; assigning a group of anchor boxes a larger scaling factor for a convolutional layer further down the series of convolutional layers. 5. Computerized method according to claim 4, wherein the plants are corn, aspect ratio is 1.0/7.0, 1.0/5.0, 1.0/3.0, 0.5, 1.0, 2.0, 3.0, 5.0 or 7.0 . 6. The computerized method of claim 1, wherein the anchoring includes matching the marked area to an anchor frame when the percentage value of the size of the intersection of the marked area with the anchor frame divided by their union is greater than a certain threshold value. 7. The computerized method of claim 1, wherein receiving the new image comprises padding the new image into a square shape and then scaling the new image into the square shape. 8. The computerized method of claim 1, wherein using the SSD includes performing non-maximum suppression (NMS) to select one of a plurality of classes where a particular region of one or more regions of the new image is assigned to several classes of the plurality of classes. 9. The computerized method of claim 1, wherein applying the SSD includes assigning one or more areas of the new image to one or more classes from a plurality of classes corresponding to one or more diseases. 10. The computerized method of claim 1, wherein the SSD includes a fixed number of filters of a fixed size for each group of anchor frames to be applied to the feature maps produced by each layer of the number of convolutional layers. 11. The computerized method according to claim 1, including receiving a new image from the client device; applying the SSD to the new image to identify symptoms of one or more diseases in one or more areas of the new image; transferring data associated with one or more diseases or one or more areas of a new image to a client device. 12. A machine-readable long-term storage medium on which one or more sequences of instructions are stored, the execution of which leads to the implementation by one or more processors of the method according to one of paragraphs. 1-11. 13. Computer system (400), including: - RAM (406), storing one or more instructions; - one or more processors (404) configured to execute one or more instructions that, when executed by one or more processors, result in the implementation of the method according to one of claims. 1-11.