BR102022026428A2 - VIRTUAL SAFETY BUBBLES FOR SAFE NAVIGATION OF AGRICULTURAL MACHINERY - Google Patents

Abstract

An autonomous agricultural machine navigable in an environment to perform agricultural action(s) is disclosed. The agricultural machine receives a notification from a manager that there are no obstacles in the detection system's blind spots. The agricultural machine applies an obstacle detection model to captured images to verify that there are no obstacles in unobstructed views. The agricultural machine determines an agricultural machine configuration. The agricultural machine determines a virtual security bubble for the agricultural machine to autonomously execute the agricultural action(s) based on the determined configuration. The agricultural machine detects an obstacle in the environment by applying the obstacle detection model to the captured images. The farming machine determines that the obstacle is entering the virtual security bubble. In response to the determination that the obstacle is entering the virtual security bubble, the farm machine terminates the farm machine operation and/or takes preventive measures.

Description

VIRTUAL SAFETY BUBBLES FOR SAFE NAVIGATION OF AGRICULTURAL MACHINERY REFERENCE TO RELATED REQUESTS

[001] This Order claims the benefit and priority of United States Provisional Order no. 63/296,145 filed on January 3, 2022 and United States Provisional Application no. 63/296,144 filed January 3, 2022, both incorporated by reference in their entirety.

TECHNICAL FIELD

[002] In general, this disclosure refers to agricultural machines with autonomous navigation capabilities. Agricultural machines are generally used in agricultural environments to carry out agricultural actions.

BACKGROUND

[003] In general, agricultural machines outperform human operators and various other objects in their environment. With such a large size, these agricultural machines often have difficulty in judging the proximity of objects to the agricultural machines. Therefore, operators must move slowly and very carefully, especially in confined or dense environments. The challenge of navigating these environments is even greater in the case of automated agricultural machinery.

SUMMARY

[004] An agricultural machine is configured to generate and maintain a virtual security bubble that allows the agricultural machine to autonomously perform agricultural actions in the environment safely. The agricultural machine is a navigable vehicle with at least one detection system and one control system, among other components to carry out agricultural actions. The sensing system includes one or more sensing mechanisms, e.g., cameras, for capturing image data of the surrounding environment. The detection system can have a 360-degree field of view, aggregated from the individual fields of view of the detection mechanisms. The control system, for example, a general computing system, controls the operation of the various components. To assist in the safe navigation of the agricultural machine, the control system can generate and maintain a virtual safety bubble based on a configuration of the agricultural machine. The control system can dynamically and automatically adjust the virtual safety bubble as the agricultural machine switches between different configurations and/or performs different agricultural actions. The control system detects when an obstacle breaks the virtual safety bubble. When it is determined that an object has been breached, i.e., is within the virtual security bubble, the control system may terminate or interrupt operations and/or may enact other preventive measures. Preventative measures include redirecting the farm machine around the obstacle, changing a farm machine setting, requesting information from an operator or manager, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

[005] FIG. 1A illustrates an isometric view of an agricultural machine according to one or more embodiments.

[006] FIG. 1B illustrates a top view of the agricultural machine in FIG. 1B, according to one or more embodiments.

[007] FIG. 1C illustrates an isometric view of a second agricultural machine according to one or more embodiments.

[008] FIG. 2 illustrates a block diagram of the system environment for the agricultural machine according to one or more embodiments.

[009] FIG. 3A illustrates an agricultural machine with one or more inherent blind spots based on the positioning of a plurality of sensing mechanisms in accordance with one or more embodiments.

[010] FIG. 3B illustrates an agricultural machine with one or more blind spots caused by components of the agricultural machine that obstruct a partial view of the detection mechanism according to one or more embodiments.

[011] FIG. 4A illustrates a first virtual security bubble around an agricultural machine according to one or more embodiments.

[012] FIG. 4B illustrates the dynamic modification of the virtual security bubble around an agricultural machine, according to one or more embodiments.

[013] FIG. 5 illustrates a flowchart of the method for dynamically modifying the virtual security bubble during operation of the agricultural machine according to one or more embodiments.

[014] FIG. 6 illustrates a verification process of agricultural machine detection systems according to one or more embodiments.

[015] FIG. 7 illustrates a notification that the agricultural machine is establishing the virtual security bubble according to one or more embodiments.

[016] FIG. 8 illustrates a notification transmitted to a client device regarding a detected obstacle, according to one or more embodiments.

[017] FIG. 9 illustrates actions that the agricultural machine can implement when detecting an object in the virtual security bubble, according to one or more embodiments.

[018] FIG. 10 illustrates actions that the agricultural machine can implement when detecting an object in the virtual security bubble, according to one or more embodiments.

[019] FIG. 11 illustrates an agricultural machine navigation workflow according to one or more embodiments.

[020] FIG. 12 illustrates the navigation of an agricultural machine on a straight path, according to one or more embodiments.

[021] FIG. 13 illustrates navigation of an agricultural machine off the path on a straight path, according to one or more embodiments.

[022] FIG. 14 illustrates the navigation of an agricultural machine on a path and off-center of a straight path, according to one or more embodiments.

[023] FIG. 15A illustrates the navigation of an agricultural machine when off the path, but perceived as being on the path, according to one or more embodiments.

[024] FIG. 15B illustrates the navigation of an agricultural machine when on the path, but perceived as being off the path, according to one or more embodiments.

[025] FIG. 16 illustrates the navigation of an agricultural machine when turning on a curved path, according to one or more embodiments.

[026] FIG. 17 illustrates the navigation of an agricultural machine when out of curve on a curved path, according to one or more embodiments.

[027] FIG. 18 illustrates an example of a computing system according to one or more embodiments.

DETAILED DESCRIPTION I. INTRODUCTION

[028] An agricultural machine is configured to generate and maintain a virtual security bubble that allows the agricultural machine to autonomously perform agricultural actions in the environment safely. The agricultural machine is generally a navigable vehicle that comprises a detection system, a treatment mechanism, a verification mechanism, among other components. The sensing system includes one or more sensing mechanisms, e.g., cameras, for capturing image data of the surrounding environment. The detection system can have a 360-degree field of view, aggregated from the individual fields of view of the detection mechanisms. The agricultural machine treatment engine applies the treatment as part of the executable agricultural action(s) and the verification engine verifies that the treatment was successful. The agricultural machine may further comprise a control system, for example, a general computing system, for controlling the operation of the various components.

[029] To assist in the safe navigation of the agricultural machine, the control system can generate and maintain a virtual safety bubble that is activated when an obstacle breaks the virtual safety bubble. When it is determined that an object has been breached, i.e., is within the virtual security bubble, the control system may terminate or interrupt operations and/or may enact other preventative measures. Preventative measures include redirecting the farm machine around the obstacle, changing a farm machine setting, requesting information from an operator or manager, etc.

[030] The control system generates the virtual security bubble based on a configuration of the agricultural machine. As the agricultural machine changes configuration, the control system can automatically and/or dynamically adjust the virtual safety bubble, adjusting the characteristics of the virtual safety bubble. For example, when the agricultural machine accelerates to a higher speed, the control system can automatically and/or dynamically adjust a size of the virtual safety bubble to be larger than before to provide additional distance to enact preventive measures. In another example, the agricultural machine may change its configuration to perform different agricultural actions and the control system may automatically and/or dynamically adjust the parameters of the virtual security bubble in response to the changed configuration.

II. FIELD MANAGEMENT AND TREATMENT PLANS Field Management

[031] Agricultural managers (“managers”) are responsible for managing agricultural operations in one or more fields. Managers work to implement an agricultural objective within these fields and select from a variety of agricultural actions to implement that agricultural objective. Traditionally, managers are, for example, a farmer or agronomist working in the field, but they can also be other people and/or systems set up to manage agricultural operations in the field. For example, a manager can be an automated agricultural machine, a machine-learned computer model, etc. In some cases, a manager may be a combination of the managers described above. For example, a manager may include a farmer assisted by a machine-learned agronomy model and one or more automated agricultural machines, or it may be a farmer and an agronomist working together.

[032] Managers implement one or more agricultural objectives for a field. An agricultural objective is typically a macro-level goal for a field. For example, macro-level agricultural objectives may include treating crops with growth promoters, neutralizing weeds with growth regulators, harvesting a crop with the best possible yield, or any other suitable agricultural objective. However, agricultural objectives can also be a micro-level goal for the field. For example, micro-level agricultural objectives may include treating a specific plant in the field, repairing or fixing a part of an agricultural machine, requesting feedback from a manager, etc. Of course, there are many possible agricultural objectives and combinations of agricultural objectives, and the examples described above are not intended to be limiting.

[033] Agricultural objectives are achieved by one or more agricultural machines that perform a series of agricultural actions. Agricultural machines are described in more detail below. Agricultural actions are any operation implementable by an agricultural machine within the field that works towards an agricultural objective. Consider, for example, an agricultural goal of harvesting a crop with the best possible yield. This agricultural objective requires a series of agricultural actions, for example, planting the field, fertilizing the plants 104, watering the plants 104, weeding the field, harvesting the plants 104, evaluating the yield, etc. Likewise, each agricultural action regarding crop harvesting can be an agricultural objective in its own right. For example, planting the field may require its own set of agricultural actions, e.g. preparing the soil, digging the soil, planting a seed, etc.

[034] In other words, managers implement a treatment plan in the field to achieve an agricultural objective. A treatment plan is a hierarchical set of macro- and/or micro-level objectives that fulfill the manager's agricultural objective. Within a treatment plan, each macro or micro objective may require a set of agricultural actions to be carried out, or each macro or micro objective may be an agricultural action in itself. So to expand, the treatment plan is a temporally sequenced set of agricultural actions to be applied to the field that the manager expects will meet the agricultural objective.

[035] When executing a treatment plan in a field, the treatment plan itself and/or its constituent agricultural objectives and agricultural actions have various outcomes. An outcome is a representation of whether, or how well, an agricultural machine performed the treatment plan, agricultural objective and/or agricultural action. A result can be a qualitative measure, such as “accomplished” or “not accomplished,” or it can be a quantitative measure, such as “18.15 kilograms harvested” or “0.50 hectare treated”. The results can also be positive or negative depending on the configuration of the agricultural machine or the implementation of the treatment plan. Furthermore, results can be measured by sensors on the agricultural machine, entered by managers or accessed from a data store or network.

[036] Traditionally, managers have leveraged their experience, specific knowledge and technical knowledge when implementing agricultural actions in a treatment plan. In a first example, a manager might check weed pressure in various areas of the field to determine when a field is ready for weeding. In a second example, a manager might consult past implementations of a treatment plan to determine the best time to begin planting a field. Finally, in a third example, a manager may rely on established best practices to determine a specific set of agricultural actions to take in a treatment plan to achieve an agricultural objective.

[037] Leveraging manager and historical knowledge to make decisions for a treatment plan affects the spatial and temporal characteristics of a treatment plan. For example, agricultural actions in a treatment plan have historically been applied to the entire field rather than small parts of a field. To illustrate, when a manager decides to plant a crop, he plants the entire field instead of just a corner of the field with the best planting conditions; or, when the manager decides to weed a field, he weeds the entire field instead of just a few rows. Likewise, each farm action in a treatment plan's farm action sequence is historically performed at approximately the same time. For example, when a manager decides to fertilize a field, he fertilizes the field at approximately the same time; or, when the manager decides to harvest the field, he does so at approximately the same time.

[038] Notably, however, agricultural machinery has greatly advanced in its capabilities. For example, agricultural machinery continues to become more autonomous, includes an increasing number of sensors and measurement devices, employs greater amounts of processing power and connectivity, and implements various machine vision algorithms to enable managers to successfully implement a treatment plan.

[039] Due to this increased capacity, managers are no longer limited to spatially and temporally monolithic implementations of agricultural actions in a treatment plan. Instead, managers can leverage the advanced capabilities of agricultural machinery to implement highly localized treatment plans determined by real-time measurements in the field. In other words, instead of a manager applying a “best guess” treatment plan to an entire field, they can implement informed, individualized treatment plans for each plant in the field.

III. AGRICULTURAL MACHINE Overview

[040] An agricultural machine that implements agricultural actions of a treatment plan can have a variety of configurations, some of which are described in greater detail below.

[041] FIG. 1A is an isometric view of an agricultural machine 100 that performs agricultural actions of a treatment plan, in accordance with an example embodiment, and FIG. 1B illustrates a top view of the agricultural machine 100 in FIG. 1A. FIG. 1C is an isometric view of another agricultural machine 100 that performs agricultural actions of a treatment plan, according to one or more embodiments.

[042] The agricultural machine 100 generally includes a detection mechanism 110, a treatment mechanism 120 and a control system 130. The agricultural machine 100 may additionally include an assembly mechanism 140, a verification mechanism 150, a power source , digital memory, communication apparatus or any other suitable component that allows the agricultural machine 100 to implement agricultural actions in a treatment plan. Furthermore, the described components and functions of agricultural machine 100 are examples only and an agricultural machine 100 may have different or additional components and functions in addition to those described below.

[043] Agricultural machine 100 is configured to perform agricultural actions on a field 160 and the implemented agricultural actions are part of a treatment plan. To illustrate, agricultural machine 100 implements an agricultural action that applies a treatment to one or more plants 104 and/or substrate 106 within a geographic area. Here, agricultural treatment actions are included in a treatment plan to regulate plant growth. Thus, treatments are typically applied directly to a single plant 104, but may alternatively be applied directly to multiple plants 104, indirectly applied to one or more plants 104, applied to the environment 102 associated with the plant 104 (e.g., soil, atmosphere or other suitable part of the plant environment adjacent to or connected by environmental factors such as wind), or otherwise applied to plants 104.

[044] In a specific example, agricultural machine 100 is configured to implement an agricultural action that applies a treatment that necrotizes the entire plant 104 (e.g., weeding) or part of the plant 104 (e.g., pruning). In this case, the agricultural action may include dislodging the plant 104 from the supporting substrate 106, incinerating a portion of the plant 104 (e.g., with directed electromagnetic energy such as a laser), applying a working fluid treatment concentration (e.g., , fertilizer, hormone, water, etc.) to plant 104, or treat plant 104 in any other suitable manner.

[045] In another example, agricultural machine 100 is configured to implement an agricultural action that applies a treatment to regulate plant growth. Regulation of plant growth may include promoting the growth of the plant, promoting the growth of a part of the plant, preventing (e.g., retarding) the growth of the plant 104 or part of the plant, or controlling the growth of the plant. Examples of regulating plant growth include applying growth hormone to plant 104, applying fertilizer to plant 104 or substrate 106, applying a disease treatment or insect treatment to plant 104, electrically stimulating plant 104, watering plant 104, pruning plant 104, or otherwise treating plant 104. Plant growth may further be regulated by pruning, necrosis, or other treatment of plants 104 adjacent to plant 104.

Operating Environment

[046] The agricultural machine 100 operates in an operational environment 102. The operational environment 102 is the environment 102 that surrounds the agricultural machine 100 while it implements agricultural actions of a treatment plan. The operating environment 102 may also include the agricultural machine 100 itself and its corresponding components.

[047] The operating environment 102 typically includes a field 160 and the agricultural machine 100 generally implements agricultural actions of the treatment plan in the field 160. A field 160 is a geographic area where the agricultural machine 100 implements a treatment plan. The field 160 may be an outdoor plant field, but it may also be an indoor location that houses plants, such as a greenhouse, a laboratory, a grow house, a set of containers, or any other suitable environment 102.

[048] A field 160 may include any number of field parts. A field part is a subunit of a field 160. For example, a field part may be a part of the field 160 small enough to include a single plant 104, large enough to include many plants 104, or some other size. The agricultural machine 100 can perform different agricultural actions for different parts of the field. For example, agricultural machine 100 may apply a herbicide to some field portions in field 160, while applying a pesticide to another field portion. Furthermore, a field 160 and a field part are largely interchangeable in the context of the methods and systems described herein. That is, treatment plans and their corresponding agricultural actions can be applied to an entire field 160 or to a part of the field, depending on the circumstances at stake.

[049] The operating environment 102 may also include plants 104. In this way, the agricultural actions that the agricultural machine 100 implements as part of a treatment plan may be applied to the plants 104 in the field 160. The plants 104 may be crops, but they can also be weeds or any other suitable plant 104. Some examples of crops include cotton, lettuce, soybeans, rice, carrots, tomatoes, corn, broccoli, cabbage, potatoes, wheat or any other suitable cash crop. Weeds can be grasses, broadleaf weeds, thistles or any other suitable determinant weed.

[050] More generally, plants 104 may include a stem that is disposed superiorly (e.g., above) the substrate 106 and a root system attached to the stem that is located below the plane of the substrate 106 (e.g., below ground) . The stem can support any branches, leaves and/or fruits. Plant 104 may have a single stem, leaf or fruit, multiple stems, leaves or fruits, or any number of stems, leaves or fruits. The root system may be a taproot system or fibrous root system, and the root system may support the position of the plant 104 and absorb nutrients and water from the substrate 106. In various examples, the plant 104 may be a vascular plant 104, plant non-vascular 104, woody plant 104, herbaceous plant 104 or any suitable type of plant 104.

[051] Plants 104 in a field 160 can be grown in one or more rows of plants 104 (e.g., plant beds 104). The rows of plan 104 are normally parallel to each other, but they need not be. Each plant row 104 is generally spaced between 2 inches and 45 inches apart when measured in a perpendicular direction from an axis representing the plant row 104. The plant rows 104 may have wider or narrower spacings or may have variable spacing between multiple rows (e.g., a 12-inch spacing between a first and second row, a 16-inch spacing between a second and third row, etc.).

[052] Plants 104 within a field 160 may include the same type of crop (e.g., same genus, same species, etc.). For example, each field portion in a field 160 may include corn crops. However, the plants 104 within each field 160 may also include multiple crops (e.g., a first crop, a second crop, etc.). For example, some parts of the field may include lettuce crops, while other parts of the field include weeds, or, in another example, some parts of the field may include beans while other parts of the field include corn. Furthermore, a single part of the field may include different types of crops. For example, a single field part may include a soybean plant 104 and a weed.

[053] The operating environment 102 may also include a substrate 106. In this way, agricultural actions that the agricultural machine 100 implements as part of a treatment plan may be applied to the substrate 106. The substrate 106 may be soil, but may, alternatively, be a sponge or any other suitable substrate 106. Substrate 106 may include plants 104 or may not include plants 104 depending on their location in field 160. For example, one part of substrate 106 may include a row of crops, while another part of the substrate 106 between the crop rows does not include plants 104.

III.A EXAMPLES OF MACHINE SETTINGS Detection mechanism(s)

[054] The agricultural machine 100 may include a detection mechanism 110. The detection mechanism 110 identifies objects in the operating environment 102 of the agricultural machine 100. To do this, the detection mechanism 110 obtains information that describes the environment 102 (e.g. , sensor, or image data) and processes this information to identify pertinent objects (e.g., plants 104, substrate 106, people, etc.) in the operating environment 102. Identifying objects in the environment 102 further allows the agricultural machine 100 to implement agricultural actions in field 160. For example, detection engine 110 may capture an image of field 160 and process the image with a plant identification model 104 to identify plants 104 in the captured image. The agricultural machine 100 then implements agricultural actions in the field 160 based on the plants 104 identified in the image.

[055] Agricultural machine 100 may include any number or type of sensing mechanism 110 that may assist in determining and implementing agricultural actions. In some embodiments, the detection mechanism 110 includes one or more sensors. For example, the detection mechanism 110 may include a multispectral camera, a stereo camera, a CCD camera, a single-lens camera, a CMOS camera, hyperspectral imaging system, LIDAR (light detection and ranging system) system, a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor or any other suitable sensor. Additionally, the sensing mechanism 110 may include a sensor array (e.g., a camera array) configured to capture information about the environment 102 surrounding the agricultural machine 100. For example, the sensing mechanism 110 may include an array of cameras configured to capture an array of images representing the environment 102 around the agricultural machine 100. The detection mechanism 110 may also be a sensor that measures a state of the agricultural machine 100. For example, the detection mechanism 110 may be a speed sensor, a heat sensor, or some other sensor that can monitor the state of a component of the agricultural machine 100. Furthermore, the sensing mechanism 110 can also be a sensor that measures components during the implementation of an agricultural action. For example, the sensing mechanism 110 may be a flow rate monitor, a grain harvest sensor, a mechanical stress sensor, etc. Whatever the case, the sensing mechanism 110 detects information about the operating environment 102 (including the agricultural machine 100).

[056] A detection mechanism 110 may be mounted at any point on the mounting mechanism 140. Depending on where the detection mechanism 110 is mounted in relation to the treatment mechanism 120, one or the other may pass through a geographic area in the field 160 before the other. For example, the sensing mechanism 110 may be positioned in the mounting mechanism 140 so that it passes through a geographic location ahead of the handling mechanism 120 as the agricultural machine 100 moves through the field 160. In other examples, the sensing mechanism 110 may be positioned in the mounting mechanism 140. detection 110 is positioned on the mounting mechanism 140 so that the two traverse a geographic location at substantially the same time as the agricultural machine 100 moves through the field. Likewise, the detection mechanism 110 may be positioned on the mounting mechanism 140 so that the treatment mechanism 120 passes through a geographic location before the treatment mechanism 110 as the agricultural machine 100 moves through the field 160. sensing mechanism 110 may be statically mounted to mounting mechanism 140 or may be removably or dynamically coupled to mounting mechanism 140. In other examples, sensing mechanism 110 may be mounted to some other surface of agricultural machinery 100 or may be incorporated into another component of the agricultural machine 100. The detection mechanism 110 may be removably coupled to the agricultural machine 100.

Verification mechanism(s)

[057] The agricultural machine 100 may include a verification mechanism 150. Generally, the verification mechanism 150 records a measurement of the operating environment 102 and the agricultural machine 100 may use the recorded measurement to verify or determine the extent of an implemented agricultural action (i.e. a result of agricultural action).

[058] To illustrate, consider an example in which an agricultural machine 100 implements an agricultural action based on a measurement of the operating environment 102 by the detection mechanism 110. The verification mechanism 150 records a measurement of the same geographic area measured by the detection mechanism detection 110 and where the agricultural machine 100 implemented the given agricultural action. The agricultural machine 100 then processes the recorded measurement to determine the result of the agricultural action. For example, the scanning engine 150 may record an image of the geographic region around a plant 104 identified by the detection engine 110 and treated by a treatment engine 120. The agricultural machine 100 may apply a treatment detection algorithm to the image. recorded to determine the result of the treatment applied to plant 104.

[059] The information recorded by the verification mechanism 150 can also be used to empirically determine the operational parameters of the agricultural machine 100 that will obtain the desired effects of the implemented agricultural actions (e.g., to calibrate the agricultural machine 100, to modify the agricultural treatment, etc.). For example, the agricultural machine 100 may apply a calibration detection algorithm to a measurement recorded by the agricultural machine 100. In this case, the agricultural machine 100 determines whether the actual effects of an implemented agricultural action are the same as its intended effects. If the effects of the implemented agricultural action are different from the intended effects, the agricultural machine 100 may perform a calibration process. The calibration process changes the operational parameters of the agricultural machine 100 so that the effects of future agricultural actions implemented are the same as their intended effects. To illustrate, consider the previous example where agricultural machine 100 recorded an image of a treated plant 104. There, agricultural machine 100 may apply a calibration algorithm to the recorded image to determine whether the treatment is properly calibrated (e.g., in its intended location in the operating environment 102). If the agricultural machine 100 determines that the agricultural machine 100 is not calibrated (e.g., the applied treatment is in an incorrect location), the agricultural machine 100 may calibrate itself so that future treatments are in the correct location. Other exemplary calibrations are also possible.

[060] Verification engine 150 may have various configurations. For example, the verification mechanism 150 may be substantially similar to (e.g., be the same type of mechanism as) the detection mechanism 110 or may be different from the detection mechanism 110. In some cases, the detection mechanism 110 and the scan mechanism 150 may be one in the same (e.g., the same sensor). In an exemplary configuration, the verification mechanism 150 is positioned distal to the detection mechanism 110 with respect to the direction of travel 115 and the treatment mechanism 120 is positioned between them. In this configuration, the verification mechanism 150 traverses a geographic location in the operating environment 102 after the processing mechanism 120 and the detection mechanism 110. However, the assembly mechanism 140 may retain the relative positions of the system components in any other configuration. proper. In some configurations, the checking mechanism 150 may be included in other components of the agricultural machine 100.

[061] Agricultural machine 100 may include any number or type of verification mechanism 150. In some embodiments, verification mechanism 150 includes one or more sensors. For example, scanning engine 150 may include a multispectral camera, a stereo camera, a CCD camera, a single-lens camera, a CMOS camera, hyperspectral imaging system, LIDAR (light detection and ranging system) system, a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor or any other suitable sensor. Additionally, the scanning mechanism 150 may include a sensor array (e.g., a camera array) configured to capture information about the environment 102 surrounding the agricultural machine 100. For example, the scanning mechanism 150 may include an array of cameras configured to capture an array of images representing the operational environment 102.

Treatment mechanism(s)

[062] The agricultural machine 100 may include a treatment mechanism 120. The treatment mechanism 120 may implement agricultural actions in the operating environment 102 of the agricultural machine 100. For example, an agricultural machine 100 may include a treatment mechanism 120 that applies a treatment to a plant 104, a substrate 106, or some other object in the operating environment 102. More generally, the agricultural machine 100 employs the treatment mechanism 120 to apply a treatment to a treatment area 122 and the treatment area 122 may include any thing within the operating environment 102 (e.g., a plant 104 or the substrate 106). In other words, the treatment area 122 can be any part of the operating environment 102.

[063] When the treatment is a plant treatment, the treatment mechanism 120 applies a treatment to a plant 104 in the field 160. The treatment mechanism 120 may apply treatments to identified plants or unidentified plants. For example, the agricultural machine 100 may identify and treat a specific plant (e.g., plant 104) in the field 160. Alternatively, or additionally, the agricultural machine 100 may identify some other trigger that indicates a plant treatment and the treatment mechanism 120 can apply a plant treatment. Some examples of plant treatment mechanisms 120 include: one or more spray nozzles, one or more electromagnetic energy sources (e.g., a laser), one or more physical implements configured to manipulate plants, but other treatment mechanisms 120 of 104 plants are also possible.

[064] Furthermore, when the treatment is a plant treatment, the effect of treating a plant 104 with a treatment mechanism 120 may include any plant necrosis, stimulation of plant growth, necrosis or removal of plant part, stimulation of plant part growth or any other suitable treatment effect. Additionally, treatment mechanism 120 may apply a treatment that dislodges a plant 104 from substrate 106, separates a plant 104 or part of a plant 104 (e.g., cutting), incinerates a plant 104 or part of a plant 104, stimulates electrically a plant 104 or part of a plant 104, fertilize or promote the growth (e.g., with a growth hormone) of a plant 104, water a plant 104, apply light or some other radiation to a plant 104, and/or inject one or more fluids in the substrate 106 adjacent to a plant 104 (e.g., within a threshold distance from the plant). Other plant treatments are also possible. When applying a plant treatment, the treatment mechanisms 120 can be configured to spray one or more of: a herbicide, a fungicide, insecticide, some other pesticide, or water.

[065] When the treatment is a substrate treatment, the treatment mechanism 120 applies a treatment to some parts of the plant 106 in the field 160. The treatment mechanism 120 may apply treatments to identified areas of the substrate 106 or unidentified areas of the substrate 106. For example, the agricultural machine 100 may identify and treat an area of substrate 106 in the field 160. Alternatively, or additionally, the agricultural machine 100 may identify some other trigger that indicates a substrate treatment 106 and the treatment mechanism 120 may applying a treatment to substrate 106. Some examples of treatment mechanisms 120 configured to apply treatments to substrate 106 include: one or more spray nozzles, one or more sources of electromagnetic energy, one or more physical implements configured to manipulate substrate 106, but other mechanisms of treatment 120 of substrate 106 are also possible.

[066] Obviously, the agricultural machine 100 is not limited to treatment mechanisms 120 for plants 104 and substrates 106. The agricultural machine 100 may include treatment mechanisms 120 to apply various other treatments to objects in the field 160.

[067] Depending on the configuration, the agricultural machine 100 may include various numbers of treatment mechanisms 120 (e.g., 1, 2, 5, 20, 60, etc.). A treatment mechanism 120 may be fixed (e.g., statically coupled) to the mounting mechanism 140 or attached to the agricultural machine 100. Alternatively, or additionally, a treatment mechanism 120 may be movable (e.g., translatable, rotatable, etc. ) on the agricultural machine 100. In one configuration, the agricultural machine 100 includes a single treatment mechanism 120. In this case, the treatment mechanism 120 may be operable to align the treatment mechanism 120 with a treatment area 122. In a second variation, the agricultural machine 100 includes a treatment mechanism assembly 120 comprising an array of treatment mechanisms 120. In this configuration, a treatment mechanism 120 may be a single treatment mechanism 120, a combination of treatment mechanisms 120, or the set of the treatment mechanism 120. Thus, a single treatment mechanism 120, a combination of treatment mechanisms 120, or the entire set may be selected to apply a treatment to a treatment area 122. Likewise, the single set, combined or entire can be actuated to align with a treatment area as needed. In some configurations, the agricultural machine 100 may align a handling mechanism 120 with an identified object in the operating environment 102. That is, the agricultural machine 100 may identify an object in the operating environment 102 and actuate the handling mechanism 120 so that its treatment area aligns with the identified object.

[068] A treatment mechanism 120 may be operable between a standby mode and a treatment mode. In the standby mode, the treatment mechanism 120 does not apply a treatment, and in the treatment mode, the treatment mechanism 120 is controlled by the control system 130 to apply the treatment. However, the treatment mechanism 120 may be operable in any other suitable number of operating modes.

[069] The configuration of the processing mechanism 120 may affect the parameters of the virtual security bubble. For example, the handling engine 120 may be collapsed in a compact configuration or deployed in an expanded configuration, and the control system that generates the virtual security bubble may automatically and/or dynamically adjust the parameters of the security bubble based on whether the treatment mechanism 120 is in the collapsed configuration or the expanded configuration. In another example, the treatment mechanism 120 may be operable for multiple agricultural actions. Based on the agricultural action, the control system can automatically and/or dynamically adjust the safety bubble parameters.

Control system

[070] The agricultural machine 100 includes a control mechanism 130. The control system 130 controls the operation of the various components and systems in the agricultural machine 100. For example, the control system 130 may obtain information about the operating environment 102, process this information to identify an agricultural action to be implemented and implement the identified agricultural action with components of the agricultural machine system 100. The control system 130 may further assist in navigating the agricultural machine around the operating environment 102. Navigation may include collect and analyze data related to the environment from one or more sensors and generate navigation instructions based on the data.

[071] The control system 130 may receive information from the detection mechanism 110, the verification mechanism 150, the treatment mechanism 120 and/or any other component or system of the agricultural machine 100. For example, the control system 130 may receive measurements from the detection engine 110 or verification engine 150, or information related to the state of a treatment engine 120 or implemented agricultural actions from a verification engine 150. Other information is also possible.

[072] Likewise, the control system 130 may provide input to the detection mechanism 110, verification mechanism 150, and/or treatment mechanism 120. For example, the control system 130 may be configured for input and control of operational parameters of the agricultural machine 100 (e.g., speed, direction). Likewise, the control system 130 may be configured to input and control operating parameters of the detection mechanism 110 and/or verification mechanism 150. The operational parameters of the detection mechanism 110 and/or verification mechanism 150 may include timing. processing, location and/or angle of detection mechanism 110, image capture intervals, image capture settings, etc. Other inputs are also possible. Finally, the control system may be configured to generate machine inputs to the treatment engine 120. This translates an agricultural action of a treatment plan into machine instructions implementable by the treatment engine 120.

[073] The control system 130 can be operated by a user operating the agricultural machine 100, fully or partially autonomously, operated by a user connected to the agricultural machine 100 by a network, or any combination of the foregoing. For example, the control system 130 may be operated by a farm manager seated in a cab of the farm machine 100, or the control system 130 may be operated by a farm manager connected to the control system 130 via a wireless network. . In another example, control system 130 may implement an array of control algorithms, machine vision algorithms, decision algorithms, etc. that allow it to operate autonomously or partially autonomously.

[074] The control system 130 can be implemented by a computer or a distributed computer system. Computers can be connected in various network environments. For example, the control system 130 may be a series of computers implemented in the agricultural machine 100 and connected by a local network. In another example, the control system 130 may be a series of computers implemented in the agricultural machine 100, in the cloud, a client device, and connected by a wireless area network.

[075] Control system 130 may apply one or more computer models to determine and implement agricultural actions in field 160. For example, control system 130 may apply a plant identification module to images acquired by detection mechanism 110 to determine and implement agricultural actions. The control system 130 may be coupled to the agricultural machine 100 so that an operator (e.g., a driver) can interact with the control system 130. In other embodiments, the control system 130 is physically removed from the agricultural machine 100 and communicates with system components (e.g., detection engine 110, handling engine 120, etc.) wirelessly.

[076] In some configurations, the agricultural machine 100 may additionally include a communication apparatus, which functions to communicate (e.g., send and/or receive) data between the control system 130 and a set of remote devices. The communication apparatus may be a Wi-Fi communication system, a cellular communication system, a short-range communication system (e.g., Bluetooth, NFC, etc.), or any other suitable communication system.

Other Machine Components

[077] In various configurations, agricultural machine 100 may include any number of additional components.

[078] For example, agricultural machine 100 may include a mounting mechanism 140. Mounting mechanism 140 provides a mounting point for components of agricultural machine 100. That is, mounting mechanism 140 may be a chassis or frame to which the components of the agricultural machine 100 may be attached, but may alternatively be any other suitable mounting mechanism 140. More generally, the mounting mechanism 140 statically retains and mechanically supports the positions of the sensing mechanism 110, the treatment 120 and the checking mechanism 150. In an exemplary configuration, the mounting mechanism 140 extends outward from an agricultural machine body 100 so that the mounting mechanism 140 is approximately perpendicular to the direction of travel 115. In some configurations , the mounting mechanism 140 may include an array of handling mechanisms 120 positioned laterally along the mounting mechanism 140. In some configurations, the agricultural machine 100 may not include a mounting mechanism 140, the mounting mechanism 140 may be positioned alternatively, the mounting mechanism 140 may be incorporated into any other component of the agricultural machine 100.

[079] Agricultural machine 100 may include a locomotion mechanism. The locomotion mechanisms may include any number of wheels, continuous tracks, articulated legs, or some other locomotion mechanism. For example, the agricultural machine 100 may include a first set and a second set of coaxial wheels, or a first set and a second set of continuous tracks. In either example, the axis of rotation of the first and second set of wheels/tracks are approximately parallel. Furthermore, each assembly is disposed along opposite sides of the agricultural machine 100. Typically, the locomotion mechanisms are connected to a drive mechanism that causes the locomotion mechanisms to translate the agricultural machine 100 through the operating environment 102. For example, agricultural machine 100 may include a drive train for running wheels or tracks. In different configurations, the agricultural machine 100 may include any other suitable number or combination of locomotion mechanisms and drive mechanisms.

[080] Agricultural machine 100 may also include one or more coupling mechanisms 142 (e.g., a hitch). The coupling mechanism 142 functions to statically or removably couple various components of the agricultural machine 100. For example, a coupling mechanism may attach a drive mechanism to a secondary component such that the secondary component is pulled behind the agricultural machine 100 In another example, a coupling mechanism may couple one or more treatment mechanisms 120 to the agricultural machine 100.

[081] The agricultural machine 100 may additionally include a power source, which functions to power system components, including the detection mechanism 110, control system 130, and treatment mechanism 120. The power supply may be mounted on the mechanism mounting mechanism 140, may be removably coupled to the mounting mechanism 140 or may be incorporated into another component of the system (e.g., located in the drive mechanism). The power source may be a rechargeable power source (e.g., a set of rechargeable batteries), an energy harvesting power source (e.g., a solar system), a fuel-consuming power source (e.g., a fuel cell array or an internal combustion system), or any other suitable energy source. In other configurations, the power source may be incorporated into any other component of the agricultural machine 100.

III.B SYSTEM ENVIRONMENT

[082] FIG. 2 illustrates a block diagram of the system environment 200 for the agricultural machine 100 in accordance with one or more embodiments. In this example, the control system 210 (e.g., control system 130 ) is connected to external systems 220 and an array of machine components 230 via a network 240 within the system environment 200 .

[083] External systems 220 are any system that can generate data representing information useful for determining and implementing agricultural actions in a field. External systems 220 may include one or more sensors 222, one or more processing units 224, and one or more data stores 226. One or more sensors 222 may measure field 160, operating environment 102, agricultural machine 100, etc. . and generate data that represents these measurements. For example, the sensors 222 may include a rain sensor, a wind sensor, a heat sensor, a camera, etc. The processing units 2240 can process measured data to provide additional information that can assist in determining and implementing agricultural actions in the field. For example, a processing unit 224 may access an image of a field 160 and calculate a weed pressure from the image or may access historical weather information for a field 160 to generate a forecast for the field. Data stores 226 store historical information about the agricultural machine 100, the operating environment 102, the field 160, etc. which can be beneficial in determining and implementing agricultural actions in the field. For example, data storage 226 may store results of previously implemented treatment plans and agricultural actions for a field 160, a nearby field, and/or the region. The historical information may have been obtained from one or more agricultural machines (i.e., measuring the result of an agricultural action of a first agricultural machine with the sensors of a second agricultural machine). Additionally, data storage 226 may store results of specific agricultural actions in field 160, or results of agricultural actions performed in nearby fields with similar characteristics. Data storage 226 can also store historical weather, floods, field use, crops, etc. to the countryside and the surrounding area. Finally, data stores 226 may store any information measured by other components in the system 200 environment.

[084] The machine component array 230 includes one or more components 232. The components 222 are elements of the agricultural machine 100 that can perform agricultural actions (e.g., a treatment mechanism 120). As illustrated, each component has one or more input controllers 234 and one or more sensors 236, but a component may include only sensors 236 or only input controllers 234. An input controller 234 controls the function of the component 232. For example, an input controller 234 may receive machine commands via network 240 and actuate component 230 in response. A sensor 226 generates data representing measurements of the operating environment 102 and provides this data to other systems and components within the environment of the system 200. The measurements may be of a component 232, the agricultural machine 100, the operating environment 102, etc. For example, a sensor 226 may measure a configuration or state of component 222 (e.g., a setting, parameter, power load, etc.), measure conditions in the operating environment 102 (e.g., humidity, temperature, etc.), or measure conditions in the operating environment 102 (e.g., humidity, temperature, etc.), , capturing information representing the operating environment 102 (e.g., imagery, depth information, distance information), and generating data representing the measurement(s).

[085] The control system 210 receives information from external systems 220 and the machine component array 230 and implements a treatment plan in a field using an agricultural machine 100. Before implementing the treatment plan, the agricultural machine checks whether it is safe to operate. To do this, the control system 210 receives a notification from a manager that the environment around the agricultural machine is safe for operation and free from obstacles. The control system 210 verifies, using captured images, that there are no obstacles in the environment around the agricultural machine. The control system 210 generates a virtual security bubble for agricultural actions based on a configuration of the agricultural machine. While the agricultural machine is implementing agricultural actions, the control system 210 continually identifies and locates obstacles in the environment. If one of the obstacles is within the virtual security bubble, the control system 210 may stop the operation or enact preventative measures.

[086] The control system 210 includes a security bubble generation module 212, a classification module 214, a security module 216, a navigation module 218, and a user interface module 219. In other embodiments, the 210 control system has more or less modules. In other embodiments, the modules may be variably configured so that the functions of one may be performed by one or more other modules.

[087] The security bubble generation module 212 generates a virtual security bubble for the agricultural machine 100. The virtual security bubble may be a three-dimensional shape around the agricultural machine 100. In other embodiments, the virtual security bubble it may have a belt shape, for example, a wall of a certain height that surrounds the agricultural machine 100. Various other shapes and sizes may be envisaged. The security bubble generation module 212 defines the shape and size of the virtual security bubble based on the configuration of the agricultural machine 100. For example, the security bubble generation module 212 may determine a shape and/or size of the virtual security bubble based on whether the agricultural machine 100 is in a first configuration to navigate to an operational environment or in a second configuration to carry out a treatment plan. The security bubble generation module 212 can dynamically adjust the virtual security bubble based on sensor data. For example, the security bubble generation module 212 may increase the size of the virtual security bubble in response to a sunset that darkens the operating environment.

[088] The classification module 214 classifies the objects in the images captured by the cameras (sensor modality 222) implemented in the agricultural machine 100. The classification module 214 can use one or more models to classify pixels related to objects in the image. A model can identify obstacles as objects that are not part of the farming operation. For example, the model might classify rows of crops as non-obstacles, but would classify a wild fox or a large rock as an obstacle. Another model can perform image segmentation, classifying pixels for various types of objects, for example, the ground, sky, foliage, obstacles, etc. Still another model may calculate a velocity of objects relative to the agricultural machine 100, for example, using one or more visual odometry methods. And yet another model can predict the depth of camera objects, for example, using a depth estimation model trained to predict depth based on image data. Depth generally refers to the distance between the agricultural machine and the pixels or objects in the images. For example, a first object present in an image can be determined at a depth of 5 meters from the agricultural machine. The classification module 214 can further generate 3D point cloud representations of objects within a virtual operating environment, enabling object tracking. The various models may input other sensor data (captured by sensors 222 or sensors 236) to aid in classification, e.g., LIDAR data, temperature measurements, etc.

[089] The security module 216 evaluates whether obstacles are within the virtual security bubble. The safety module 216 may utilize a depth estimation model to predict obstacle depths relative to the agricultural machine 100. If an obstacle has a depth below the virtual safety below, i.e., some part of the obstacle breaches the safety bubble barrier. virtual security, then the security module 216 will provide this warning to the navigation module 218, for example, to stop the operation or enact preventive measures.

[090] The navigation module 218 navigates the agricultural machine 100. The navigation module 218 generates navigation instructions based on a treatment plan. The treatment plan may include one or more agricultural operations to be completed. The navigation module 218 may plot a route to navigate the vehicle. The navigation module 218 can adjust the navigation route based on sensor data. The navigation module 218 may receive warnings from the safety module 216 that an obstacle has breached the virtual safety bubble. In response to the warning, navigation module 218 may cease operations, enact other preventative measures, or some combination thereof. In an example of a preventative measure, the navigation module 218 may stop the agricultural machine 100 when a warning is given that an obstacle has breached the virtual safety bubble. As another example of a preventive measure, the navigation module 218 may plot a route around the obstacle to avoid collision. Additional details relating to navigation by navigation module 218 are described in FIGS. 11-18.

[091] The user interface module 219 maintains a graphical user interface (GUI) for displaying information to the manager of the agricultural machine 100 and receiving input from the manager. The user interface module 219 may graphically illustrate the agricultural machine 100 in operation, for example, when moving along a path or when performing one or more agricultural actions. The GUI can also display any obstacles or other objects in the operating environment. The GUI may further be configured to receive inputs for controlling the agricultural machine 100. Exemplary inputs include switching a speed for the agricultural machine 100, manual adjustment of the virtual safety bubble, etc. In one embodiment, the GUI may notify a manager of the agricultural machine 100 that an obstacle has breached the virtual security bubble, the GUI may request action or input from management on how to respond. Examples of user interfaces are further described in FIGs. 6-10.

[092] In one or more embodiments, models used by control system 110 may be trained as machine learning models using training data. Training can be supervised, unsupervised or semi-supervised. Various types of machine learning model architectures can be implemented, for example, neural networks, decision trees, support vector machine learning, etc.

[093] Network 240 connects nodes of system environment 200 to allow microcontrollers and devices to communicate with each other. In some embodiments, the components are connected within the network as a Controller Area Network (CAN). In this case, within the network each element has an input and output connection, and the network 250 can translate information between the various elements. For example, network 250 receives input information from camera array 210 and component array 220, processes the information, and transmits the information to control system 230. Control system 230 generates an agricultural action based on the information and transmits instructions for implementing the agricultural action to the appropriate component(s) 222 of the component matrix 220.

[094] Additionally, the environment of system 200 may be other types of network environments and include other networks or a combination of network environments with multiple networks. For example, the environment of system 200 may be a network such as the Internet, a LAN, a MAN, a WAN, a wired or wireless mobile network, a private network, a virtual private network, a direct communication line, and the like.

IV. OBSTRUCTED VIEWS AND UNOBSTRUCTED VIEWS

[095] As described above, an agricultural machine is configured with one or more sensing mechanisms (“sensing system”) to measure the environment. In one configuration, the detection system may be an array of detection mechanisms configured to capture images of the environment. The image data in the image represents the various objects in the environment around the agricultural machine. Thus, the detection system is configured to capture image data from the environment.

[096] The detection system has a field of view, and because the detection system is an array of detection mechanisms, the detection system's field of view may include multiple fields of view that may be composited to form a view of 360 degrees. That is, each detection mechanism has its own field of view, and the fields of view, together, form the field of view of the detection system.

[097] There may be one or more blind spots in a field of view caused by the configuration of the detection system. Some blind spots may include areas outside the range of any detection mechanism and obstructed views, for example, views within the field of view of the detection system but obstructed by one or more objects. Obstructed views comprise image data in images where an object obstructs an object or objects behind it (so that the obstructed objects are obscured from view). Unobstructed views comprise image data in images where no object obstructs an object or objects behind it. For example, consider a detection mechanism capturing images of a tire attached to agricultural machinery and the surrounding environment. Because the tire is obscuring the image data of objects behind the tire (e.g. soil, rocks, etc.), it is an obstructed view. The rest of the image is an unobstructed view because there are no objects obscuring other objects.

[098] Obstructed views are problematic in autonomous farming due to their inherent safety issues. For example, an object that may be a significant obstacle may be obscured by another object in an obstructed view. The agricultural machine may therefore be unable to identify and explain a problematic obstacle. Methods are presented in this document for establishing a virtual safety bubble to prevent obstacles from entering obstructed views of the agricultural machine.

[099] FIG. 3A illustrates an agricultural machine 300 (an embodiment of the agricultural machine 100) pulling an implement 305 and equipped with a detection system. The agricultural machine 300 has a detection system with a total of six detection mechanisms 310. Three detection mechanisms 310A, 310B and 310C are positioned at a front end of the agricultural machine 300, with the remaining three detection mechanisms 310D, 310E, 310F positioned at a rear end of the agricultural machine 300 toward the implement 305. As noted above, the field of view of the detection system may aggregate the individual fields of view 315 of the detection mechanisms. The 310A detection mechanism has a 315A field of view. The 310B detection mechanism has a 315B field of view. The 310C detection mechanism has a 315C field of view. The 310D detection mechanism has a 315D field of view. The 310E detection mechanism has a 315E field of view. The 310F detection engine has a 315F field of view.

[0100] FIG. 3A also illustrates two obstacles. The first obstacle 320 is near the right front tire of the farm machine. The second obstacle 330 is in front right of the farm machine. The agricultural machine 300 is configured to apply an obstacle detection model to images captured by the detection mechanism to identify obstacles in the environment. That is, the agricultural machine 300 employs the obstacle detection model to determine which pixels in the images represent obstacles and locates the approximate location of these obstacles in the environment (e.g., estimating depth).

[0101] As described above, the detection system of agricultural machine 300 includes several blind spots. Blind spots are areas in the environment not visible to the agricultural machine because, for example, a part of the agricultural machine obstructs the view (e.g. behind a tire), or detection mechanisms are not positioned to capture that part of the environment ( for example, under the tractor).

[0102] FIG. 3B illustrates one or more blind spots of the agricultural machine in FIG. 3A. Blind spots are indicated by polygons. The first blind spot 340A may be a combination of the footprint of the agricultural machine 300 (including vehicle frame, tires, other components that obstruct views) and spaces between the sensing mechanisms (e.g., a space to the left between the sensing mechanisms 310B and 310E and another space to the right between detection mechanisms 310C and 310F). The second blind spot 340B may be an obstructed view caused by the agricultural implement 305 having some height blocking views.

[0103] FIG. 3B also illustrates the two obstacles. Here, the first obstacle 320 is partially in the blind spot 340A and the second obstacle 330 is in the unobstructed view and within the field of view of the detection system. Thus, when applying the obstacle detection model to the images captured by the detection system, the agricultural machine would not be able to identify and locate the first obstacle 320, but would be able to identify and locate the second obstacle 330. V. OBSTACLE-FREE CHECK

[0104] The agricultural machine can be configured to begin implementing agricultural actions autonomously only if a manager of the agricultural machine checks the environment. That is, an agricultural machine manager must verify that there are no obstacles in the obstructed and/or unobstructed views of the agricultural machine. In essence, the manager must go around the agricultural machine to check that there are no obstacles in areas undetectable by the detection system. In some configurations, the verification process may include sounding sirens and flashing lights to make it clear that the agricultural machine is about to begin farming autonomously. Lights and sirens make it more likely that any humans in the room will leave the room.

[0105] As part of the verification process, the agricultural machine may communicate with a control system operated by the manager. In other words, the agricultural machine can transmit and receive information from a control system operated by a manager. For example, the agricultural machine may transmit a request to the manager to check the environment, and the agricultural machine may receive an environment check in response (once the manager checks the environment). SAW. GENERATION OF A VIRTUAL SAFETY BUBBLE

[0106] The agricultural machine includes a virtual security bubble generation module configured to generate a virtual security bubble. A virtual safety bubble is an area in the environment that allows the agricultural machine to operate autonomously without colliding with obstacles. A virtual security bubble may be an area in the environment (1) directly around the farm machine, (2) in a path forward of the farm machine, (3) in a path behind the farm machine, (4) along of an expected path of the agricultural machine and/or some area in the environment.

[0107] The agricultural machine generates the virtual security bubble based on the configuration of the agricultural machine. Here, “configuration” is a term used to describe various aspects of the agricultural machine, implement and environment that can be used to generate the virtual security bubble. The following is a non-exhaustive list of aspects of farm machine configuration that may affect the virtual security bubble.

[0108] Machine Path. The machine path can describe a current path of a machine or an expected path of the machine. The machine path can be in any direction relative to the current position of the agricultural machine. In addition, the virtual security bubble for the machine path can consider the characteristics of the agricultural machine. For example, the virtual safety bubble for a large agricultural machine along the machine's path is larger than that of a smaller agricultural machine.

[0109] Speed. The speed can be a current or programmed speed of the agricultural machine. As implemented by the agricultural machine, the speed can be scalar or vector.

[0110] Acceleration. Acceleration can be a current or programmed acceleration of the agricultural machine. As implemented by the agricultural machine, acceleration can be scalar or vector.

[0111] Expected Obstacle Characteristics. Expected obstacle characteristics are characteristics of obstacles that an agricultural machine can expect to encounter in its environment. For example, an agricultural machine operating near a building might expect to encounter different obstacles than one operating in a field. This way, each environment can have correspondingly different virtual security bubbles.

[0112] Implement Type. The implement type is the type of implement being employed by the agricultural machine (if any). As an example, an implement can be a motor cultivator, a seeder, a sprayer, etc. Thus, each implementation can indicate parameters for its virtual security bubbles.

[0113] Type of Mounting Mechanism. The type of mounting mechanism describes how various parts of the agricultural machine are attached to the structure. For example, a mounting mechanism may be a hitch, and the hitch may be a movable hitch or a static hitch. Consequently, the type of mounting mechanism can indicate parameters for the virtual security bubble.

[0114] Type of Agricultural Shares. Agricultural actions are described in detail above. Different agricultural stocks may indicate different parameters for the virtual security bubble. For example, a virtual safety bubble for spraying weeds may be different than a virtual safety bubble for plowing a field. The agricultural machine control system may determine the direction that an agricultural action would face to assist in determining the parameters of the virtual security bubble (e.g., the shape and size of the virtual security bubble). For example, the control system may define a virtual security bubble format to be predominantly in front of the agricultural machine based on the agricultural action being cultivated. In another example, the control system may define a virtual safety bubble shape with a radius around the treatment mechanism in a spray mode. The control system may also access a physical configuration of the agricultural machine based on the agricultural action being performed, for example, a first agricultural action may place the agricultural machine in a first physical configuration, while a second agricultural action may place the machine in a physical configuration. agriculture in a second physical configuration that is different from the first physical configuration.

[0115] Characteristics of Implementation of Agricultural Actions. Implementation characteristics describe the details of how an agricultural machine implements an agricultural action. Some characteristics may include, for example, the spray speed of a spray nozzle, the height of a combine head, etc.

[0116] Machine Characteristics for Agricultural Machinery. Machine characteristics describe the physical manifestation of the agricultural machine. That is, the size, shape and spatial characteristics of the agricultural machine. The agricultural machine can store a digital representation of its machine characteristics that can be accessed by generating a virtual security bubble.

[0117] Implement Characteristics for Implement. Implement characteristics describe the physical manifestation of the agricultural implement. That is, the size, shape and spatial characteristics of the agricultural implement. The agricultural implement can store a digital representation of the characteristics of its implement that can be accessed by generating a virtual security bubble.

[0118] Characteristics of Other Attachments. Other attachments may include any component that is attached to the agricultural machine or implement. For example, the agricultural machine can be equipped with additional reflectors that can expand the dimensional profile of the agricultural machine.

[0119] Environmental Characteristics. Environmental characteristics describe the working environment of the agricultural machine. Some examples of environmental characteristics include the size, shape, and spatial characteristics of the field in which the agricultural machine operates. Features of the environment can also describe the climate.

[0120] Type of Obstacle. Obstacles can be dynamic (i.e., moving) or static (i.e., immobile). The agricultural machine can generate a different virtual safety bubble for an identified dynamic and/or static obstacle.

[0121] Manager Entry. Manager input is manager information that can be used to generate a virtual security bubble. Manager input can include any of the previously mentioned configuration information.

[0122] Local Regulations. The control system can keep a record of different local regulations depending on the geographical location of the agricultural machine. In one or more examples, a first country may have different regulations than a second country; a first state may have different regulations than a second state; a first city may have different regulations than a second city; or some combination thereof. Different regulations may limit agricultural actions, for example, speed limit, permitted period of operation, permitted climate for operation, other regulations, etc.

[0123] To update, the agricultural machine uses a machine configuration to determine a virtual security bubble around the agricultural machine. The machine configuration can be any of the configuration information mentioned above. The virtual security bubble can be represented as a relative distance, an absolute distance, a depth, a time (e.g. based on speed and/or acceleration), legal requirements, or any other suitable metric to quantify the virtual security bubble .

[0124] The agricultural machine continuously monitors the environment so that no obstacles are within the virtual security bubble. In other words, the detection mechanisms capture images, the agricultural machine applies an obstacle identification model to the images and identifies and locates an obstacle in the environment. If the agricultural machine identifies an obstacle in the virtual security bubble, it terminates the operation of the agricultural machine.

[0125] Notably, the agricultural machine can treat obstacles and objects in different ways. For example, an agricultural machine could identify a large pile of leaves in a virtual security bubble, identify it as an object, and continue performing agricultural actions because the leaves would not damage the agricultural machine upon contact. On the contrary, an agricultural machine can identify a log in a virtual safety bubble, identify it as an object, classify it as an obstacle, and fail to perform agricultural actions because the log would damage the agricultural machine in a collision.

[0126] In some examples, the agricultural machine may treat different types of obstacles in different ways. For example, a dynamic obstacle (e.g. a human being, a moving car, etc.) can guarantee different virtual safety bubbles compared to a static obstacle (e.g. a log, a chair, etc.). Naturally, dynamic obstacles likely indicate larger virtual safety bubbles because of their ability to move through the environment, while static obstacles likely indicate smaller virtual safety bubbles because they remain stationary. In some examples, the agricultural machine may treat humans differently than all other obstacles. For example, the agricultural machine can generate a virtual bubble of safety for humans that is larger than all other objects and obstacles. In one or more embodiments, the agricultural machine can generate a plurality of simultaneously used virtual security bubbles. A first virtual security bubble may be defined for a first class of objects (e.g., humans) and a second virtual security bubble may be defined for a second class of objects (e.g., obstacles).

[0127] FIGS. 4A and 4B illustrate the dynamic modification of the virtual security bubble around an agricultural machine. In FIGS. 4A and 4B, the agricultural machine is configured similarly to the agricultural machine in FIG. 3A, having a detection system comprising at least six detection mechanisms that compose fields of view to create the 360 degree field of view around the agricultural machine.

[0128] FIG. 4A illustrates a first virtual security bubble 420 around an agricultural machine according to one or more embodiments. The agricultural machine is autonomously implementing agricultural actions in the environment (e.g. cultivation) using a specific configuration. The configuration of the agricultural machine describes its spatial characteristics (e.g. positions of tires, implements, etc.), the agricultural actions to be implemented, and the implementation characteristics of these agricultural actions (e.g. type, path, speed, acceleration, settings implementation, etc.).

[0129] Based on the configuration, the agricultural machine determines a virtual security bubble 420. The virtual security bubble 420 is represented by the oval surrounding the agricultural machine. The virtual security bubble 420 represents a safe operational area where there are no identified obstacles. To maintain the virtual security bubble 420, the agricultural machine continuously captures images and applies an obstacle detection model to locate obstacles in the environment. If the agricultural machine detects an obstacle 430 in the virtual security bubble 420, the agricultural machine may terminate the operation. In other words, the agricultural machine stops carrying out agricultural actions and can interrupt movement. Implementing the virtual safety bubble beyond the blind spots prevents any obstacle from being obscured and missed by the detection system, which could cause a collision and damage to the agricultural machine, object or individual.

[0130] FIG. 4B illustrates the dynamic modification of the virtual security bubble around an agricultural machine, according to one or more embodiments. To illustrate, imagine that the agricultural actions implemented by the agricultural machine in FIG. 4A are performed with a first configuration with the virtual security bubble 410. For example, the agricultural machine is performing agricultural actions at a first speed in the environment. Now consider, for example, the agricultural machine changing configuration. For example, the manager instructs the agricultural machine to move faster through the environment as it implements agricultural actions. As another example, the agricultural machine enters an environment inhabited by cattle or other animals. The agricultural machine can switch from the first configuration to a second configuration, creating a larger virtual security bubble.

[0131] The agricultural machine calculates a new virtual security bubble 420 to account for the second configuration. The new virtual security bubble 420 is illustrated by the dashed oval in FIG. 4B. Notably, the new virtual security bubble 420 is larger than the original virtual security bubble, for example, to account for the higher speed of the agricultural machine. This larger virtual safety bubble 420 may be generated for several reasons, one of which is to allow for greater time needed to implement corrective action when an obstacle is detected due to increased speed.

[0132] To illustrate, recall obstacle 430 in FIG. 4A. There, the obstacle 430 was outside the virtual security bubble 410 and the agricultural machine did not stop operating. In FIG. 4B the agricultural machine is moving faster. If the new virtual safety bubble 420 is the same size as the original virtual safety bubble 410, the agricultural machine may collide with the obstacle 430 before being able to take corrective action. However, because the virtual safety bubble 420 is larger than the original virtual safety bubble 410, the agricultural machine identifies the obstacle 430 and can take corrective action before colliding with the obstacle 430. VII SAFETY BUBBLE WORKFLOW EXAMPLE DYNAMICS

[0133] The agricultural machine can be configured to generate a virtual security bubble around the agricultural machine that allows for the safe and autonomous implementation of agricultural actions. FIG. 5 illustrates a process flow for generating a virtual security bubble, according to an exemplary embodiment. Although FIG. 5 is described from the perspective of the agricultural machine, any component of the agricultural machine can perform one or more of the steps (e.g., control system 130 or 210). In other embodiments, there may be more or fewer steps. In other embodiments, the listed steps may occur in a different order.

[0134] For context, an autonomous agricultural machine is configured with a detection system. The detection system may include six cameras positioned around the agricultural machine that provide the agricultural machine with a 360-degree field view of the environment. Within the field of view are obstructed views and unobstructed views. Obstructed views are image data within the field of view where an object in the environment obscures parts of the environment behind the object from the detection mechanism (for example, behind a tire or under the cabin). Unobstructed views are image data within the field of view that is not obstructed.

[0135] The agricultural machine receives a notification to begin autonomous implementation of agricultural actions in the environment. In response, the agricultural machine transmits a request to verify that the agricultural machine's operating environment is safe. The check may include transmitting a notification to the manager to verify that there are no obstacles in the obstructed views of the detection system. The manager checks that there are no obstacles and transmits a notification to the agricultural machine reflecting the check.

[0136] The agricultural machine receives 510 a notification that there are no obstacles in the blind spots of the detection system. The manager can provide this notification, for example, through a GUI running on a mobile phone application.

[0137] The agricultural machine checks 520 that there are no obstacles in unobstructed views of the environment using an obstacle detection model. That is, the agricultural machine captures one or more images of the environment using the detection system and applies an obstacle detection model to the images. The obstacle detection model analyzes images to determine whether any of the pixels in the image represent an obstacle.

[0138] The agricultural machine receives 530 instructions from the farmer to begin carrying out agricultural actions autonomously in the field. In an exemplary configuration, the agricultural machine may be unable to initiate autonomous performance without a verification by the manager that there are no obstacles in the obstructed views and verification (himself) that there are no obstacles in the unobstructed views.

[0139] The agricultural machine determines 540 a configuration of the agricultural machine to perform the prescribed agricultural actions in the environment. Determining the configuration may include accessing an implement capability, a computer model of the farm machine, types of farm actions, and implementation characteristics that define how the farm machine implements the farm actions (e.g., speed, path, etc.).

[0140] The agricultural machine determines 550 a virtual security bubble based on the determined configuration. The virtual safety bubble represents an area around the agricultural machine where if an obstacle is detected in the area, the agricultural machine will stop operation. The virtual security bubble can be a distance, a time, a depth, a relative position or any other measurement of a virtual security bubble.

[0141] The agricultural machine detects 560 an obstacle in the environment based on applying the obstacle detection model to the images captured by the detection system. As the agricultural machine performs agricultural actions in the field, the sensing mechanism continuously captures images of the environment. Furthermore, the agricultural machine continuously applies the obstacle detection model to the captured images to identify obstacles in the environment.

[0142] The agricultural machine determines 570 that the obstacle is entering the virtual security bubble. The agricultural machine can determine that the obstacle has breached the virtual safety bubble if the depth of the obstacle is at or below the virtual safety bubble. Depth can be determined using a detection and ranging sensor or a depth estimation model applied to images.

[0143] In response to determining that an obstacle is in the virtual security bubble, the agricultural machine terminates operation 560. That is, the agricultural machine stops performing agricultural actions in the field. In other embodiments, the agricultural machine may enact other preventive measures in response to detecting an obstacle that has breached the virtual security bubble. VIII. EXAMPLES OF INTERACTIONS WITH THE MANAGER

[0144] As described above, the agricultural machine can interact with a manager when performing agricultural actions in the field. Some of these interactions can be activated when the agricultural machine detects an object in its virtual security bubble. Once detected, the agricultural machine can transmit or receive information from an agricultural machine manager. The agricultural machine can also transmit and receive information by establishing a virtual security bubble around the agricultural machine. FIGS. 6 to 10 illustrate several examples of a client device that interacts with an agricultural machine.

[0145] FIG. 6 illustrates a verification process of the agricultural machine's detection systems. The verification process may include checking that there are no visible obstacles in obstructed views of the agricultural machinery. In the left panel, the GUI illustrates the agricultural machine and implement with six zones where cameras are positioned and aimed. The GUI asks the manager to “walk around the machine to validate the cameras”. As the manager physically walks through the agricultural machine, each of the sensing mechanisms (e.g., cameras) can capture data that is used by the agricultural machine to validate the ability of the sensing mechanisms to detect the manager. The right pane shows a complete tour with checkmarks next to each detection mechanism (e.g. camera).

[0146] FIG. 7 illustrates a notification that the agricultural machine is establishing the virtual security bubble. In other words, once implemented, the virtual security bubble will be maintained according to the methods described above. Therefore, if a human or object enters the virtual security bubble, the agricultural machine can perform corresponding actions as described above. The left panel shows a slider 710 that allows a manager to engage the agricultural machine in agricultural actions. Sliding slider 710 to the right is an embodiment of step 530 in FIG. 5 to provide and receive instructions to start agricultural activities autonomously.

[0147] FIG. 8 illustrates a notification transmitted to a client device about a detected obstacle. Notification can occur when the object is detected within the virtual security bubble. The notification may include characteristics that describe the detected object. In the left pane, an obstacle notification 810 is displayed as a pop-up notification on a mobile device. Upon receiving a click from the manager, the mobile application can expand to provide a detailed obstacle report 820, shown in the middle panel, providing additional details about the detected obstacle. The detailed obstacle report 820 may include an option to access a video feed of the obstacle 830 captured by a detection engine, shown in the right panel. The detailed obstacle report 820 may further include preventive measures that can be carried out by the agricultural machine.

[0148] FIG. 9 illustrates actions that the agricultural machine can implement when detecting an object in the virtual security bubble. For example, the agricultural machine can bypass the object in the field. The GUI can illustrate a route around the obstacle and the farm machine's progress in navigating the route, shown in the left panel. After route completion, the GUI can notify the manager of successful routing around the obstacle, shown in the middle panel. The right panel illustrates another example screenshot that illustrates an alternative route around an obstacle with an actionable option to instruct the farm machine to enact the countermeasure of navigating around the obstacle.

[0149] FIG. 10 illustrates actions that the agricultural machine can implement when detecting an object in the virtual security bubble. For example, agricultural machinery may cease operation in the field. In the left panel, the GUI illustrates that the agricultural machine has ceased operations (paused) in response to the detection of an obstacle. In the middle panel, the GUI indicates that the agricultural machine is shutting down after being “idle for 30 minutes” following a pause due to obstacle detection. In the right pane, the GUI indicates that the agricultural machine is “shut down”, for example, switching to an idle state. IX. EXAMPLE NAVIGATION WORKFLOW

[0150] FIG. 11 illustrates an agricultural machine navigation workflow 1100 according to one or more embodiments. The agricultural machine may implement the control system 210 as described in FIG. 2. In other embodiments, navigation workflow 1100 may include more steps, fewer steps, steps in a different order, or some combination thereof. Although the following description is from the perspective of the control system 210, the general agricultural machine can also perform the navigation workflow (e.g., through distributed systems in contrast to a control system).

[0151] Control system 210 begins by detecting objects in an operating environment of the agricultural machine. The control system 210 utilizes a space engine 1105 that generates an occupancy grid 1110. The occupancy grid 1110 is a virtual representation of the spatial environment of the agricultural machine. The control system 210 may further utilize a route motor 1120 that generates an active path 1125 for the agricultural machine to travel. The control system 210 may further receive GPS coordinates 1130, for example, from a GPS receiver. The control system 210 performs passive mapping 1115, detecting objects 1135 in the environment of the agricultural machine. The control system 210 performs object tracking 1140, for example, by constantly updating a position of an object relative to the agricultural machine within the occupancy grid 1110.

[0152] In one or more embodiments, the control system 210 may utilize object tracking 1140 to determine whether an object may have entered a blind spot. The control system 210 can track an object present in a plurality of images. Upon determining that the object has disappeared from view, i.e., is no longer present in any of the images, the control system 210 may determine that the object has entered a blind spot. In other embodiments, the control system 210, knowing that an object is likely to be in a blind spot, may prompt a user to check whether the object has been released or remains in the blind spot. In response to the user providing input indicating that the object has been released, so control system 210 can continue operation 1185. In response to user providing input indicating that the object remains in the blind spot, control system 210 can redirect. Control system 210 may request additional input from the manager via step 1155.

[0153] The control system 210 detects an obstacle in the active path 1145. As noted, the control system 210 may utilize a virtual safety bubble to detect when obstacles have breached the virtual safety bubble. In response to detecting that the obstacle has breached the virtual security bubble, the control system 210 stops 1150 operations (or enacts other preventative measures). The control system 210 notifies 1155 the manager of the obstacle in the path (e.g., as shown in FIGS. 6 to 10). The control system 210 receives input 1160 from the manager, for example, to approve 1165 of the object, i.e., to replace the object as no longer an obstacle, allowing operation 1185 to continue. Otherwise, the manager may provide input to redirect 1175 In response, control system 210 may redirect path 1180 around the obstacle. Once released, the control system 210 can continue 1185 agricultural actions.

[0154] In one or more embodiments, the control system 210 may routinely update the bounding boxes of objects. The control system 210 may routinely evaluate whether a bounding box for an object is accurately defined for the object. If not precisely defined, the control system 210 may implement Spark AI 1194 to produce corrected bounding boxes 1196 for the various objects. Having accurate bounding boxes increases detection accuracy, i.e. when object detection breaks the virtual security bubble. X. EXAMPLES OF NAVIGATION SCENARIO

[0155] FIG. 12 illustrates navigation of an agricultural machine 1210 along the path on a straight path 1230, according to one or more embodiments. The agricultural machine 1210 is an embodiment of the agricultural machine 100 that comprises the control system 210. The agricultural machine 1210 generates the virtual safety bubble 1220 to assist the navigation of the agricultural machine 1210. As the agricultural machine 1210 is driving on the path 1230 and encounters an obstacle 1240 (i.e., the obstacle 1240 breaches the virtual security bubble 1220), then the agricultural machine 1210 may halt operations and/or enact other preventive measures.

[0156] FIG. 13 illustrates navigation of an agricultural machine 1310 off the path on a straight path 1330, according to one or more embodiments. The agricultural machine 1310 is an embodiment of the agricultural machine 100 that comprises the control system 210. The agricultural machine 1310 generates the virtual security bubble 1320 to assist navigation of the agricultural machine 1310. In this scenario, the agricultural machine 1310 is significantly outside the path. If the agricultural machine 1310 determines that it is off path, then the agricultural machine 1310 may generate course correction navigation instructions to route the agricultural machine 1310 back onto path 1330. The agricultural machine 1310 may also stop operations and/or or provide a notification to a manager indicating that the 1310 farm machine is out of the way, requesting subsequent instructions. Even out of the way, if the agricultural machine 1310 encounters an obstacle 1340 (i.e., the obstacle 1340 breaks the virtual security bubble 1320), then the agricultural machine 1310 may cease operations and/or enact other preventive measures.

[0157] FIG. 14 illustrates navigation of an agricultural machine 1410 on a path and off-center of a straight path 1430, according to one or more embodiments. The agricultural machine 1410 is an embodiment of the agricultural machine 100 that comprises the control system 210. The agricultural machine 1410 generates the virtual security bubble 1420 to assist navigation of the agricultural machine 1410. In this scenario, the agricultural machine 1410 is in the way, but outside the center. If the agricultural machine 1410 determines that it is off-center, then the agricultural machine 1410 may generate course correction navigation instructions to route the agricultural machine 1410 back to the center of the path 1430. Even off the path, if the agricultural machine 1410 encounters an obstacle 1440 (i.e., the obstacle 1340 breaches the virtual security bubble 1420), then the agricultural machine 1410 may cease operations and/or enact other preventative measures.

[0158] FIG. 15A illustrates the navigation of an agricultural machine 1510 when off the path, but perceived as being on the path, according to one or more embodiments. Agricultural machine 1510 is an embodiment of agricultural machine 100 comprising control system 210. Agricultural machine 1510 may receive GPS coordinates such that a perceived position 1515 of agricultural machine 1510 is on the path, i.e., path 1530. However, in fact, the 1510 agricultural machine is out of the way. The agricultural machine 1510 utilizes the virtual security bubble 1520, but will only enact preventative measures when the obstacle 1540 (which is out of the way) enters the virtual security bubble 1520. Obstacles that are in the actual path 1530 may not breach the security bubble virtual 1520, so that the agricultural machine continues operations. In some embodiments, the agricultural machine 1510 may receive corrected GPS coordinates locating the agricultural machine 1510 off the path, although previously perceived to be in the path, at which point the agricultural machine 1510 may generate and enact course correction navigation to navigate the agricultural machine 1510 back to path 1530.

[0159] FIG. 15B illustrates the navigation of an agricultural machine 1510 when on the path, but perceived as being off the path, according to one or more embodiments. This scenario is the opposite of the scenario in FIG. 15A. If the agricultural machine 1510 encounters the obstacle 1550 on the path 1530, although it is perceived to be off the path, e.g., the perceived obstacle 1555 is not on the path 1530, the agricultural machine 1510 will enact preventive measures.

[0160] FIG. 16 illustrates navigation of an agricultural machine 1610 when turning on a curved path 1630, according to one or more embodiments. When turning refers to the perceived turning curvature of the control system corresponding to the target turning curvature to remain on the curved path 1630 when making the turn. Deviation refers to the perceived turning curvature of the control system rotatably displaced from the target turning curvature to remain on the curved path 1630. Agricultural machine 1610 is an embodiment of agricultural machine 100 comprising control system 210. When in a curved path 1630, the agricultural machine 1610 may adjust the virtual safety bubble 1620 to take into account the turning radius of the agricultural machine 1610. For example, the virtual safety bubble 1620 may be extended in a direction of rotation of the agricultural machine 1610. When the agricultural machine 1610 detects that one or more obstacles 1640 and 1650 are within the virtual security bubble 1620, the agricultural machine 1610 may enact preventive measures.

[0161] FIG. 17 illustrates navigation of an agricultural machine when off-curve on a curved path 1730, according to one or more embodiments. Agricultural machine 1710 is an embodiment of agricultural machine 100 comprising control system 210. Agricultural machine 1710 may have a perceived orientation that is distorted from the actual orientation. In this scenario, the agricultural machine 1710 is traveling along a perceived curved path 1735 that is offset from the curved path 1730. The agricultural machine 1710 may enact course correction navigation to align the orientation of the agricultural machine 1710, or that is, to align the perceived path 1735 with the actual path 1730. In one or more embodiments, the agricultural machine 1710 may use the detection mechanisms to locate obstacles 1740 and 1750 on the path 1730 as markers on the path 1730. COMPUTER SYSTEM EXAMPLE

[0162] FIG. 18 is a block diagram illustrating components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them on a processor (or controller). Specifically, FIG. 18 shows a schematic representation of a machine in the exemplary form of a computer system 1800, within which program code (e.g., software or software modules) for causing the machine to perform any one or more of the methodologies discussed herein can be executed. The program code may be comprised of instructions 1824 executable by one or more processors 1802. In alternative embodiments, the machine operates as a stand-alone device or may be connected (e.g., networked) to other machines. In a network deployment, the machine may operate in the capacity of a server machine or a client machine in a server-to-client network environment, or as a peer-to-peer machine in a peer-to-peer (or distributed) network environment.

[0163] The machine can be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA ), a cell phone, a smartphone, a web device, a network router, switch, or bridge, or any machine capable of executing instructions 1824 (sequential or otherwise) that specify actions to be performed by that machine. Furthermore, although only a single machine is illustrated, the term “machine” should also be considered to include any collection of machines that individually or together execute instructions 1824 to execute any one or more of the methodologies discussed herein.

[0164] Exemplary computer system 1800 includes a processor 1802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (digital signal processor - DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination thereof), a main memory 1804 and a static memory 1806, which are configured to communicate with each other via a bus 1808. The computer system 1800 may further include visual display interface 1810. The visual interface may include a software driver that allows display of user interfaces on a screen (or display). The visual interface may display user interfaces directly (e.g. on the screen) or indirectly on a surface, window or the like (e.g. via a visual projection unit). To facilitate discussion, the visual interface can be described as a screen. The visual interface 1810 may include or may interact with a touch screen. Computer system 1800 may also include alphanumeric input device 1812 (e.g., a keyboard or touchscreen keyboard), a cursor control device 1814 (e.g., a mouse, a trackball, a joystick, a sensor motion device or other pointing instrument), a storage unit 1816, a signal generating device 1818 (e.g., a speaker), and a network interface device 1820, which are also configured to communicate via bus 1808 .

[0165] The storage unit 1816 includes a machine-readable medium 1822 on which instructions 1824 (e.g., software) incorporating any one or more of the methodologies or functions described herein are stored. Instructions 1824 (e.g., software) may also reside, completely or at least partially, within main memory 1804 or within processor 1802 (e.g., within processor cache memory) during their execution by computer system 1800, the main memory 1804 and the processor 1802 also constitute machine-readable media. Instructions 1824 (e.g., software) may be transmitted or received over a network 190 via network interface device 1820 .

[0166] Although the machine-readable medium 1822 is shown in an exemplary embodiment as a single medium, the term “machine-readable medium” should be considered to include a single medium or multiple mediums (e.g., a centralized database or distributed, or associated caches and servers) capable of storing instructions (e.g., 1824 instructions). The term “machine-readable medium” should also be considered to include any medium capable of storing instructions (e.g., instructions 1824) for execution by the machine and that cause the machine to execute any one or more of the methodologies disclosed herein. The term “machine-readable medium” includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media. XII. ADDITIONAL CONSIDERATIONS

[0167] Throughout this Descriptive Report, multiple instances may implement components, operations or structures described as a single instance. Although the individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously and nothing requires that the operations be performed in the order illustrated. Structures and functionalities presented as separate components in exemplary configurations can be implemented as a combined structure or component. Likewise, structures and functionalities presented as a single component can be implemented as separate components. These and other variations, modifications, additions and improvements are within the scope of the matter covered here.

[0168] Certain embodiments are described herein as including logic or a series of components, modules, or mechanisms. Modules may constitute software modules (e.g., code embedded in a machine-readable medium or in a broadcast signal) or hardware modules. A hardware module is a tangible unit capable of performing certain operations and can be configured or arranged in a certain way. In exemplary embodiments, one or more computer systems (e.g., a stand-alone computer system, client, or server) or one or more hardware modules of a computer system (e.g., a processor or group of processors) may be configured by software (e.g., an application or part of the application) as a hardware module that operates to perform certain operations as described in this document.

[0169] In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may include dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, as a field programmable gate array (FPGA), or as an application integrated circuit). (application-specific integrated circuit - ASIC)) to perform certain operations. A hardware module may also include programmable logic or circuitry (e.g., as encompassed in a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically, on dedicated and permanently configured circuits or on temporarily configured circuits (e.g. software configured) may be driven by cost and time considerations.

[0170] Accordingly, the term “hardware module” should be understood to encompass a tangible entity, whether an entity that is physically constructed, permanently configured (e.g., wired), or temporarily configured (e.g., programmed) to operate in a a certain way or to perform certain operations described herein. As used in this document, “hardware-implemented module” refers to a hardware module. Considering embodiments in which the hardware modules are temporarily configured (e.g., scheduled), each of the hardware modules does not need to be configured or instantiated at any instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. The software can therefore configure a processor, for example, to constitute a specific hardware module at one time instance and to constitute a different hardware module at a different time instance.

[0171] Hardware modules can provide information to and receive information from other hardware modules. Consequently, the described hardware modules can be considered as communicatively coupled. When several such hardware modules exist simultaneously, communications can be accomplished through signal transmission (e.g., through appropriate circuits and buses) that connect the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, by storing and retrieving information in memory structures to which the multiple hardware modules have access. access. For example, a hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. Another hardware module can then later access the memory device to retrieve and process the stored output. Hardware modules can also initiate communications with input or output devices and can operate on a resource (for example, a collection of information).

[0172] The various exemplary method operations described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether configured temporarily or permanently, these processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some exemplary embodiments, comprise processor-implemented modules.

[0173] Likewise, the methods described herein can be at least partially implemented by the processor. For example, at least some of the operations of a method may be performed by one or processor-implemented hardware modules. Performance of some of the operations may be distributed among one or more processors, not just residing on a single machine, but deployed on one or more machines, e.g., computer system 700. In some exemplary embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment, or as a server farm), while in other embodiments the processors may be distributed across multiple locations.

[0174] The one or more processors may also operate to support the performance of relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (such as machines including processors), such operations being accessible over a network (e.g., the Internet) and through one or more appropriate interfaces ( for example, application program interfaces (APIs).)

[0175] The performance of some of the operations may be distributed among one or more processors, not only residing on a single machine, but deployed on multiple machines. It should be noted that when an operation is described as being performed by “one processor”, this should be interpreted as also including the process being performed by more than one processor. In some exemplary embodiments, one or more processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other exemplary embodiments, one or more processors or processor-implemented modules may be distributed across multiple geographic locations.

[0176] Some parts of this Descriptive Report are presented in terms of algorithms or symbolic representations of operations on data stored as bits or binary digital signals within a machine memory (e.g., a computer memory). These algorithms or symbolic representations are examples of techniques used by those skilled in data processing techniques to convey the substance of their work to others skilled in the art. As used herein, an “algorithm” is a self-consistent sequence of operations or similar processing that leads to a desired result. In this context, algorithms and operations involve physical manipulation of physical quantities. Typically, but not necessarily, these quantities may take the form of electrical, magnetic, or optical signals capable of being stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is sometimes convenient, mainly for reasons of common usage, to refer to such signals using words such as “data”, “content”, “bits”, “values”, “elements”, “symbols”, “characters”, “ terms”, “numbers”, “numerals” or similar. These words, however, are merely convenient labels and must be associated with appropriate physical quantities.

[0177] Unless specifically stated otherwise, discussions here using words such as “processing”, “computation”, “calculation”, “determination”, “presentation”, “display” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical quantities (e.g., electronic, magnetic, or optical) within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers or other machine components that receive, store, transmit, or display information.

[0178] As used herein, any reference to “a(one) embodiment” or “an(an) embodiment” means that a particular element, feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment . Appearances of the phrase “in an embodiment” in various places in the Descriptive Report do not necessarily refer to the same embodiment.

[0179] Some modalities can be described using the expressions “coupled” and “connected” together with their derivatives. It should be understood that these terms are not synonymous with each other. For example, some embodiments may be described using the term “connected” to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled”, however, can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The modalities are not limited in this context.

[0180] As used herein, the terms “comprises”, “comprising”, “includes”, “including”, “has”, “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article or apparatus comprising a list of elements is not necessarily limited to those elements only, but may include other elements not expressly listed or inherent in such process, method, article or apparatus. Furthermore, unless expressly stated otherwise, “or” refers to an inclusive or and not an exclusive or. For example, a condition A or B is satisfied by any of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0181] Furthermore, the use of “a” or “an” is employed to describe elements and components of the modalities presented here. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it means otherwise.

[0182] Upon reading this disclosure, those skilled in the art will further appreciate additional alternative structural and functional designs for a system and a process for providing CMC change assessment through the principles disclosed herein. Thus, although specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, alterations and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the disclosed principles.

Claims (21)

Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, comprising a detection system having a field of view of an environment around the agricultural machine, the field of view comprising one or more blind spots of the environment and characterized by which the method comprises: receiving a notification from a manager of the agricultural machine that there are no obstacles in the blind spots of the detection system; verify that there are no obstacles in unobstructed views of the detection system by applying an obstacle detection model to images captured by the detection system; determine an agricultural machine configuration to carry out agricultural actions autonomously in the environment with agricultural machine implements; determine a virtual security bubble for the agricultural machine to autonomously carry out agricultural actions based on the determined configuration; detect an obstacle in the environment by applying the obstacle detection model to the images captured by the detection system; determine that the obstacle is within the virtual security bubble; and in response to the determination that the obstacle is within the virtual security bubble, terminate the operation of the agricultural machine. Method for Establishing Virtual Safety Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that receiving notification from the manager that there are no obstacles in the blind spots comprises: encouraging the manager to go around the agricultural machine in a position of rest; and capturing images of the manager walking around the agricultural machine, where the detection of the completed walk indicates that there are no obstacles in the blind spots of the detection system. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that applying the obstacle detection model comprises: segmenting image pixels as belonging to one of the types of objects, including a first type from object to obstacles. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 3, characterized in that applying the obstacle detection model further comprises: identifying a first obstacle from pixels classified as belonging to the first type of object for obstacles. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that it further comprises: changing the agricultural machine to a second configuration; and dynamically adjust the virtual security bubble based on the second configuration. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that the configuration is a combination of: trajectory of the machine, speed of the agricultural machine, acceleration of the agricultural machine, one or more characteristics of an expected obstacle, a type of implement employed by the agricultural machine, a type of mounting mechanism for mounting the implement to the agricultural machine, one or more agricultural actions performed by the agricultural machine, one or more characteristics of an agricultural action, one or more characteristics of the agricultural machine, one or more characteristics of the environment, a type of obstacle and input from a manager. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that determining the virtual security bubble comprises determining a shape and a size of the virtual security bubble. Method for Establishing Virtual Safety Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that determining that the obstacle is entering the virtual safety bubble comprises: determining a depth of the obstacle; and determine that the depth of the obstacle is less than the depth of the virtual security bubble. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 8, characterized in that determining the depth of the obstacle comprises applying a depth estimation model to one or more images of the obstacle captured by the detection system to predict the depth of the obstacle. Method for Establishing Virtual Security Bubble Around Autonomous Agricultural Machine, according to Claim 1, characterized in that it further comprises: in response to determining that the obstacle is within the virtual security bubble, adopting one or more preventive measures to avoid collision with the obstacle. Computer Readable Non-Transitory Storage Medium, characterized in that it stores instructions for establishing a virtual security bubble around an autonomous agricultural machine comprising a detection system with a field of view of an environment around the agricultural machine, the field of vision comprising one or more blind spots of the environment and instructions that, when executed by a computer processor, cause the computer processor to perform operations comprising: receiving notification from an agricultural machine manager that there are no obstacles at the points detection system blind; verify that there are no obstacles in unobstructed views of the detection system by applying an obstacle detection model to images captured by the detection system; determine an agricultural machine configuration to carry out agricultural actions autonomously in the environment with agricultural machine implements; determine a virtual security bubble for the agricultural machine to autonomously carry out agricultural actions based on the determined configuration; detect an obstacle in the environment by applying the obstacle detection model to images captured by the detection system; determine that the obstacle is within the virtual security bubble; and in response to the determination that the obstacle is within the virtual security bubble, terminate the operation of the agricultural machine. Computer Readable Non-Transitory Storage Medium according to Claim 11, characterized in that receiving notification from the manager that there are no obstacles in the blind spots comprises: inciting the manager to go around the agricultural machine in a resting position; and capturing images of the manager walking around the agricultural machine, where the detection of the completed walk indicates that there are no obstacles in the blind spots of the detection system. Computer Readable Non-Transitory Storage Medium according to Claim 11, characterized in that applying the obstacle detection model comprises: segmenting pixels of the images as belonging to one of the object types, including a first object type for obstacles. Computer Readable Non-Transitory Storage Medium according to Claim 13, characterized in that applying the obstacle detection model further comprises: identifying a first obstacle from pixels classified as belonging to the first object type for obstacles. Computer Readable Non-Transitory Storage Medium according to Claim 11, characterized in that the operations further comprise: changing the agricultural machine to a second configuration; and dynamically adjust the virtual security bubble based on the second configuration. Non-Transitory Computer Readable Storage Medium according to Claim 11, characterized in that the configuration is a combination of: trajectory of the machine, speed of the agricultural machine, acceleration of the agricultural machine, one or more characteristics of an expected obstacle, a type of implement employed by the agricultural machine, a type of mounting mechanism for mounting the implement to the agricultural machine, one or more agricultural actions performed by the agricultural machine, one or more characteristics of an agricultural action, one or more characteristics of the agricultural machine, a or more characteristics of the environment, a type of obstacle and input from a manager. Computer Readable Non-Transitory Storage Medium according to Claim 11, wherein determining the virtual security bubble comprises determining a shape and a size of the virtual security bubble. Computer Readable Non-Transitory Storage Medium according to Claim 11, wherein determining that the obstacle is entering the virtual security bubble comprises: determining a depth of the obstacle; and determine that the depth of the obstacle is less than the depth of the virtual security bubble. Computer Readable Non-Transitory Storage Medium according to Claim 18, wherein determining the depth of the obstacle comprises applying a depth estimation model to one or more images of the obstacle captured by the detection system to predict the depth of the obstacle . Computer Readable Non-Transitory Storage Medium according to Claim 19, wherein the operations further comprise: in response to determining that the obstacle is within the virtual security bubble, taking one or more preventative measures to avoid the collision with the obstacle. Autonomous Agricultural Machine, navigable in an operational environment to perform one or more agricultural actions, characterized in that the agricultural machine comprises: a detection system comprising one or more cameras configured to capture images of the operational environment around the agricultural machine, the detection system having a field of vision comprising one or more blind spots in the environment; a treatment mechanism configured to apply treatment to a treatment area as part of one or more agricultural actions; a control system for establishing a virtual security bubble around agricultural machinery comprising: a computer processor and a non-transitory computer-readable storage medium that stores instructions that, when executed by the computer's processor, cause the processor of the computer perform operations comprising: receiving a notification from a manager of the agricultural machine that there are no obstacles in the blind spots of the detection system; verify that there are no obstacles in unobstructed views of the detection system by applying an obstacle detection model to images captured by the detection system; determine an agricultural machine configuration to carry out agricultural actions autonomously in the environment with agricultural machine implements; determine a virtual security bubble for the agricultural machine to autonomously carry out agricultural actions based on the determined configuration; detect an obstacle in the environment by applying the obstacle detection model to the images captured by the detection system; determine that the obstacle is within the virtual security bubble; and in response to determining that the obstacle is within the virtual security bubble, terminate operation of the agricultural machine

Images