WO2017178666A1

WO2017178666A1 - Autonomous set of devices and method for detecting and identifying plant species in an agricultural crop for the selective application of agrochemicals

Info

Publication number: WO2017178666A1
Application number: PCT/ES2016/070655
Authority: WO
Inventors: Diego Hernan Perez Roca
Original assignee: Diego Hernan Perez Roca
Priority date: 2016-04-12
Filing date: 2016-09-20
Publication date: 2017-10-19
Also published as: AR104234A1

Abstract

The invention relates to an autonomous set of devices for detecting and identifying wild and cultivated plant species on a farm, using software which, by obtaining a video in real time, can detect, isolate and identify different wild and cultivated plant species by using convolutional neural networks able to distinguish distinctive aspects of the morphology, taxonomy and philotaxy of the plants. By previously training the convolutional neural networks on the characteristics that distinguish one species from another, the system allows the particular identification of each species. By means of a system of video cameras mounted along a transport vehicle, and with the data being obtained in real time, the computer system can determine the agrochemical to be applied according to the plant identified and electronically or mechanically actuate the opening of the valve of a spray nozzle. In this way, the plant receives the exact dose and the specific agrochemical according to the necessary treatment.

Description

AUTONOMOUS DEVICE SET AND METHOD FOR THE DETECTION AND IDENTIFICATION OF VEGETABLE SPECIES IN A AGRICULTURAL CULTURE FOR THE APPLICATION OF AGROCHEMICALS IN A SELECTIVE FORM

The present invention relates to Application technologies in the agro industrial field. Particularly, it is a set of devices autonomous for the detection and identification of species vegetables in an agricultural crop for the application of Agrochemicals selectively. The set consists of multiple cameras that must be arranged on the wing or boom arm of for example a machine sprayer, a detection device and plant species identification, a circuit electronic responsible for managing the opening and closing of the agrochemical product sprinkler peaks, and a Ultrasound sensor for each chamber in the set.

More particularly, through obtaining of digital images provided by the set of cameras and automatically sent to the device processing which is able to detect, segment and identify the different species for sure vegetables found in the image scene processed. In case of identifying a plant species previously designated as a species to be removed or weeds, the device sends a signal to the electronic circuit which is responsible for the management of opening and closing of different solenoid valves of the sprinkler peaks of agrochemical product to open during a period of predetermined time a specific peak, falling from this way a defined dose of agrochemical product over the desired plant. More particularly, the processing device is able to diagnose the agrochemical to use based on a table of correspondence comprising the different species vegetables, what is your specific agrochemical for treatment and the recommended dose to use.

There are currently several advances technology that have allowed the agro industrial sector incorporate new tools, techniques and machinery able to increase their efficiency and achieve better yields This incorporation and application of technologies to the agro industrial sector was defined as "Precision Agriculture." Within this new definition encompasses various areas as per example satellite geo positioning technology, management software, electronics and robotics software, Like to name a few.

It should also be noted that there is a new field in the computer industry called vision artificial, technical vision or also vision by computer (computer vision) developed in the last 10 years that really started to get older transcendence and bear fruit of efficiency in this last time. All this, thanks to the advances, availability in the market and at affordable prices of high definition digital video cameras, with great number of frames achieved per second, together with computers with faster processors (CPUs), the incorporation of coprocessors dedicated solely to graphical load calculation called graphics cards (GPU), allowing real efficiency to be achieved for Real-time video management and analysis.

Artificial vision is a subfield of the artificial intelligence whose purpose is to program a computer to "understand" the characteristics of an image.

The typical objectives of artificial vision include: detection, segmentation, location and recognition of certain objects in images; evaluation of the results, such as segmentation, registration; registration of different images of the same scene or object, that is to make agree same object in different images; tracking a object in a sequence of images; mapping a scene to generate a three-dimensional model of it, what that could be used by a robot to navigate the scene; estimation of the three-dimensional positions of humans; and search for digital images by content.

These objectives are achieved through pattern recognition, statistical learning, projection geometry, image processing, graph theory and other techniques.

Continuous signal images are reproduced by analog electronic devices that record image data accurately using several methods, such as a sequence of fluctuations of an electrical signal or changes in the chemical nature of an emulsion of a film, which vary continuously in different aspects of the image. To process or display on the computer a continuous signal or an analog image must be convert first to an understandable digital format to the computer. This process applies to all images regardless of origin, complexity and if they are black and white or grayscale, or all color. A digital image is composed of a matrix rectangular or square of pixels representing a series of intensity values ordered in a system of coordinates in a plane (x, y).

In the field of Artificial Vision just they are taking the first steps of application in the farming. There are several research projects in progress, carried out by government entities and / or universities, but so far none has passed the research and development stage in the laboratory, or prototype adaptation in the field of action, so that to date there is no commercial availability any for the acquisition of this technology by of the consumer belonging to the agro industrial field. However, it should be clarified that the appearance of these new inventions through the use of vision artificial are more focused on achieving applications to provide mechanical metal robots with systems of visual guidance through the field.

Another of the latest technological advances with what we have is Artificial Intelligence, applied through the use of neural networks convolutional They are a type of artificial neural network where neurons correspond to receptive fields of a way very similar to the neurons in the cortex Primary visual (V1) of a biological brain. This type network is a variation of a multilayer perceptron, but their operation makes them much more effective for artificial vision tasks, especially in the Image classification Per perceptron must understand an artificial neuron and basic unit of inference in the form of linear discriminator, this is a algorithm capable of generating a criterion to select a subgroup from one more component group big.

The basics of neural networks convolutional are based on the Neocognitron, concept introduced by Kunihiko Fukushima in 1980, having been improved later by Yann LeCun et al. in 1998 to introduce a learning method based on Backpropagation to train the system correctly. In 2012 these networks were refined by Dan Ciresan et al. and implemented in a GPU getting So results never imagined.

Also, a parallel processing multi-thread (“multithreading”) allows you to execute efficiently multiple threads at the same time on the same GPU, managing to process several algorithms in the form concurrent, and in this way take full potential of the processor and in a shorter space of time, being able to as needed share the same logical resources and / or system physicists.

The convolutional neural networks consist in multiple layers with different purposes. To the principle is the extraction phase of characteristics, composed of convolutional neurons and of sampling reduction. At the end of the network you they find simple perceptron neurons to make the final classification on features extracted.

The feature extraction phase is resembles the stimulating process in the cells of the visual cortex This phase consists of alternate layers of convolutional neurons and reduction neurons of sampling. As the data progresses along this phase, its dimensionality is reduced, being the neurons in distant layers much less sensitive to disturbances in the input data, but at the same time being these activated by characteristics every more and more complex

In the feature extraction phase, the simple neurons of a perceptron are replaced by matrix processors that perform an operation about the 2D image data that passes through them, in place of a single numerical value

The convolution operator has the effect of filter the input image with a kernel previously trained. This transforms the data in such a way that certain characteristics, determined by the shape of the core, become more dominant in the output image having these a higher numerical value assigned to the pixels that represent them. These cores have specific image processing skills, such as edge detection that can be perform with cores that highlight a gradient in a particular address. However, the nuclei that they are trained by a convolutional neural network they are generally more complex to be able to extract other more abstract and non-trivial features.

Neural networks have some tolerance to small disturbances in the data of entry. For example, if two almost identical images differentiated only by a transfer of some pixels laterally are analyzed with a neural network, The result should be essentially the same. This is gets, in part, given the reduction in sampling that It occurs within a convolutional neural network. To the reduce resolution, same features will correspond to a greater activation field in the input image.

Originally, neural networks convolutional used a subsampling process to carry out this operation. However, studies recent have shown that other operations, such as by max-pooling example, they are much more effective in summarizing characteristics about a region. In addition, there is evidence that this type of operation is similar to how the visual cortex can summarize information internally.

The max-pooling operation finds the value maximum between a sample window and pass this value as a summary of characteristics about that area. How result, the size of the data is reduced by a factor equal to the size of the sample window on the which one is operated

After one or more extraction phases of characteristics, the data finally reaches the phase of classification. By then, the data has been debugged up to a series of unique features to the input image, and it is now the work of the latter phase to classify these characteristics towards a label or other, depending on training objectives.

Given the nature of the convolutions within of the convolutional neural networks, these are suitable to learn to classify all types of data where these are distributed in a continuous way to along the entry map, and in turn be statistically similar anywhere on the map input For this reason, they are especially effective. to classify images, for example for Self-tagging of images.

Another important point to keep in mind is that convolutional neural networks have demonstrated their success in many applications, due to its ability to solve certain problems with relative ease of application. We have been able to solve problems without the need to understand or learn the properties analytics and statistics thereof, nor the steps of the solution. Neural Network Research convolutional has resulted in a wide variety of models and learning algorithms, but only in the last two years is where there has been a change of exceptional paradigm.

Progress has been made in the field of networks artificial convolutional neuronal that are being able to use similar methodologies for identification of plant species, but all of them they are based on ideal conditions of acquisition of the Image mainly on white background, isolated from other plants, still image, full copy, etc., which does not work in real environments and / or in crops, where plants interfere with each other, overlapping the leaf mass with each other, adverse lighting conditions, movements of the themselves by the wind and, what is fundamental, they are not acquired the images in video format and at speeds up to 20 km / h, with the consequent need for processing of up to 7 meters of land per second.

In relation to the method used for training of a convolutional neural network, it You can cite the patent US 7747070 B2 granted on 29 June 2010 to Microsoft Corp. Such patent is refers to a convolutional neural network implemented in a graphics processing unit, which is then trained through a series of steps forward and backward, with convolutional cores and matrices modified biases in each step backwards according with a gradient of an error function. The application take advantage of the parallel processing capabilities of pixel shader units in a GPU, and use a set of formulas from beginning to end to Schedule calculations in pixel shaders. The entry and exit to the program is done through textures, and a summation process of multiple steps when sums are needed through records of pixel shader units.

In this way, neural networks convolutional are being used for the Image recognition and classification. At recognition process using a classifier based on a convolutional neural network, a image to the network and after several repetitions of convolution operations in a maximum space of sampling and complete connection, is extracted as a result of recognition an accurate classification of the image and a maximum level of security of the result.

There are also two products on the market similar commercials in its technological conception known as "Trimble WeedSeeker" and "PSB-Weedit". Both systems claim as a commercial argument to be a weed detector but for its characteristics and functionalities are only able to recognize the presence of chlorophyll through its light sensors infrared This is useful only when a field is fallow, this is a field not grown in anticipation of a new planting, or in premergence if it was sown, since not being able distinguish between the specific crop plant and the plants to be treated as weeds is not useful in other situation

Object tracking (in English object tracking) is a process that allows you to estimate over time the location of one or more mobile objects using the use of a camera The improvements achieved in form accelerated in the quality and resolution of the sensors image, together with the impressive increase of The computing power achieved in the last decade has favored the creation of new algorithms and applications by tracking objects.

Object tracking can be a process slow due to the large amount of data contained in a video, which can increase its complexity in the face of possible need to use recognition techniques of objects to track.

Video cameras capture information on objects of interest in the form of a set of pixels When modeling the relationship between the appearance of object of interest and pixel value corresponding, an object follower values the location of this object in time. The relationship between the object and projection of its image is very complex and it may depend on more factors than just the object position, which implies that the tracking of objects is a difficult goal to achieve.

The main challenges to have in account in the design of an object follower are related to the similarity of aspect between the object of interest and other objects in the scene, as well as the variation of appearance of the object itself. Since the appearance of both other objects and the background can be similar to the object of interest, this may Interfere with your observation. In that case, the features extracted from those unwanted areas it can be difficult to differentiate from what is expected that the object of interest generates. This phenomenon is known. with the name of disorder ("clutter").

In addition to the follow-up challenge caused by the "Clutter", the changes of aspect of the object in the plane of the image makes it difficult to track caused by one or more of the following factors: position changes, ambient lighting, noise, occlusions.

In a tracking scenario, an object is you can define as anything of interest for later analysis. The objects can be represent through their forms and appearances, generally: points, primitive geometric shapes, object silhouette and contour, articulated models of shape, skeletal models.

There are also several ways to represent the appearance characteristics of objects. Must have keep in mind that shape representations are also can be combined with those of appearance to carry out the tracing. Some of the aspect representations most common are: the probability density of the aspect of objects, templates, active appearance models, Multivist-looking models.

Select the appropriate features It has a fundamental role in monitoring. In In general, the most desired visual feature is the uniqueness because objects can be distinguished easily in the space of features. The details of the most common features are the following: color, margins, optical flow, texture.

Each tracking method requires an object detection mechanism, either in each frame or when the first object appears in the video. A common method for object detection is the use of single-frame information. However, some object detection methods make use of the temporal information calculated from a sequence of images to reduce the number of false detections. This temporary information is generally calculated using the “frame differencing” technique, which shows the changing regions in consecutive sections. Once the regions of the object in the image are taken into account, it is then the task of the follower to perform the object correspondence from one frame to another to generate the tracking. The most popular methods in the context of object tracking are: point detectors, background subtraction, segmentation.

Point detectors are used to find points of interest in images that have an expressive texture in their respective locations. Points of interest have been used for a long time. time in the context of the movement and in the problems of follow up. A desirable feature in terms of the points of interest is its invariance in the changes of illumination and in the point of view of the camera.

Object detection can be achieved by building a representation of the scene called background model and then finding the model deviations for each incoming frame. Any significant change in a region of the background model image represents an object in movement. The pixels that make up the regions in change process are marked for later processing In general, a component algorithm connected is applied to get connected regions that correspond to the objects. This process is known. as background subtraction.

The objective of segmentation algorithms of the image is to divide the image into regions perceptually similar. Each algorithm of segmentation covers two problems, the criteria for a good partition and the method to get the efficient partition. There are different techniques of segmentation of moving objects that can be separate into two large groups: those based on movements and feature based spacetime.

These techniques make use mainly of the movement information. Within this group we can differentiate two types: those who work with the two-dimensional movement (2D) and those who do it in three (3D). Within two-dimensional techniques We found: techniques based on discontinuities of optical flow and techniques based on the detection of changes

2D motion models are simple, But less realistic. As a consequence, the systems of 3D segmentation are the most used in the practice. Within three-dimensional methods, They can distinguish two different algorithms: structure from the SFM movement (acronym for English structure from motion) and parametric algorithms.

The SFM generally handles 3D scenes that contain relevant depth information while that in parametric methods this is not assumed depth. Another important difference between the two algorithms is that in the SFM a movement is assumed rigid while in parametric algorithms only stiffness of movement is assumed in parts of the scene.

In spacetime techniques, the methods segmentation based solely on motion are sensitive to the inaccuracies of the valuation of movement. To solve these problems, in the spacetime methods it is proposed to complement the movement through the use of spatial information. There are two dominant approaches: based on limits and based on regions.

Object tracking is a very task important within the field of video processing. He main objective of the monitoring techniques of objects is to generate the trajectory of an object through of time, positioning it within the image. We can classify techniques according to three large groups: point tracking, tracking core (kernel) and silhouettes tracking.

According to point tracking techniques the objects detected in consecutive images are represented each by one or several points and the association of these is based on the state of the object in the previous image, which may include position and movement. An external mechanism is required that Detect the objects of each frame. This technique may present problems in scenarios where you object presents occlusions and in the entrances and exits of these. Point tracking techniques can be Also classify into two broad categories: deterministic and statistical.

Core tracking techniques perform a calculation of the movement of the object, which is represented by an initial region of an image to the next The movement of the object is expressed in general in the form of parametric movement (translation, rotation, related ...) or through the flow field Calculated in the following frames. We can distinguish two categories: tracking using density templates and appearance models of probability and follow-up based on models multivist

Silhouettes tracking techniques are performed by valuing the region of the object in each image using the information it contains. This information can be in the form of density of look or shape models that are generally presented with margin maps. It has two methods: correspondence of form and monitoring of contour.

Tracking objects of interest on video It is the basis of many applications ranging from video production to remote surveillance, and from Robotics to interactive games. The Object followers are used to improve the understanding of certain video data sets of medical and safety applications; to increase the productivity by reducing the amount of labor that it is necessary to complete a task and to give rise to Natural interaction with machines.

The optical flow (in English "optical flow") is the pattern of the apparent movement of objects, surfaces and edges in a scene caused by the relative movement between an observer's eye or a Camera and scene.

The concept of optical flow was studied by first time in the 1940s and finally it was published by the American psychologist James J. Gibson in his article "Theory of Affordances" of 1977, where he describes all the possibilities of action that are materially possible in a given context, this it is the quality of an object or environment that allows a individual perform an action.

The concept of "affordances" can be interpret through the concept of availabilities or offers. There is no translation into Spanish normally accepted of the meaning of this concept, for which they have been given varied meanings such as "permissiveness", "habilitation", “Environmental opportunities”, and even “invitations to use".

In 1988, Donald Norman used the term “Affordances” in the HCI context (acronym for English Human-Computer Interaction, this is Interaction Human-Computer) to refer to those possibilities of action that are immediately perceived by the Username. In his book "The Design of Everyday Things" establishes the concept "affordances" not only within the user's physical abilities, but also in the its ability to feed on past experiences, goals, plans, estimates comparing other types of experiences, etc.

A second definition then more refined define the term "affordances" as the possibilities of action that a user is aware of being able to perform. Applications of optical flow such as motion detection, object segmentation, the time until the collision and the calculation approach of expansions, motion coding compensated and Stereoscopic disparity measurement use this movement of the surfaces and edges of the objects.

These technologies may be applied. in, for example, US Patent 6038337 A which refers to a hybrid system of neural networks for the object recognition that exhibits local sampling of image, a neural network of self-organized maps, and a hybrid convolutional neural network. The car map organization provides quantification of samples image in a topological space where entries that are close in the original space are also in the exit space, then providing the dimensionality reduction and change invariance minors in the image sample, and the neural network convolutional hybrid provides partial invariance to the translation, rotation, scale, and deformation. The net convolutional hybrid extracts features successively larger in a set of layers hierarchical Alternative embodiments are described. using the Karhunen-Lo + E, gra e + EE that transforms instead of the own organization map, and a multilayer perceptron instead of the convolutional network.

For its part, the US patent application 20150245565 A1 refers to a device and method for the application of chemicals to plants and parts of plants in specific natural environments, as well as Farmlands. In one embodiment preferred, an autonomous vehicle carries the device chemical application and is, in part, controlled by the processing requirements of device vision component responsible for detecting and assigning lists of objectives to chemical ejectors that target these target points while the device evolves to Through the countryside or a natural environment.

However, the device request US20150245565A1 patent is only able to detect the presence of a plant, but is unable to identify What plant species is it? Can only distinguish two characteristics that are absolutely different, how is the soil of plants, and determine only with a certain probability whether it is a crop or not.

This system can only distinguish the crop as long as certain parameters are given (paragraph [0024]) with a high margin of error, which translates into poor performance in the handling of the agrochemical to be applied. This is done through the use of "computer vision" always distinguishing the main crop line, thus basically being able to apply herbicide or any agrochemical on any plant that is outside the main line. Therefore, in the first place for the device of the application in question, it is the crop plants only those that are within the crop row, so if they are outside it they are considered weeds. If it is inside the row but the shape of the leaf is somewhat different, it is also considered weed; if it is in the row and the shape of the leaf is similar but the space between plant and plant is not the one that is usually used for planting that crop, it is also considered weed. Instead, this system is unable to distinguish between different plant species, not only if it is a plant or soil.

The system proposed in the request for US patent 20150245565 A1 has the great limitation of Only be able to work in crops in continuous rows. In agricultural crops most of the time it cultivate in a continuous row, but it is virtually impossible to keep this uniform in the whole lot, mainly because the vehicle that previously made the sowing performs several deviations in his career, some inevitable; eventually causing failures in the attempt to maintain the trajectory of the applicator vehicle of agrochemical when trying to follow the main line.

In addition, this system is unable to distinguish what plant species is it to apply the specific herbicide and thus eliminate said species. By therefore, it is not a system that works in all kinds of terrain without having a certain pattern to maintain its trajectory, it cannot identify the kind of species in question, you can't make a intelligent application of the necessary agrochemical with a great cost savings in agrochemicals and efficiency unbeatable in weed management.

On the other hand, when you mention the use multispectral camera (paragraph [0031]) clearly specifies that it is not currently viable for a real-time use because it only works with pictures, not live video sources. Further, need a normalization and / or calibration of the source of lighting prior to each analysis, which is totally impracticable in an uncontrolled environment of lighting as it is an outdoor site.

Therefore, it is still necessary to have agricultural machinery that serves not only to apply a herbicide but also to identify fit specific which plants are herbs differentiating them from cultivation, and direct herbicide application only on herbs dosing it according to its identification and behavior to achieve a result maximum satisfactory. In other words, it is necessary have agricultural equipment to eliminate effectively and selectively plant herbs of the crop with fair and specific doses as the grass identified thus protecting the crop development and the environment.

Therefore, the purpose of this invention is an autonomous set of devices for the detection and identification of plant species in an agricultural crop for the application of agrochemicals selectively, said set comprises: a chemical application device that comprises at least one container container of agrochemicals linked in fluid communication with a plurality of sprinkler peaks through a valve, a plurality of cameras that are arranged on the autonomous vehicle and focused on cultivation, where each camera has an associated ultrasound sensor for the height measurement to the crop in real time, and in where each camera is tilted forward at 45 degrees from normal; a device of detection and identification of plant species connected to the cameras to receive information from video of them, an electronic circuit in charge to manage the opening and closing of the valves of the sprinkler peaks of agrochemicals connected to detection device that manages through said circuit the opening and closing of the valves of the sprinkler peaks, and where, the set of devices is mounted on a vehicle of transport.

Preferably, the transport vehicle is a self-propelled vehicle or a tow vehicle.

More preferably, the vehicle self-propelled is a vehicle for fumigation with arms sides arranged perpendicularly to it (mosquito).

More preferably, the device Detection and identification consists of a processor.

Even more preferably, the processor comprises a tool based on computer software developed in C ++ language, a vision framework artificial, and a neural network framework convolutional

It is another object of the present invention, a method for the detection and identification of species vegetables in an agricultural crop for the application of agrochemicals selectively with the autonomous set of devices described above, said method Understand the steps of:

a) detect and classify plant species in an agricultural crop during the whole tour autonomous of devices through cultivation; b) analyze the information obtained in a) determining area and perimeter of weeds; and c) spray the dose proper of an agrochemical in the right place taking consider the speed of advance of the equipment.

Preferably, step a) of the method allows distinguish weed cultivation.

More preferably, step a) of the method allows to identify plant species to Determine the agrochemical to apply.

Preferably, in step c) of the method the dose of agrochemical is sprayed by opening a solenoid valve.

Also preferably, the species Vegetables correspond to the crop and weeds.

More preferably, the agrochemical is a herbicide, a foliar fertilizer, an insecticide, a fungicide, or a protective compound.

Even more preferably, the step b) also allows to determine the state of the crop.

Additionally, step c) of the method comprises selecting a specific herbicide from a set of herbicides for each weed identified in step a) regarding the crop.

Also additionally, step c) of the method comprises selecting a specific foliar fertilizer of a set of foliar fertilizers for cultivation identified in step a) according to their status.

Additionally, step c) of the method comprises selecting a specific insecticide from a set of insecticides for the crop identified in step a) according to its state of deterioration.

Still additionally, step c) of the method comprises selecting a specific fungicide from a set of fungicides for the identified crop in step a) according to its state of deterioration.

In addition, step c) of the method comprises select a specific protective compound from a set of protective compounds for cultivation identified in step a) according to their status.

More specifically, the method for the detection and identification of plant species in an agricultural crop comprises the steps of: a) obtaining a Real Time Video Stream from a plurality of cameras positioned along the wings of the autonomous set of devices for the detection and identification of plant species in an agricultural crop for the application of agrochemicals in a selective way; b) process each of the frames obtained; c) convert the frame to a numerical matrix with the representation of RGB (Red-Green-Blue English) colors of each pixel in the image; d) trim the matrix to select the area of the frame to be processed; e) assign an area of the image to the corresponding sprinklers, so that if weeds are detected in that area, the opening order is sent to the corresponding sprinkler; f) apply 4 filters to obtain a mask of the predominant colors of the plant species to be identified; g) identify the contours of the image on the color mask by saving the information of the position of each one; h) estimate the travel speed with the positions of the contours found in the current frame and the positions of those same contours in a previous frame; i) obtain a pixel speed per frame and make the speed passage in meters per second using a pixel-meter ratio and a frame-second ratio; j) crop the image into small squares that contain the contours; k) resize each of the squares cut out of a part of the image to a preferred size; l) send the squares of images to the first layer (input layer) of the previously trained convolutional neural network for analysis and categorization; m) processing each square within the previously trained neural network, wherein the image is taken, it is performed in parallel in one pass (forward) and the result is sent to the output layer of the neural network; n) obtain the result of the last layer (output layer) of the convolutional neural network; ñ) determine the agrochemical to be used based on the numerical value of the identified plant species; o) correlate the complete representation of the plant species identified with the agrochemical necessary to apply in each of the plants that appear in the processed frame that was obtained from the main video stream, and p) send the order according to the species identified for each image processed by the neural network that is associated with a sprinkler by the area where it is located, so that the spray is exactly on the plant species to be treated;

Preferably, in step f) of the method previous separates strange elements like earth, dry vegetable residue and species stones vegetables, where: a first filter transforms the YCbCr color format matrix; a second filter subtracts two channels in the RGB format depending on the color to filter; a third filter is a logical AND operation (bit by bit) between the filter results previous first and second; and a fourth filter applies to the previous result a blur (Gaussian blur), converting the image to black and white and removing the noise.

[Fig.1a] schematically represents the way the data goes through different types of tests, in order to make a decision in A three layer network.

[Fig. 1b] schematically represents the way in which the input layers of the network contain neurons that encode pixel values from entry.

[Fig.1c] schematically represents a possible architecture with rectangles denoting the subnets in order to show how the convolutional neural networks.

[Fig. 2a] schematically represents a rapid detection to classify plants that allows distinguish weed cultivation.

[Fig.2b] schematically represents the area and perimeter analysis performed by the system a Once weeds are detected.

[Fig.2c] schematically represents the spray with precision and accuracy that performs the weed system once the species is detected Vegetable and its size.

[Fig.3] represents a frame of a Flow Real Time Video obtained from a camera.

[Fig. 4] represents a table of equivalences between number of seals (or shutters) per second and vehicle speed in movement that carries the cameras.

[Fig. 5] represents the frame of FIG. 3 converted to a numeric matrix with representation of RGB (Red-Green-Blue English) colors of each Image pixel

[Fig. 6] represents an ideal size of central horizontal strip of the image to be process from the frame of [FIG.3].

[Fig. 7a] represents a transformation that make a first matrix filter in color format YCbCr.

[Fig.7b] represents a transformation that makes a second filter by subtracting two channels in the RGB format depending on the color to filter.

[Fig.7c] represents a transformation AND logic (bit by bit) between the results of the two previous filters as shown in [FIG.7a] and [FIG.7b] that performs a third filter.

[Fig.7d1] represents a transformation of a fourth filter that applies to the previous result of the [FIG. 7c] a Gaussian blur.

[Fig.7d2] represents the conversion of the Image from [FIG.7d1] to black and white.

[Fig.7d3] represents the elimination of noise of [FIG.7d2] corresponding to the points scattered whites.

[Fig. 8] represents the identification of Contours of the image on the color mask.

[Fig. 9] represents the image cropped in small squares of approximately the same size containing the contours of [FIG. 8].

[Fig. 10] represents one of the squares cropped from [FIG. 9] with image size changed to a preferred size of 256 x 256 pixels.

[Fig. 11a] represents another of the squares cut out of [FIG. 9] corresponding to a weed present in cultivation.

[Fig. 11b] represents a sequence where the square of [FIG. 11a] of a 256 x 256 image pixels is sent to the first layer or input layer of the previously trained convolutional neural network for analysis and categorization until you reach a last layer

[Fig. 12] represents a result in value average success of each of the categories obtained from the last layer or network exit layer convolutional neuronal according to the sequence of the [FIG. 11a].

[Fig. 13] is a complete representation of the processed frame of the main video stream according to the sequence of [FIG. 11a], where the unwanted plant species identified framed in red to apply a necessary agrochemical in each One of the weeds.

[Fig. 14] represents an AlexNet model that It consists of 5 convolutional layers according to the architecture chosen for the training of the Caffe network.

[Fig. 15] schematically represents a preferred embodiment of the autonomous assembly of devices for the detection and identification of plant species according to the present invention, showing how the boards with a microcontroller and development environment IDE (acronym for "Integrated Drive Electronics") Integrated Control Electronics) with inputs and outputs analog and digital Arduino to the CPU (acronym for English "Central Processing Unit", Processing Unit Central) through a USB port (acronym for English "Universal Serial Bus", Universal Serial Bus). At Arduino boards are connected cameras, the Ultrasound sensors and sprinklers.

[Fig. 16] represents a detail of the end of a side arm of a spraying unit where you can observe the cameras tilted towards forward at an angle of approximately 45 degrees with respect to the lower vertical axis and a plurality of associated sprinklers.

[Fig.17] schematically represents a camera tilted forward at an angle α of approximately 45 degrees from the vertical axis bottom that is installed on a side arm of a spray unit showing image and size approximate of the scene to process with that decline.

The present invention is constituted mainly by an autonomous set of devices for the detection and identification of plant species in an agricultural crop for the application of Agrochemicals selectively. The set consists of multiple cameras that must be arranged on the wing or arm of the boom of, for example, a machine sprayer, a detection device and plant species identification, a circuit electronic responsible for managing the opening and closing of the agrochemical product sprinkler peaks, and a Ultrasound sensor for each chamber in the set.

More specifically, it is the object of the present invention an autonomous set of devices for the detection and identification of plant species in an agricultural crop for the application of agrochemicals selectively said set comprises: a product application device chemicals comprising at least one container container of agrochemicals linked in fluid communication with a plurality of sprinkler peaks through a valve, a plurality of cameras that are arranged on the autonomous vehicle and focused on cultivation, where each camera has an associated ultrasound sensor for the height measurement to the crop in real time, and in where each camera is tilted forward at 45 degrees from normal; a device of detection and identification of plant species connected to the cameras to receive information from video of them, an electronic circuit in charge to manage the opening and closing of the valves of the sprinkler peaks of agrochemicals connected to detection device that manages through said circuit the opening and closing of the valves of the sprinkler peaks, and where, the set of devices is mounted on a vehicle of transport.

[FIG. 15], [FIG. 16] and [FIG. 17] show diagrams and diagrams of the autonomous set of devices for the detection and identification of plant species in an agricultural crop for application of agrochemicals selectively according to a preferred form of the present invention.

Also, the transport vehicle is a self-propelled vehicle or a tow vehicle. Particularly, the self-propelled vehicle is a fumigation vehicle with side arms arranged perpendicular to it (mosquito).

The detection and identification device consists of a processor that comprises a tool based on computer software developed in language C ++, and the use of the artificial vision framework OpenCV, and the neural network framework convolutional “Caffe” by Berkeley Vision and Learning Center By using this tool you can achieve recognition of different species Vegetables with 96% effectiveness.

Phases of the Convolutionary Neural Network

In the operation of a neural network we can clearly distinguish two phases or modes of operation: (i) a learning or training phase and (ii) a operation or execution phase.

During the first phase, the phase of learning, the network is trained to perform a certain type of processing Once reached a adequate training level, you go to the phase of operation, where the network is used to carry out the task for which she was trained.

Training or Learning of the Neural Network Convolutionary:

Below is detailed how is the architecture used of the convolutional neural network, and how it is trained for knowledge and classification of different plant species known in agricultural fields.

It is supplied to the Convolutionary Neural Network a set of data (Data Set) of a minimum of 50,000 photographs of the different plant species that you want to identify during training, taking into account the specific region of the planet and the species predominant in that place. These photographs are loaded through different folders or directories that represent the category to which each one belongs of them. The different photographs are supplied for example in JPEG format and at a minimum size of 80 x 80 pixels, preferably in a recommended size 256 x 256 pixels, which are included for each species different situations of the seedling namely loose leaves, partial leaves, whole plant, flowers, plant in context, etc. The architecture chosen for Caffe network training is the AlexNet model consisting of 5 convolutional layers [See FIG. 14].

For the operation and training of the convolutional neural network is necessary to have a computer (CPU) with x86 architecture and a board Nvidia video graphic with CUDA 7.0 support in forward, with at least 2500 CUDAs of capacity processing, which allows to perform in parallel the execution of arithmetic operations and increase the computing power through the use of the unit GPU (graphics processing unit). Once executed the training, the system will return a file "Deploy.prototxt" to be used from now on, within of detection processing software e Identification of plant species.

Once the learning or training phase is finished, the network generates a “ deploy.prototxt ” file, which is basically the learning model, and in this way it can already be used to perform the task for which it was trained. One of the main advantages of this model is that the network learns the relationship between the data, acquiring the ability to generalize concepts. In this way, a convolutional neural network can operate with information that was not presented during the training phase.

About the operation of the system classification is incorporated here by way of reference teachings of patent application US 2015036920 A1 published on February 5, 2015 and requested by FUJITSU LTD., Which refers to a classifier based on convoluted neural networks, a method of classification by using a classifier based in convoluted neural networks and a method for classifier training based on neural networks convoluted The network based classifier convolved neurals comprises: a plurality of feature mapping layers, at least one map of characteristics in at least one of a plurality of function mapping layers that are divided into a plurality of regions; and a plurality of templates convolutional corresponding to the plurality of regions, respectively, each of the templates convolutional is used to obtain a value response of a neuron in the region correspondent.

According to the present invention, [FIG.1a] is a example that illustrates how data passes through different types of tests, in order to take a decision in a three layer network.

[FIG.1b] is an example that illustrates how network input layers contain neurons that encode the values of the input pixels. The data from this neural network are formed with images of 28 x 28 pixels, so the input layer contains 784 = 28 × 28 neurons.

[FIG.1c] is an example that illustrates a possible architecture, with rectangles denoting the subnets This is not meant to be a realistic approach to solve the problem of detection and identification of plant species, it is only by way of example for understand how neural networks work convolutional

Example of Rapid Diagramming of the Process of Detection and Identification used in Spraying Herbicide Selectively on Weeds in a Soya plantation.

Detection: The rapid classification of plants allows us to distinguish weed culture [FIG. 2a].

Analysis: The set of devices detects weed. Its area and perimeter are calculated to download the exact agrochemical dose in place correct [FIG.2b].

Spraying: Once the species is detected vegetable and its size by the system, taking into account the forward speed of the equipment, the valve opens solenoid bypassing the dose of agrochemical, spraying precisely and accurately on the species vegetable [FIG.2c].

Accessory, the set of devices allows to determine the state of the crop, and being the agrochemical a herbicide, a foliar fertilizer, a insecticide, a fungicide, or a protective compound, is you can apply the appropriate agrochemical according to each circumstance.

For example, the method allows you to select a specific herbicide of a set of herbicides for each weed identified with respect to the crop; or a specific foliar fertilizer of a set of foliar fertilizers for the crop identified according to its state; or a specific insecticide of a set of insecticides for the crop identified according to its state of deterioration; or a specific fungicide of a set of fungicides for cultivation identified according to its state of deterioration; and / or a specific protective compound of a set of protective compounds for the identified crop according to your condition

Processing Flow Step by Step in the Weed Detection and Identification Process

According to the present invention, the process flow step by step in the process of detection and identification of the plant varieties of interest is as follows: 1) A Real Time Video Stream [FIG.3] is obtained from one or several cameras positioned along the wings or arms of, for example, a spraying machine. This stage is done at 60 frames per second on shutters per second [FIG. 4]. The number of shutters (or shutters) per second depends on the speed of the moving vehicle. 2) Each of the frames obtained is processed. 3) The frame is converted to a numerical matrix with the representation of RGB colors (Red-Green-Blue English) of each pixel in the image [FIG. 5]. Each pixel has blue, green and red components. Each of these components has a range of 0 to 255, which gives a total of 2,563 different possible colors. 4) The matrix is cut to select the area of the frame to be processed [FIG. 6]. A horizontal strip size of the image to be processed is determined. This area to be processed is taken according to the subsequent ability to open the sprinkler for the application of the agrochemical exactly on the specific area. The area to be processed is exactly the middle strip, since it maintains an optimal ratio of distance to the camera, low image distortion and time that will pass between the processing and subsequent application of the agrochemical, and success in the shot on the plant . 5) An area of the image is assigned to the corresponding sprinklers, so if weeds are detected in that area, the opening order is sent to the corresponding sprinkler. 6) 4 filters are applied to obtain a mask of the predominant colors of the plant species to be identified. In this way, for example, we separate the soil, dry plant residue, stones, etc. The first filter transforms the matrix to YCbCr color format and makes a logical operation between the channels, depending on the color to be filtered [FIG. 7a]. The second filter subtracts two channels in the RGB format, depending on the color to be filtered [FIG.7b]. The third filter is a logical AND (bitwise) operation between the results of the two previous filters [FIG.7c]. Finally, the fourth filter applies a blur [FIG.7d1] or Gaussian blur to the previous result, the image is converted to black and white [FIG.7d2], and the noise [FIG.7d3] which are the points is eliminated scattered whites. 7) The contours of the image on the color mask are identified [FIG. 8] and the position information of each one is saved. 8) An estimated calculation of the velocity is made with the positions of the contours found in the current frame and their positions in a previous frame. A pixel speed per frame is obtained and a pixel-meter ratio and the frame-second ratio are used to make the speed passage to meters per second. With that speed and with the actual distance of the sprinklers to the plants that are seen in the strip of the image that is processed, the exact time in which the sprinklers should be instructed to spray is determined if a weed is then detected. 9) The image is cut into small squares that contain these contours [FIG. 9]. If a contour is very wide, it is separated in two horizontally, if a contour is very high, it is separated in two vertically and if a contour is very large, it is separated into four squares. Finally, squares of the same approximate size are obtained. 10) Each of these squares cut out of a part of the image is resized to a preferred size of 256 x 256 pixels, which is the size that the convolutional neural network works internally [FIG. 10]. 11) The 256 x 256 pixel image squares [FIG. 11a] are sent to the first layer or input layer of the convolutional neural network previously trained for analysis and categorization [FIG. 11b]. 12) Each square is processed within the Neural Network, which can process several at a time. The processing is carried out in parallel and executed internally in the GPU and not on the CPU, to achieve great performance in arithmetic operations. Previously trained within the network, the picture is taken and is performed in parallel one pass (forward) and the result to the output layer of the neural network is sent. 13) The result of the last layer or output layer of the convolutional neural network is obtained. The result of the output layer gives us an average value of success of each of the categories, the highest value being the category of plant species to which the image belongs [FIG. 12]. In one of the categories of the neural network are the NO plant species, in which there are unknown plant species and / or elements not to be taken into account, earth, sky, etc. If the result is this category, it means that in the analysis of this picture box there is no category of known or pre-trained plant species. 14) Depending on the numerical value of the identified plant species, the agrochemical to be used is determined. The system contains a table with the possible plant species to be identified and their relationship with the agrochemical to be used according to diagnosis, if applicable. 15) The device for the detection and identification of plant species already has a complete representation of the identified plant species and the agrochemical required to be applied in each of the plants that appear in the processed frame that was obtained from the main video stream [FIG. 13]. 16) For each image processed by the neural network, which is associated with a sprinkler by the area where it is located, the order is sent according to the identified species, so that the spray is exactly on the plant species to be treated. The mathematical calculation of the exact moment of activation of the electromechanical order is made, taking into account the speed of the spray vehicle and the distance from the chamber to the ground. Depending on the area of the processed frame, only the electromechanical valve corresponding to the specific field of action is activated. This avoids the spraying of agrochemicals in the places that do not apply and only applies to the previously identified plant. Depending on the required agrochemical, the valve corresponding to the specific agrochemical to be used is activated. It allows to administer multiple tanks of agrochemical product according to a specific need.

A field test of the whole was carried out Autonomous device for detection and identification of plant species in a crop agricultural for the application of agrochemicals in the form selective in the town of Las Rosas, Province of Santa Fe, on a lot of 20 hectares planted with soy.

The herbicide selected to be applied was glyphosate (RoundUp) to approximately 1.4 liters per hectare on average.

An autonomous type sprayer was used mosquito (Pla, MAP II 3250 model) composed of a 3250 liter tank, where the side arms they included a line of pads mounted TeeJet brand sprayers, commanded with solenoid valves connected directly to the computer that controls dose application of herbicide.

The height of the peaks and sensors with respect of the floor was 1 meter.

The propulsion speed of the fumigator was approximately 16 km / h during The whole application.

The herbicide application on the soybean weed was made at 10 morning hours

A quantity of 12 cameras was used with sensors distributed evenly throughout the wing of the boom 28 meters long.

The batch chosen for the trial was cultivated with 4-week post-emergence soybeans and it had a low percentage of weeds and high concentration "staining", this is weed randomly distributed in patches.

From the field test by using the autonomous set of devices according to the present invention, it was possible detect the presence of weeds on the crop with a certainty percentage that varied between 76% and 92%, where the level of success depended on the machinery movements within the crop and of the incidence of lighting (light and shadow) on The detector sensor.

The percentage of savings obtained in the herbicide application, when applied only on weeds detected reached 86% of product on what would have been an application of total coverage at 2 liters per hectare.

Claims

A stand-alone set of devices for the detection and identification of plant species in a agricultural cultivation for the application of agrochemicals in selectively, said set CHARACTERIZED BECAUSE comprises: a product application device chemicals comprising at least one container container of agrochemicals linked in fluid communication with a plurality of sprinkler peaks through a valve, a plurality of cameras that are arranged on the autonomous vehicle and focused on cultivation, where each camera has an associated ultrasound sensor for the height measurement to the crop in real time, and in where each camera is tilted forward at 45 degrees from normal; a device of detection and identification of plant species connected to the cameras to receive information from video of them, an electronic circuit in charge to manage the opening and closing of the valves of the sprinkler peaks of agrochemicals connected to detection device that manages through said circuit the opening and closing of the valves of the sprinkler peaks, and where, the set of devices is mounted on a vehicle of transport.
The autonomous set of clause 1, CHARACTERIZED BECAUSE the transport vehicle is a self-propelled vehicle or a tow vehicle.
The autonomous set of devices of the clause 2, CHARACTERIZED BECAUSE the vehicle self-propelled is a vehicle for fumigation with arms sides arranged perpendicularly to it (mosquito).
The autonomous set of devices any of clauses 1 to 3, CHARACTERIZED BECAUSE The detection and identification device consists of a processor
The autonomous set of devices of the clause 4, CHARACTERIZED BECAUSE the processor comprises a tool based on computer software developed in C ++ language, a vision framework artificial, and a neural network framework convolutional
A method for the detection and identification of plant species in an agricultural crop for application of agrochemicals selectively with the autonomous set of devices of any of the clauses 1 to 5, said method CHARACTERIZED BECAUSE Understand the steps of: a) detect and classify plant species in an agricultural crop during travel of the autonomous set of devices through of the crop; b) analyze the information obtained in a) determining area and perimeter of weeds; and c) spray the appropriate dose of an agrochemical in the right place considering forward speed of the team.
The method for the detection and identification of plant species in an agricultural crop clause 6, CHARACTERIZED BECAUSE step a) allows to distinguish the weed cultivation
The method for the detection and identification of plant species in an agricultural crop clause 6 or 7, CHARACTERIZED BECAUSE step a) allows identify plant species to determine the agrochemical to apply.
The method for the detection and identification of plant species in an agricultural crop of any of clauses 6 to 8, CHARACTERIZED BECAUSE in step c) the dose of agrochemical is sprayed by opening a solenoid valve.
The method for the detection and identification of plant species in an agricultural crop of any of clauses 6 to 9, CHARACTERIZED BECAUSE the plant species correspond to the crop and weeds.
The method for the detection and identification of plant species in an agricultural crop of any of clauses 6 to 9, CHARACTERIZED BECAUSE the agrochemical is a herbicide, a foliar fertilizer, a insecticide, a fungicide, or a protective compound.
The method for the detection and identification of plant species in an agricultural crop of any of clauses 6 to 11, CHARACTERIZED BECAUSE step b) It also allows to determine the state of the crop.
The method for the detection and identification of plant species in an agricultural crop clause 11 or 12, CHARACTERIZED BECAUSE step c) comprises also select a specific herbicide from a set of herbicides for each weed identified in step a) regarding the crop.
The method for the detection and identification of plant species in an agricultural crop clause 11 or 12, CHARACTERIZED BECAUSE step c) comprises also select a specific foliar fertilizer from a set of foliar fertilizers for cultivation identified in step a) according to their status.
The method for the detection and identification of plant species in an agricultural crop clause 11 or 12, CHARACTERIZED BECAUSE step c) comprises also select a specific insecticide from a set of insecticides for the crop identified in step a) according to its state of deterioration.
The method for the detection and identification of plant species in an agricultural crop clause 11 or 12, CHARACTERIZED BECAUSE step c) comprises also select a specific fungicide from a set of fungicides for the crop identified in step a) according to its state of deterioration.
The method for the detection and identification of plant species in an agricultural crop clause 11 or 12, CHARACTERIZED BECAUSE step c) comprises also select a specific protective compound of a set of protective compounds for cultivation identified in step a) according to their status.
The method for the detection and identification of plant species in an agricultural crop of clause 6, CHARACTERIZED BECAUSE it comprises the steps of: a) obtaining a Real Time Video Stream from a plurality of cameras positioned along the wings of the autonomous set of devices for the detection and identification of plant species in an agricultural crop for the application of agrochemicals in a selective way; b) process each of the frames obtained; c) convert the frame to a numerical matrix with the representation of RGB (Red-Green-Blue English) colors of each pixel in the image; d) trim the matrix to select the area of the frame to be processed; e) assign an area of the image to the corresponding sprinklers, so that if weeds are detected in that area, the opening order is sent to the corresponding sprinkler; f) apply 4 filters to obtain a mask of the predominant colors of the plant species to be identified; g) identify the contours of the image on the color mask by saving the information of the position of each one; h) estimate the travel speed with the positions of the contours found in the current frame and the positions of those same contours in a previous frame; i) obtain a pixel speed per frame and make the speed passage in meters per second using a pixel-meter ratio and a frame-second ratio; j) crop the image into small squares that contain the contours; k) resize each of the squares cut out of a part of the image to a preferred size; l) send the squares of images to the first layer (input layer) of the previously trained convolutional neural network for analysis and categorization; m) processing each square within the previously trained neural network, wherein the image is taken, it is performed in parallel in one pass (forward) and the result is sent to the output layer of the neural network; n) obtain the result of the last layer (output layer) of the convolutional neural network; ñ) determine the agrochemical to be used based on the numerical value of the identified plant species; o) correlate the complete representation of the plant species identified with the agrochemical necessary to apply in each of the plants that appear in the processed frame that was obtained from the main video stream, and p) send the order according to the species identified for each image processed by the neural network that is associated with a sprinkler by the area where it is located, so that the spray is exactly on the plant species to be treated;
The method for the detection and identification of plant species in an agricultural crop clause 18, CHARACTERIZED BECAUSE in step f) it separates foreign elements such as soil, dry plant residue and stones of plant species, where: a first filter transforms the matrix to YCbCr color format; a second filter subtracts two channels in RGB format depending on the color to filter; a third filter is a AND logical operation (bit by bit) between the results of the first and second previous filters; and a quarter filter applies a blur to the previous result Gaussian), converting the image to black and white and Eliminating noise