KR102188521B1 - Method and Apparatus for Identification of Crops and Weeds with Neural Network Learning - Google Patents

Abstract

The present invention includes the steps of recognizing the heat of crops using a Convolutional Encoder-Decoder (CED) neural network technology to recognize all plants between rows as weeds; For a small number of weeds present in the crop laces, it relates to a technique for identifying crops and weeds, which consists of steps to identify weeds from the crops using an additional CED neural network. Convolutional Encoder-Decoder (CED) neural network consists of several stages between the input terminal and the output terminal, and it is a structure that decreases and grows in size as it goes to the middle stage, and is a neural network that performs convolutional operations on each stage. The present invention is composed of two neural networks, such as a CED neural network for extracting crop rows and a CED neural network for identifying crops and weeds. CED neural networks have various structures modified from the basic structure, but CED neural networks of basic structures or other modified structures can also be used in the practice of the present invention. In the CED neural network that extracts the crop heat, crop images are input to the input side, and the learning target images are displayed as graphic lines at the positions corresponding to the crop rows of the input images, and they are learned, thereby extracting the crop rows from the crop images. Let them build their own skills. In addition, crop-weed identification CED neural network inputs a close-up image of a crop thermal image, and a different kind (color/shape) for each type of crop or weed at the position corresponding to the crop or weed in the input image. By placing the symbol of and displaying the image as a learning target image and letting them learn, the technology to identify crops and weeds is built on their own through learning. When the crop row position information and the row weed position information obtained through the present invention are sent to the mower, the mower sends the weeding device to the corresponding position so that the weeding can be performed. The present invention can be applied to both rice field crops and field crop cultivation, but the present specification shows an example of using the present invention to identify blood that is most difficult to distinguish from rice among weeds in rice field farming.

Description

{Method and Apparatus for Identification of Crops and Weeds with Neural Network Learning}

The present invention is a technology belonging to the field of image recognition of a neural network. First, a CED neural network is used to recognize the heat of a crop, so that all plants other than the heat are regarded as weeds and are collectively removed, and the heat and a few weeds remaining around it are removed. It relates to a weed identification method characterized in that the weeds can be precisely identified by learning another CED neural network.

Weeding is one of the agricultural works that require the most labor for growing eco-friendly crops. If weeding is not done effectively, it can cause fatal damage to the growth of crops, so farmers have a difficulty in weeding removal throughout the growing season. Therefore, automation of weeding work is a dream of farmers, and research for this has been attempted for a long time.

Zhang et al. (1995) analyzed and suggested the criteria for distinguishing weeds found in wheat fields in three aspects: color, shape, and texture. Woebbecke et al. (1995a) performed color analysis for separation of weeds and background in images. In particular, it was shown that the crystal color, 2g-r-b, and green chromatic coordinates distinguish weeds from the surrounding environment. Tian et al. (1997) developed and tested a mechanical visual system capable of locating young tomatoes and weeds in the field. In Korea, research has been conducted in this field as well.Seo-in Jo et al. (1999) conducted a study showing the possibility of detecting weeds in the open field by extracting their geometrical features using purslane, barangi, and Myeongaju as target weeds. I did. However, since the weed recognition method using the color, shape, and texture spectrum of the plant is based on rules, it is not adaptable to variously changed environments and shapes, so it is insufficient for practical use.

There was a need for a robust weed recognition technology that can be applied in a variety of environments. As the deep learning neural network technology has been greatly developed recently, there have been attempts to recognize weeds using it. Cicco et al. (Dec 2016, CVPR) trained the dataset created using a graphic tool on SegNet (CoRR 2015), one of the Deep Learning neural networks, to distinguish weeds and crops. In addition, Potena et al. (Feb. 2017) significantly improved the accuracy of classification compared to existing algorithmic methods by using Deep Learning neural network technology for classification of crops and weeds. In these methods, since the aim was to extract the shape of the whole weed separately, it can be applied to sparsely distributed weeds, but in an environment where many grasses and crops are mixed, the plants overlap each other and there is no distinction. Therefore, practical use is difficult. As a slightly different approach, a method of recognizing the location of nearby crops using a mechanical contact sensor (application number 10-2013-0115057) is also suggested, but it is difficult to apply to young mothers, and to solve this problem, A method of using a laser sensor and antennae to be applicable {application number 1020090113990 (2009.11.24)} has also been developed, but it has the disadvantage that the position of the young mother swaying in the wind is not accurate and is only effective in a very close range.

references

Zhang, N. and C. Chaisattapagon. 1995. Effective criteria for weed identifying in wheat fields using machine vision. Transactions of the ASAE 38(3):965-974.

Woebbecke, D. M., G. E. Meyer, K. Von Bargen and D. A. Mortensen. 1995a. Shape features for identifying young weeds using image analysis. Transactions of the ASAW 38(1): 271-281.

Tian, L., D. C. Slaughter and R. F. Norris. 1997. Outdoor field vision identification of tomato seedlings for automated weed control. Transactions of the ASAE 40(6):1761-1768.

Cho Sung-in, Lee Dae-sung, and Bae Young-min. 1999. Weed identification using machine vision. Journal of Korean Agricultural Machinery Expansion Vol. 24, No. 1, 59-66

aurilio Di Cicco, Ciro Potena, Giorgio Grisetti and Alberto Pretto. 2016. Automatic Model Based Dataset Generation for Fast and Accurate Crop and Weeds Detection. CVPR. arXiv:1612.03019v1 [cs.CV] 9 Dec 2016C

iro Potena, Daniele Nardi, and Alberto Pretto. 2017.Fast and Accurate Crop and Weed dentification with Summarized Train Sets for Precision Agriculture. Intelligent Autonomous Systems vol 14, 105-121. Feb 2017.

The present invention for solving the above problems is to develop a technology for extracting crop heat from a crop image and a technology for identifying crops and weeds using artificial intelligence technology, and learning a technology for specifying the position to be weeding. It is about the technology that lets you build yourself through.

In order to achieve the above object, the present invention extracts heat from crops using the Convolutional Encoder-Decoder (CED) neural network technology to allow a mower to remove the plants between the heat and the heat as weeds; For the small number of weeds present on the crop row, an additional CED neural network is used to identify and remove weeds from the crop.

The system structure for this is composed of a CED neural network used to extract heat from crops and a CED neural network used to identify crops and weeds. The CED neural networks are all composed of several stages between the input terminal and the output terminal, have a narrower structure toward the middle stage, and consist of a CED neural network that performs convolutional operations on each stage. The CED neural network has various structures modified from the basic structure as shown in FIG. 1, and there are differences in performance and characteristics between them, but any modified structure of the CED neural network is the object of the present invention, which is to extract heat from crops and distinguish crops and weeds. Can be used for

For the CED neural network that extracts the rows of crops, a training data set that uses a crop image as an input image of the CED neural network and a line image displayed by graphically drawing the positions of crop rows in the input image as a learning target image. Provided in large quantities;

The CED neural network iteratively learns the training data set to self-learn a technique for extracting crop heat from a crop image.

In addition, for the crop-weed identification CED neural network, an image photographed close enough to identify the shape of individual crops and weeds is used as an input image, and symbols of different shapes or colors at the positions of crops or weeds in the input image Preparing a large amount of training data sets using the image displayed as the learning target image;

By repeatedly learning the training data set, the CED neural network identifies crops and weeds from the field/field image, and makes it self-learning through neural network learning.

The present invention does not rely on an algorithm based on an existing image processing technology to achieve the goal of extracting crop heat and identifying crops/weeds, but directly creating an image desired to appear as an output image as a graphic and learning it. Therefore, there is an effect of developing a new technology development method that allows the CED neural network to build itself through learning the technology required to obtain the desired image recognition result. In addition, by using this technology to recognize the heat of the crop and to identify the crop-weeds, there is an effect that the mower can accurately and automatically weed.

1 is a structure of a Convolutional Encoder-decoder neural network usable in the present invention;
A) Convolutional Encoder-decoder Neural Network Structure Used for Thermal Recognition of Crop (Rice)
B) Convolutional Encoder-decoder neural network structure used for identification and location recognition of crops (rice) and weeds (blood)
C) Unet (skip) CED neural network, d) Dense CED neural network and DensNet blocks are a modified CED neural network structure that can be used to realize the present invention
2 is a structure of a system for detecting crop heat and crop-weed identification and location for weeding;
3 illustrates some of the training data of the CED neural network for crop heat extraction;
4 is an example of a test result of the learned CED neural network for extracting crop heat;
FIG. 5 is an illustration of some of the learning data of the CED neural network for distinguishing hair and blood; FIG. 6 is an example of the test result of the CED neural network for identifying the learned crop-skin.

The Convolutional Encoder-Decoder (CED) neural network used in the present invention is composed of several stages between the input terminal and the output terminal as shown in FIG. 1, and is a neural network having a structure that gradually decreases and increases in size. An increasingly smaller part of the first half is called an encoder part, and an increasingly larger part of the second half is called a decoder part. 1(A) and 1(B) are the CED neural network structures used in the implementation of the crop heat recognition and the weed detection method on the crop heat in the present invention, but the two structures may be interchangeably implemented. In addition, as shown in Fig. 1(c), a modified CED neural network called Unet or skip structure in which the outputs of each layer of the encoder part are skipped to the input of the same layer of the decoder part. A structured CED neural network can also be used, and a modified CED neural network structure called DensNet having connections across each layer of the Unet structure as shown in Fig. 1(D) can also be used. In addition, it is also possible to use a neural network modified differently based on the CED neural network. In this case, the present invention can be implemented by employing a different number of layers and a different number of filters for each structure, and the object of the present invention can be achieved.

The present invention includes a crop heat recognition step of recognizing heat of crops using the Convolutional Encoder-Decoder (CED) neural network technology; It consists of a crop-weed identification step that allows the few weeds present on the crop row to be identified from the crop using an additional CED neural network. The hardware for this is composed of two neural networks, such as the CED neural network 230 used in the step of recognizing the heat of crops and the CED neural network 260 used in the step of identifying weeds from the crops as shown in FIG. do.

Each CED neural network presents the task to be performed to the CED neural network as an example by applying the output to be obtained as an output along with the input image as a learning database (220 and 250), and the neural network is the task through learning. Lets you install the skills to perform.

To this end, first, a large amount of crop images are obtained, and the graphic display image 210 of the crop column position and the symbol display image 240 of the crop-weed position for each crop image are produced in large quantities, and the crop image-crop column image database (220) and crop image-type and location image database 250 of crops and weeds are configured and stored in a computer hard disk. The crop image-crop thermal image database 220 and the crop image-crop and weed type and position image database 250 are CED neural network 230 for recognizing each crop row and the crop and weed identification and location recognition It is learned by the CED neural network 260. The learned neural network has the data form of each connection parameter value for the structure of the neural network designed in advance. The following is a detailed description of the learning process of the CED neural network 230 for crop heat recognition and the CED neural network 260 for identification and location recognition of crops and weeds.

Learning to recognize crop heat

In order to develop a technology for recognizing crop heat in the CED neural network, training data as an example are prepared in the form of images on a large scale. 3 illustrates a case of seedling (rice) among crops. When input images such as the left images are applied, a task is assigned to the CED neural network to extract and present the thermal images of the crop such as the images on the right. will be. More specifically, the learning target image is presented by displaying graphical lines like the images on the right side of FIG. 3 along a position corresponding to each column of the simulation as the left images of FIG. 3. That is, when the right image is superimposed on the left image, each line of the right image is produced in a learning target image that is located at the center of the crop row in the left image. However, when the mower is weeding along the rows of crops, it follows the rows of crops in the front, so when producing the learning objective video, the lines in the front are drawn with emphasis, and the rows facing left and right can be omitted. In addition, as shown in Fig. 3(D), an extension line is made even to the location where the hair is lost in the middle, so that the neural network learns the human skill of drawing the extension line.

When the learning data is prepared in this way, augmentation techniques such as rotation, enlargement and rotation, reduction and rotation of various angles, and vertical movement of the left and right are used to learn more various shapes when learning about this. After creating a much larger number of training data through the program, the neural network is trained. The learning used at this time uses the backpropagation learning method commonly used for neural network learning, and repeats learning until the error falls below a predetermined threshold.

4 is a diagram illustrating the test result of the CED neural network learned for crop heat recognition. When an input image, such as the leftmost position, is applied to the CED neural network, the output is an image such as the middle position. You can get them. In order to check whether the lines on the output images accurately designate the crop row position of the input images, the images in which the CED neural network outputs are superimposed on the input images are the same as the rightmost images of FIG. 4. As can be seen in the figure, it can be seen that the positions of the output lines of the CED neural network accurately specify the positions of the crop rows in the input image.

Learning to identify individual crops and weeds

In the present invention, a neural network having a structure similar to that of the Convolutional Encoder-Decoder (CED) was used for identification between crops and weeds. The training data is displayed as one type of symbol at the base of the crop in the learning target image, and an image represented by a different type of symbol at the base of the weed is created so that the neural network learns it. If a symbol is displayed at the base, effective weeding is possible by crushing this area by a weeding machine.

The images on the left of FIG. 5 are images of rice and blood used as a neural network input, and the images on the right are learning target images, indicated by a gray circular symbol at a place corresponding to the base of rice in the left images, and a place corresponding to the base of blood is black. These are examples of images displayed with circular symbols. For the training data of the CED neural network to identify the hair and blood of rice, a large amount of images such as the left image are taken and prepared as input images, and learning target images such as the right image are created for each input image. The learning target image here is an example in which an image in which gray and black circular symbols are displayed at the base of mona blood on a blank image is shown, but the symbols may be superimposed on the input image. In addition, various colors and shapes of symbols to be displayed can be selected. In this way, a learning data set is constructed and learned until an error is less than a predetermined threshold using a backpropagation learning method. 6 illustrates some of the test results of the CED neural network for crop heat recognition. The leftmost images of FIG. 6 are input test images and are images not included in the training data. The middle images of FIG. 6 are neural network output images, and the images on the right are images in which the input image and the output images are superimposed. In the output image and the superimposed image, the gray circular symbol indicates the position of rice, and the black circular symbol indicates the position of blood. As you can see in the resulting image, it can be seen that the location of hair and blood is accurately recognized and displayed. However, it is not possible to identify rice paddies that do not show the base because the rice paddies overlap, but this is because we learned to identify them based on the shape of the base. However, it can be seen that most of the blood of the size and location that can clearly confirm the shape is identified and detected.

The resulting images recognized in this way are sent to the weeding robot to find the actual location corresponding to the hair and blood so that it can weed.

The role of the weeding system as the brain

In order to use the present invention as a brain role of the weeding system, an image is acquired in real time through a camera 280, and the acquired image is used for the CED neural network 230 for crop heat recognition and the identification and location of crops and weeds. It must be simultaneously applied to the edible CED neural network 260 and processed at high speed. The CPU 100 is used to control the flow of such information and to configure and execute the neural network in software. In addition, the CPU 100 performs a role of post-processing and analyzing the outputs of neural networks, and using the result to generate an appropriate control signal required for autonomous driving of an externally connected mower. During this process, in order to configure CED neural networks and perform their operation, a special high-speed signal processing is required. For this purpose, the GPU 110 is used as an auxiliary device.

Composition of mower control system

Since the parameters of the learned CED neural network already include information necessary for crop heat recognition and crop-weed identification extracted from the crop image-crop thermal image database 220 and the crop image-crop type and location image database 250 For miniaturization of the device, the databases are removed, and a camera 280, a CED neural network 230 for crop heat recognition, and a CED neural network 260 for weed identification and location recognition; The mower control system 300 can be simply configured with only the CPU 100 and the GPU 110.

In order for the configured mower control system to send an appropriate weeding operation command to the mower 290, an image is first acquired through the camera 280 in real time, and the acquired image is obtained from the CED neural network 230 for crop heat recognition and the crop. It must be simultaneously applied to the CED neural network 260 for identification and location recognition of weeds and processing at high speed. The CPU 100 is used to control the flow of such information and to configure and execute the neural network in software. In addition, the CPU 100 performs a role of post-processing and analyzing the output of the neural network, and using the result to generate an appropriate control signal required for the autonomous weeding operation of the mower 290 connected to the outside. During this process, in order to configure the CED neural network and perform its operation, a special high-speed signal processing is required. For this purpose, the GPU 110 is used as an auxiliary device.

100: CPU
110: GPU
200: crop weed identification system
210: Produce graphic display image of crop row position
220: crop image-crop thermal image database
230: CED neural network for crop heat recognition
240: crop-weed position symbol display video production
250: crop image-crop type location image database
260: CED neural network for identification and location recognition of crops and weeds
280: camera
290: mower

Claims (13)

Recognizing the rows of crops by CED neural network learning for weeding for cultivation of paddy or field crops, so that all plants out of the rows are regarded as weeds and removed; A method of identifying crops and weeds by CED neural network learning, comprising the step of separating and identifying the rows and weeds remaining around them through another CED neural network learning. The method according to claim 1, wherein the CED neural network used in the step of recognizing the heat of the crop and the step of identifying crops and weeds is composed of several convolutional layers, an encoder that becomes smaller and a decoder that becomes larger again ( A method of identifying crops and weeds by learning a CED neural network, characterized by having a CED neural network in which the decoder) part is combined The CED neural network according to claim 2, wherein the CED neural network is a CED neural network in which the outputs of each layer of the encoder part are summed to the input of the same layer of the decoder part, that is, a modified Unet structure. A method of identifying crops and weeds by learning a CED neural network, characterized in that a structured CED neural network can also be used The CED neural network of claim 3, wherein the CED neural network has a CED neural network called a Unet structure as a basic structure and a modified structure called DensNet, which has an additional connection that skips each layer. A method of identifying crops and weeds by learning a CED neural network, characterized in that it can also be used The method of claim 1, wherein the method of recognizing the heat of crops is that when a crop image is applied as an input of the CED neural network, the CED neural network is the output terminal of the CED neural network at a location corresponding to the crop row in the crop image applied as an input. The CED neural network is repeatedly trained to be displayed by drawing a graphic line on the CED, and by learning the same method for many other crop row images, when a random test crop image is applied, the location corresponding to the crop row is displayed as a graphic. A method of identifying crops and weeds by learning a CED neural network, characterized in that the line is drawn and displayed The method of claim 1, wherein when the crop is rice and the weeds are blood, the method of distinguishing them is that in the case of the mock of rice, several rice seeds are collected and germinated. Because it is scattered and germinated, it is characterized by discriminating rice and blood by using the visual difference in the density of the root of the plant by using the fact that the density of the root of the plant is low. Identification method The method of claim 6, wherein the learning target image data for distinguishing rice and blood by using the visual difference in the density of the base is displayed in a symbol of a different color or shape at a place corresponding to the base of rice and blood in the input image. A method of identifying crops and weeds by learning a CED neural network, characterized by creating a learning objective image for discriminating rice and blood to distinguish rice and blood The method according to claim 5, wherein the database required for learning a CED neural network for crop heat recognition includes, as an input image, a crop heat image taken from a direction to be followed by a mower; An image created with a graphic line at a position corresponding to each crop row in the input image as a learning target image; A method of identifying crops and weeds by learning a CED neural network, characterized by a method of preparing a database for learning a crop row in which a large amount of a learning dataset consisting of one set of the input image-learning target image is prepared. The method of claim 6, wherein the learning database required for the CED neural network for crop-weed identification captures images including crops and weeds on the crop row, and close-up images so that the shape of the crops and weeds can be clearly distinguished. As the input image of the CED neural network; In each input image, an image using symbols of different shapes or colors is created so that the positions corresponding to each crop and weed can be distinguished, and this is used as the learning target image; A method of identifying crops and weeds by learning a CED neural network, characterized by a method of creating a database for learning to identify crops and weeds having a large amount of training data sets including the input image-learning target image as a set. The method of claim 9, wherein the creation of the learning target image in the learning database required for the CED neural network for crop-weed identification is performed in a variety of ways to distinguish each object on an initially blank image or an image copied from the input image. A method of identifying crops and weeds by learning a CED neural network, characterized by a method of creating a learning objective image that is created by graphically using color lines, figures, or symbols The method of claim 9, wherein when creating a learning target image in a learning database required for a CED neural network for crop-weed identification, the position and size of each symbol are determined based on the input image; A CED neural network characterized by a method of displaying the location and size of the symbol in the learning target image, which is displayed on the learning target image by determining the location and size of the symbol center point so that as many features with high discrimination between objects can be included in the symbol area. How to identify crops and weeds by learning A crop image-crop thermal image database 220 created as a graphic display image of a crop row position produced by the method of claim 5 or 8; A CED neural network (230) for crop heat recognition learning by the method of any one of claims 1 to 4 as the image database (220); Crops produced by the method of claim 6, 7 and 9 to 11-crop image created with symbol display images of weed types and locations-crop type location image database 250; A CED neural network (260) for recognizing the types and locations of crops and weeds that are learned by the method of claim 1 to the image database 250; A camera 280 that captures a crop-weed image in real time and applies it as an input of the CED neural networks 230 and 260; A CPU that configures and executes the CED neural network in software, analyzes the result of the CED neural network, uses the result to generate a control signal necessary for the autonomous weeding operation of an externally connected mower, and provides it to the mower (100) and; An apparatus for identifying crops and weeds by learning a CED neural network, comprising: a GPU 110 that assists the CPU and performs a CED neural network at high speed. The method of claim 12, wherein the parameters of the learned CED neural networks include: a crop image-crop thermal image database (220); Since the crop image-crop type and location image database 250 includes information necessary for crop heat recognition and crop-weed identification, the databases are removed for miniaturization of the device, and the camera 280 and crop heat recognition CED neural network 230 and; CED neural network 260 for identification and location recognition of weeds; A CPU 100; A device for identifying crops and weeds by learning a CED neural network, characterized by simply configuring a mower control system with only the GPU 110

Images