WO2019083336A1

WO2019083336A1 - Method and device for crop and weed classification using neural network learning

Info

Publication number: WO2019083336A1
Application number: PCT/KR2018/012883
Authority: WO
Inventors: 김형석; 박동선; 아디카리샴; 양희찬; 김용진; 양창주
Original assignee: 전북대학교산학협력단
Priority date: 2017-10-27
Filing date: 2018-10-29
Publication date: 2019-05-02

Abstract

The present invention relates to a technology for crop and weed classification, comprising the steps of: recognizing rows of crops by using a convolutional encoder-decoder (CED) neural network technique, so as to classify all plants between the rows as weeds; and for a small number of weeds located in the rows of the crops, distinguishing the weeds from the crops. A CED neural network for extracting rows of crops receives an image of crops as an input image and learns a to-be-learned image obtained by marking graphic lines on the input image at locations corresponding to rows of the crops, so as to develop, by itself, a skill of extracting rows of crops from an image of crops; and a CED neural network for crop-weed classification receives, as an input image, a close-up captured image of crops and learns a to-be-learned image obtained by marking different kinds (colors/shapes) of symbols according to species of crops or weeds on the input image at locations corresponding to crops or weeds, so as to develop, by itself, a skill of classifying crops and weeds. Information on the locations of rows of crops and information on the locations of weeds on the rows, acquired through the technology of the present invention, can be transmitted to a weeding machine so as to allow the weeding machine to transfer the weeding tool thereof to the corresponding locations and perform weeding.

Description

Method and apparatus for identifying crops and weeds by neural network learning

The present invention belongs to the image recognition field of a neural network. First, the CED neural network is used to recognize the heat of a crop, and all the plants other than heat are regarded as weeds and are collectively removed, and a small number of weeds To be precisely identified by the learning of another CED neural network so as to be weeded.

One of the agricultural work that needs the most labor to cultivate environmentally friendly crops is weeding. Farmers are having difficulty with hanging out weeds throughout the crop growth period because the herbicide is not effective and causes catastrophic damage to crops. Therefore, the automation of weeding work is a dream of farmers, and research for this has long been attempted.

Zhang et al. (1995) analyzed and presented the criteria that can distinguish weeds found in wheat fields from color, shape, and texture. Woebbecke et al. (1995a) performed color analysis for separating weeds and backgrounds from images. In particular, the corrected color, 2g-r-b, and the chromatic coordinate showed that the weeds were well distinguished from the surrounding environment. Tian et al. (1997) developed and tested a machine vision system that can locate young tomatoes and weeds in the open field. In Korea, we have conducted research in this area, and the authors (1999) conducted a study showing the possibility of detecting weeds in the field by extracting geometric features of weed, weed, Respectively. However, weed recognition methods using the color, shape and texture spectrum of such plants are not adaptable to various changed environments and shapes because they are based on rules.

We need a robust weed recognition technology that can be applied in various environments. Recently, as the Deep Learning Neural Network technology has greatly developed, there have been attempts to recognize weeds using this technology. Cicco et al. (Dec 2016, CVPR) learned the data sets generated by graphical tools on SegNet (CoRR 2015), one of the Deep Learning Neural Networks, to distinguish between weeds and crops. Potena et al. (Feb. 2017) also used the Deep Learning Neural Network technology to classify crops and weeds, which greatly improved the accuracy of classification compared to existing algorithmic methods. These methods are aimed at extracting the shape of the entire weed, so it is possible to apply it to the weed species distributed at the same time, but in an environment where many grasses and crops are mixed, the plants overlap each other and are not distinguishable So it is difficult to put it to practical use. As a different approach, there is also disclosed a method (Patent No. 10-2013-0115057) for recognizing the position of the near-field crops by using the mechanical contact sensor, but it is difficult to apply to the young model. To solve this problem, (Application No. 1020090113990 (Nov. 24, 2009)) was also developed, but there is a disadvantage that the position of the young simulated shaking wind is not accurate and is only very close to the ground.

references

Zhang, N. and C. Chaisattapagon. 1995. Effective criteria for weed identification in wheat fields using machine vision. Transactions of the ASAE 38 (3): 965-974.

Woebbecke, D. M., G. E. Meyer, K. Von Bargen and D. A. Mortensen. 1995a. Shape features for identifying young weeds using image analysis. Transactions of the ASAW 38 (1): 271-281.

Tian, L., D. C. Slaughter and R. F. Norris. 1997. Outdoor field vision identification of tomato seedlings for automated weed control. Transactions of the ASAE 40 (6): 1761-1768.

In addition, 1999. Weed identification using machine vision. Korea Agricultural Machinery Survey Volume 24, Issue 1, 59-66

aurilio Di Cicco, Ciro Potena, Giorgio Grisetti and Alberto Pretto. 2016. Automatic Model Based Dataset Generation for Fast and Accurate Crop and Weeds Detection. CVPR. arXiv: 1612.03019v1 [cs.CV] 9 Dec 2016C

iro Potena, Daniele Nardi, and Alberto Pretto. 2017. Fast and Accurate Crop and Weed Dentification with Summarized Train Sets for Precision Agriculture. Intelligent Autonomous Systems vol 14, 105-121. Feb 2017.

In order to solve the above problems, the present invention provides a technique for extracting crop heat from a crop image using artificial intelligence technology, a technology for identifying crops and weeds, and a technique for designating a location for weeding To construct themselves by means of the network.

In order to achieve the above object, the present invention extracts heat of crops using Convolutional Encoder-Decoder (CED) neural network technology so that weeds are removed as weeds, For a small number of weeds present on the crop line, additional CED neural networks are used to identify and remove weeds from crops.

The system structure consists of CED neural network used for heat extraction of crops and CED neural network used for crop and weed identification. The CED neural networks are composed of several stages between the input and output stages. The CED neural networks have a constricted structure toward the middle stage and a CED neural network that performs convolutional computation at each stage. As shown in FIG. 1, the CED neural network has various structures modified from the basic structure. Although there is a difference in performance and characteristics among them, a CED neural network of any modified structure is used for heat extraction of the crop, Lt; / RTI >

For the CED neural network that extracts the rows of the crops, a learning data set in which a crop image is used as an input image of the CED neural network and a line image in which the positions of crop lines in the input image are graphically displayed is used as a learning target image In large quantities;

The CED neural network repeatedly learns the learning data set so as to learn the technique of extracting the crop row from the crop image.

In addition, for the crop-weed identification CED neural network, an image obtained by taking close-up images so that the shape of individual crops and weeds can be identified is used as an input image, and the shape of the crop or weeds in the input image, A large number of sets of learning data in which an image representing the target image is displayed as a learning target image;

The learning data set is repeatedly learned, and the CED neural network learns the technique of identifying the crops and weeds from the field / field image, and designating the position to the position, through the neural network learning.

In order to accomplish the goal of identifying crops and weeds, the present invention directly draws an image that is desired to appear as an output image, and does not rely on an algorithm based on a conventional image processing technique, and learns the image directly. Therefore, it is effective to develop a new technology development method that allows the CED neural network to construct the technology required for obtaining the desired image recognition result through learning. It also has the effect of recognizing the heat of the crop and identifying the crops - weeds using this technology, so that the mowers can precisely and automatically weed.

Figure 1 is a convolutional encoder-decoder neural network structure usable in the invention;

A) Convolutional encoder-decoder neural network structure used for thermal recognition of crops (rice)

B) Convolutional encoder-decoder neural network structure used for identification and localization of crop (rice) and weed (p)

C) Uned (skip) CED neural network, d) Dense CED neural network and DensNet blocks are the modified CED neural network structure

Figure 2 shows the structure of crop row and crop-weed identification and position detection system for weeding;

Figure 3 illustrates some of the training data of the CED neural network for crop row extraction;

FIG. 4 is an illustration of a test result of the learned CED neural network for crop heat extraction; FIG.

FIG. 5 illustrates a part of the learning data of the CED neural network for mother and child identification; and FIG. 6 illustrates an example of the test results of the learned CED neural network.

Explanation of symbols

100: CPU

110: GPU

200: Crop Weed Identification System

210: Graphic display of crop location

220: Crop Image - Crop Thermal Image Database

230: CED neural network for crop heat recognition

240: Produce symbol display image of crop-weed location

250: Crop Image - Crop Type Location Video Database

260: CED neural network for identification and localization of crops and weeds

280: camera

290: Mowers

Among the modes for carrying out the present invention, the accuracy of the CED neural network using the Unet of Fig. 1 (c) is higher than that of the CED neural network shown in Fig. 1 (a) and (b) The CED neural network using DensNet is superior, and the CED neural network using the dense net is somewhat better than the CED neural network using the Yu net. However, in the case of a CED neural network using a dense net, the required parameters are much larger than those of a CED neural network using a Yu net, and it is also difficult to select an appropriate hyper-parameter. In addition, the CED neural network using the dense net has a disadvantage that the execution speed is slower than the CED neural network using the U net.

Therefore, the best embodiment considering both accuracy, implementation, and economy is a form including a CED neural network using a uNet.

As shown in FIG. 1, the convolutional encoder-decoder (CED) neural network used in the present invention has a plurality of stages between an input terminal and an output terminal, and is a neural network having a small size. An increasingly smaller portion of the first half is referred to as an encoder portion, and an increasingly larger portion of the latter half is referred to as a decoder portion. Fig. 1 (A) and Fig. 1 (B) may be used to implement the CED neural network structure or the two structures used in the implementation of the method for detecting the crop heat and the crop weed in the present invention. Also, as shown in Fig. 1 (C), a CED neural network, i.e., a Unet, or a skipped structure, in which the outputs of the respective layers of the encoder portion are skipped immediately to the input of the same layer of the decoder portion, A CED neural network of the structure can also be used and a modified CED neural network structure called DensNet having connections across each layer of the Unet structure as shown in Figure 1 (a) can also be used. It is also possible to use a neural network that is modified based on the CED neural network. In this case, the present invention can be implemented by adopting different numbers of layers and different numbers of filters for each structure, and the object of the present invention can be achieved.

The present invention relates to a crop recognition method for recognizing the heat of crops using the Convolutional Encoder-Decoder (CED) neural network technology; And a crop-weed identification step that identifies the weeds from the crop using an additional CED neural network for a small number of weeds present on the crop line. The hardware for this is composed of two neural networks such as a CED neural network 230 used in the step of recognizing the rows of crops and a CED neural network 260 used in the step of identifying the weeds from the crop, do.

In each CED neural network, the task to be performed to the CED neural network is exemplified by applying the output data to be obtained as an output together with the input image to the learning database (220 and 250), and the neural network To install the technology to perform.

In order to do this, a large number of crop images are firstly obtained, and a large number of graphic display images 210 of the crop row positions and a symbol display image 240 of the crop-weed position are obtained for each crop image, And a type and position image database 250 of the crop image-crop and weed, and stores it in a computer hard disk. The crop image-crop column image database 220 and the crop image-crop and weed species types and location image database 250 are provided for each crop column recognition CED neural network 230 and for crop and weed identification and location recognition And is learned by the CED neural network 260. The learned neural network has the data type of each connection parameter values for the structure of the pre-designed neural network. The following is a detailed description of the learning process of the CED neural network 230 for crop heat recognition, the CED neural network 260 for identifying crops and weeds, and the location recognition.

Learning for crop heat recognition

In order to develop the technology of crop heat recognition on the CED neural network, the preparation of the learning data, which is an example, in various forms in the form of images. FIG. 3 illustrates a case of rice (rice) in a crop. When input images such as left images are applied, the CED neural network assigns a task to extract and present a crop image of the same crop as the right image will be. More specifically, the learning target image displays graphical lines like the right images of FIG. 3 along the positions corresponding to the respective simulation lines as shown in the left images of FIG. That is, when the right image is superimposed on the left image, a target image is generated in which each line of the right image is located at the center of the crop line in the left image. However, when lawn mowers weed along the row of crops, they work along the rows of crops in the front, so they focus on the lines on the front when creating the learning target image, and the rows toward the left and right sides may be omitted. In addition, as shown in FIG. 3 (d), by extending an extension line to the position where the nodule is lost in the middle, the neural network learns a human technique of drawing an extension line.

When the learning data is prepared, augmentation techniques such as rotation, enlargement, rotation, reduction, rotation, left / right up / down movement of various angles, After a much larger number of learning data is generated, the neural network is learned. The learning is repeated until the error falls below a predetermined threshold by using the backpropagation learning method, which is commonly used for neural network learning.

FIG. 4 is a graph illustrating a test result of the CED neural network learned for the above-mentioned crop line recognition. When the input images such as the leftmost position are applied to the CED neural network, Can be obtained. The images superimposed on the CED neural network outputs on the input images are the same as the rightmost images in Fig. 4 in order to check whether the lines on the output images correctly specify the crop row positions of the input images. As can be seen in the figure, it can be seen that the positions of the CED neural network output lines precisely specify the positions of the crop lines in the input image.

Identification learning of individual crops and weeds

In the present invention, a neural network having a structure similar to that of the Convolutional Encoder-Decoder (CED) is used for discrimination between crops and weeds. The learning data is displayed with one kind of symbol in the bottom part of the crop in the learning target image, and the image displayed in the base part of the weeds with different kinds of symbols is created and the neural network learns it. If a symbol is displayed on the bottom part, it is possible to make effective weeding by breaking the plant by a weeding machine.

The left images in FIG. 5 are the rice and blood images used as the neural network inputs, and the right target images are gray circular symbols corresponding to the bottom of the rice in the left images, These are examples of images displayed with circular symbols. The training data of the CED neural network to identify the mother and the daughter of rice are prepared as an input image by capturing a large amount of images such as the left image and creating learning target images such as the right image for each input image. Here, the learning target image is an example in which a blank image is displayed with gray and black circular symbols at the base position of the monopulse, but the symbols may be superimposed on the input image. Also, the color and shape of the symbol to be displayed can be selected in various ways. In this way, the learning data set is constructed and learned by using the backpropagation learning method until the error falls below a predetermined threshold value. Figure 6 illustrates some of the test results of the CED neural network for crop heat recognition. The leftmost images in FIG. 6 are input test images that are not included in the learning data. The intermediate images in FIG. 6 are neural network output images, and the right images are images in which an input image and an output image are superimposed. In the output image and the superimposed image, the gray circular symbol means the position of the rice, and the black circular symbol means the position of the blood. As can be seen from the result image, it can be seen that the position of the mother blood is accurately recognized and displayed. However, we can not discern rice that has overlapping rice paddies but does not show a base, because we have learned to discriminate based on the shape of the base. However, it can be seen that most of the blood of size and position that can clearly confirm the shape is identified and detected.

The resultant images thus recognized are sent to a weeding robot to be able to weed out the actual position corresponding to the mother and the blood.

The Role of the Herbicide System as a Brain

In order to utilize the present invention as a brain role of the weed system, an image is acquired in real time through a camera 280, and the obtained image is used to identify the CED neural network 230 for crop heat recognition, It needs to be simultaneously applied to the food CED neural network 260 and processed at a high speed. The CPU 100 is used to control the flow of information and configure the neural network in software. In addition, the CPU 100 post-processes the outputs of the neural networks, analyzes the results, and generates appropriate control signals necessary for autonomous traveling of the external lawn mowers using the results. During this process, particularly high speed signal processing is required to construct the CED neural networks and perform their operations. For this purpose, the GPU 110 is used as an auxiliary device.

Configuration of mower control system

The parameters of the learned CED neural network include the information necessary for crop recognition and crop-weed identification already extracted from the crop image-crop column image database 220 and the crop image-crop type and location image database 250 For the miniaturization of the apparatus, the databases are removed, and a camera 280, a CED neural network 230 for crop heat recognition, a weed identification and position recognition CED neural network 260, The lawn mower control system 300 can be configured simply with the CPU 100 and the GPU 110. [

In order to send an appropriate weeding operation command to the lawn mower 290, the lawn mower control system configured as described above first acquires an image through the camera 280 in real time, and transmits the obtained image to the CED neural network 230 for crops recognition, And to identify and locate the weed CED neural network 260 at the same time. The CPU 100 is used to control the flow of information and configure the neural network in software. In addition, the CPU 100 performs post-processing of the output of the neural network, analyzes and analyzes the output of the neural network, and generates an appropriate control signal necessary for autonomous weeding operation of the weeder 290 connected to the outside. During this process, a high-speed signal processing is required to construct a CED neural network and perform its operation. For this purpose, the GPU 110 is used as an auxiliary device.

It is indispensable to remove the weeds that hinder the growth of crops by taking away soil nutrients and solar energy in agriculture. Especially in the recent international trend of cultivation of environmentally friendly crops, we do not use chemicals such as pesticides, so we need more labor for weeding. Accordingly, the present invention aims at automating and mechanization of weeding work, which occupies most of the farmer's labor force in cultivation of crops, thereby reducing production costs such as the efficiency of cultivation of crops and the employment cost of the workers for weeding work. The cultivation of crops that require work is very likely to be industrially applicable.

Claims

Recognizing the rows of crops by neural network learning for weeding for rice and field crops so as to remove all of the plants that are out of those rows as weeds; Identification of the crops and weeds by neural network learning, characterized in that the columns and their remaining weeds are separated from the crops through another neural network learning
2. The method of claim 1, wherein the neural network used in the step of recognizing the crop and the crop and weed identification step comprises a plurality of convolution layers, and the encoder part and the decoder ) With a CED neural network having a structure in which a part of the CED neural network
The CED neural network of claim 2, wherein the CED neural network includes a CED neural network in which the outputs of each layer of the encoder portion are summed to the inputs of the same layer of the decoder portion, Identification of crops and weeds by neural network learning, characterized by the possibility of using CED neural network of structure
The CED neural network according to claim 3, wherein the CED neural network includes a CED neural network having a modified structure called DensNet having a basic structure of a CED neural network called a Unet structure, Identification of Crops and Weeds by Neural Network Learning
The method of claim 1, wherein, when a crop image is applied as an input to the CED neural network, the CED neural network is configured such that a position corresponding to a crop row in the crop image, The CED neural network is repeatedly learned so as to be drawn by the graphic line on the output end of the crop line image and the same method is applied to many other crop line images so that when an arbitrary test crop image is applied, A method for identifying crops and weeds by neural network learning, characterized in that the position is drawn by graphic lines and displayed
[2] The method according to claim 1, wherein when the crop is rice and weed is contaminated, a plurality of seeds are germinated by collecting a plurality of rice seeds in the case of a rice simulation, Identification of crops and weeds by neural network learning, characterized by distinguishing rice and blood distinguishing by visual differences in the density of the bottom part of aeration using the fact that the density of the bottom part of the aeration is low due to scattered germination Way
[Claim 6] The method of claim 6, wherein learning objective image data for distinguishing rice and blood using the visual difference in the density of the base is represented by symbols of different colors or shapes at positions corresponding to the base of the rice and blood of the input image A method for identifying crops and weeds by neural network learning, characterized in that a learning target image for identification of rice and blood which distinguishes rice and blood is created
[Claim 7] The method according to claim 5, wherein the database necessary for learning the neural network for crop heat recognition comprises a crop thermal image taken in a direction that the grass mower wishes to follow, as an input image; An image formed by a graphic line at a position corresponding to each crop line in the input image as a learning target image; A method for identifying a crop and a weed by learning a neural network, characterized by a method for providing a large number of sets of learning data composed of one set of the input image-learning target images
7. The method of claim 6, wherein the learning database required for the neural network for crop-weed identification is a bird's eye image, wherein the crops and weeds are included in the crop line and the close- The input image of the network; An image using different shape or color symbols is generated so as to distinguish them from each other in a position corresponding to each crop and weeds in each input image, and this is used as a learning target image; A method for identifying a crop and a weed by neural network learning characterized by a database for generating a crop-weed identification learning database in which a large number of learning data sets each including a set of the input image-learning target images as a set
10. The method according to claim 9, wherein the learning target image is generated from a learning data base required for a neural network for crop / weed identification, wherein a blank image or an input image is copied, A method of identifying a crop and a weed by neural network learning characterized by a method of creating a learning target image using a line, a figure, or symbols using a graph
10. The method of claim 9, wherein the position and size of each symbol are determined based on the input image when creating the learning target image in the learning database required for the neural network for crop-weed identification; A neural network learning method characterized by the position and size of a symbol center point and the method of displaying a symbol position and size in a learning target image to be displayed on a learning target image so as to include as many features having high distinguishability among objects as possible in a symbol area How to identify crops and weeds by
A crop image-crop thermal image database (220) made up of a graphic display image of a crop row position produced by the method of claim 5 or 8; A CED neural network (230) for crop heat recognition to be learned by the method according to any one of claims 1 to 4 to the image database (220); A crop image-crop type location image database (250) made up of a symbol display image of the crop-weed species and location produced by the method of any one of claims 6, 7, and 11 to 11; A food CED neural network 260 for identifying the type and location of crops and weeds to be learned by the method of any one of claims 1 to 4 to the image database 250; A camera 280 for capturing a crop-weed image in real time and applying it to the input of the CED neural network 230, 260; The CED neural network is constructed in a software manner and analyzed, and the result of the CED neural network is analyzed to generate an appropriate control signal necessary for an autonomous weeding operation of an externally connected lawn mower using the result of the analysis, A CPU 100; And a GPU (110) for assisting the CPU and performing the CED neural network at a high speed, characterized in that the apparatus for identifying crops and weeds by neural network learning
The parameters of the learned CED neural networks according to claim 12 include already a crop image-crop column image database (220); Crop information and crop-weed identification information extracted from the crop image-crop type and location image database 250. Therefore, in order to miniaturize the apparatus, the databases are removed, and the camera 280 and the crop- A CED neural network 230; A CED neural network 260 for identifying and locating weeds; A CPU 100; And the control system of the lawn mower is constructed simply by the GPU (110).