KR20200052397A - Plant Area Extraction System and Method Based on Deep Running and Connectivity Graphs - Google Patents

Abstract

Provided is a plant area extraction system. The plant area extraction system comprises: a plant image generation unit photographing a target plant and generating a plant image; a first plant area extraction unit generating first plant area extraction data which separates a plant area and a background area for each pixel of the plant image through a machine learning model based on a convolutional neural network; and a second plant area extraction unit utilizing the plant image and the first plant area extraction data and generating second plant area extraction data which maximizes interconnectivity between a plurality of pixels of the plant image.

Description

Plant Area Extraction System and Method Based on Deep Running and Connectivity Graphs

The present invention relates to a system and method for extracting plant regions, and specifically to a system and method for extracting plant regions based on deep learning and connectivity graphs.

[Explanation on national support R & D]

This research is under the supervision of the Rural Development Administration to support the "Development of Technology for Estimating the Expression of Image-Based Core Crop Phenotypes" of the Next Generation Bio Green 21 Project (Task No .: 1395056227, Detailed Project No .: PJ012281022018, Task Performing Organization: Korea Institute of Science and Technology) It is made by.

Phenonomics is a study of the function of cells based on the investigation of the phenotype of living organisms such as RNA and protein and measuring the amount. In order to measure and document phenotypic information, individual analysis of objects is required, and when the object of expression morphology is a plant, individual analysis of plant leaves is required.

For the individual analysis of such plant leaves, a labor-intensive method of manually identifying individual plant leaves has been performed in the past, but recently, a method of deriving the characteristics of individual plant leaves from the acquired image by studying the entire plant image has been studied. .

In the method of deriving the characteristics of individual plant leaves from the plant image, a process of distinguishing the plant region and the background region in the plant image may be important. Since the analysis of the characteristics of individual plant leaves is performed based on the divided plant image, if the classification and extraction of the plant area is not performed correctly, the accuracy of the results that are performed afterwards is not guaranteed. Particularly, in the case of monocotyledonous plants, since the leaves are long and thin, it is not easy to distinguish the plant region from the background region. In the divided plant region, there is a case where a hole that does not exist in the actual plant or the leaf is cut is separated. Can occur. In other words, in the case of separating a plant region from a background region in an elongated leafy plant image, a separation technique capable of ensuring maximum connectivity of the plant leaves is required.

The present invention has been devised to solve the above-mentioned problems, and provides a system and method for extracting plant regions based on deep learning and connectivity graphs that can classify plant regions with elongated leaves in a plant image as background regions.

A plant region extraction system according to an embodiment of the present specification includes a plant image generator configured to generate a plant image by photographing a target plant; A first plant region extraction unit generating first plant region extraction data that separates a plant region and a background region for each pixel of the plant image through a machine learning model based on a convolutional neural network; And a second plant region extraction unit that generates second plant region extraction data that maximizes interconnection between a plurality of pixels of the plant image by utilizing the plant image and the first plant region extraction data.

In one embodiment, the machine learning model based on the convolutional neural network of the first plant region extraction unit includes a convolution step (Conv) for generating a feature map for the plant image, and the resolution of the feature map It includes a transition down (TD) step to reduce and a transition up (TU) step to increase the resolution of the feature map, the data input to the transition down can be skipped (skip connetion) with the data output from the transition up. have.

In one embodiment, the machine learning model based on the convolutional neural network of the first plant region extraction unit includes a density block including a plurality of convolution steps, and within the density block The feature map generated through convolution may be accumulated so that the initial stage information is not erased and continues with the later stage information.

In one embodiment, when the plant image is larger than the size of the image that can be processed by the first plant region extracting unit, the first plant region extracting unit divides the plant image into the size of the processable image, thereby subtracting a plurality of sub plants. An image is generated, first sub plant region extraction data for each of the plurality of sub plant images are respectively generated, and the first first plant region extraction data is generated by combining each of the generated first sub plant region extraction data. can do.

In one embodiment, when the first plant region extraction data of each of the first sub-plant region extraction data includes overlapping pixel regions of the plurality of sub-plant images, the first plant region extraction unit is highest for the overlapping pixel regions. The first plant region extraction data may be combined by selecting the first sub-plant region extraction data having a probability value.

In one embodiment, the second plant region extracting unit forms a graph of a probability value of each pixel corresponding to the first plant region extraction data and a graph of interconnection between the plurality of pixels and between connected pixels. In consideration of the mutual influence, the second plant region extraction data for determining whether each pixel is a plant region may be generated.

In one embodiment, the mutual influence may be determined in consideration of a difference in distance between connected pixels and similarity of R, G, and B values of the plant image.

In one embodiment, the plant may be monocotyledonous plants.

Plant region extraction method according to another embodiment of the present invention, the plant region extraction system, photographing a target plant to generate a plant image; Generating, by the plant region extraction system, first plant region extraction data that separates a plant region and a background region for each pixel of the plant image through a machine learning model based on a convolutional neural network; And the plant region extracting system, using the plant image and the first plant region extracting data, generating second plant region extracting data that maximizes interconnection between a plurality of pixels of the plant image.

In one embodiment, the convolutional neural network-based machine learning model includes a convolution step (Conv) for generating a feature map for the plant image, and a transition down (TD) for reducing the resolution of the feature map. ) And a transition up (TU) step of increasing the resolution of the feature map, and data input to the transition down may be skipped concatenated with data output from the transition up.

In an embodiment, the machine learning model based on the convolutional neural network includes a density block including a plurality of convolution steps, and features generated through convolution within the density block The map may be accumulated so that the initial stage information is not erased, but continues with the later stage information.

In one embodiment, the step of generating the first plant region extraction data includes: dividing the plant image into a size of the processable image to generate a plurality of sub-plant images; Generating first sub plant region extraction data for each of the plurality of sub plant images; And combining the generated first sub-plant region extraction data to generate the first plant region extraction data.

In one embodiment, the step of generating the first plant region extraction data by combining each of the generated first sub-plant region extraction data may include generating the first sub-plant region extraction data of the plurality of sub-regions. When the overlapping pixel region of the plant image is included, the first plant region extraction data may be combined by selecting the first sub-plant region extraction data having the highest probability value for the overlapping pixel region.

In one embodiment, the step of generating the second plant region extraction data forms both a probability value that each pixel according to the first plant region extraction data corresponds to the plant region and an interconnection graph between the plurality of pixels. And, considering the mutual influence between the connected pixels, it is possible to generate the second plant region extraction data for determining whether each pixel is a plant region.

In one embodiment, the mutual influence may be determined in consideration of a difference in distance between connected pixels and similarity of R, G, and B values of the plant image.

In one embodiment, the plant may be monocotyledonous plants.

A computer program according to another embodiment of the present invention is stored in a medium in combination with hardware to execute the above-described plant region extraction method.

The plant region extraction system according to an embodiment of the present invention distinguishes a plant region and a background region from a plant image through a convolutional neural network, but the result data is input through a deconvolution process. It is guaranteed to have the same resolution as the image.

In addition, through a conditional probability model called a density conditional random field, it is possible to generate result data that maximizes the connectivity of pixels in a plant image.

Therefore, a plant region with elongated leaves can be more precisely extracted from the plant image.

1 is a schematic block diagram of a plant region extraction system according to an embodiment of the present invention.
2A and 2B show exemplary plant images of plants to be photographed in the plant image generation unit.
3A is a block diagram showing an operation process of the first plant region extraction unit, and FIG. 3B is a block diagram showing the configuration of the dense block of FIG. 3A.
4 is a graph showing a connection relationship between a plurality of pixels.
5A and 5B are result data obtained by extracting a plant region from the plant images of FIGS. 2A and 2B, respectively.
6 is a flowchart of a method for extracting plant regions according to another embodiment of the present invention.

For a detailed description of the present invention, which will be described later, reference is made to the accompanying drawings that illustrate, by way of example, specific embodiments in which the invention may be practiced. These embodiments are described in detail enough to enable a person skilled in the art to practice the present invention. Various embodiments of the present invention are different, but need not be mutually exclusive. For example, the specific shapes, structures, and properties described herein can be implemented in other embodiments without departing from the spirit and scope of the invention in connection with one embodiment. In addition, the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the invention. Therefore, the detailed description to be described later is not described in a limiting sense, and the scope of the present invention is limited only by the appended claims, along with all ranges equivalent to the claims. In the drawings, similar reference numerals refer to the same or similar functions in various aspects.

The terminology used in the present specification is a general terminology that is currently widely used while considering functions, but this may vary according to intentions or customs of technicians in the field or the appearance of new technologies. In addition, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, the meaning will be described in the description of the corresponding specification. Therefore, the terms used in this specification should be interpreted based on not only the names of simple terms, but the actual meanings of the terms and contents throughout the present specification.

1 is a schematic block diagram of a plant area extraction system according to an embodiment. 2A and 2B show exemplary plant images of plants to be photographed by the plant image generator 100. 3A is a block diagram showing an operation process of the first plant region extraction unit, and FIG. 3B is a block diagram showing the configuration of the dense block of FIG. 3A. 4 is a graph showing a connection relationship between a plurality of pixels. 5A and 5B are result data obtained by extracting plant regions from the plant images of FIGS. 2A and 2B, respectively.

Referring to FIG. 1, the plant region extraction system 10 according to the present embodiment includes a plant image generation unit 100, a first plant region extraction unit 110, and a second plant region extraction unit 120. The plant area extraction system according to embodiments and each device or unit constituting the same may have aspects that are entirely hardware, or partially hardware and partially software. For example, each component of the plant area extraction system is intended to refer to a combination of hardware and software driven by the hardware. The hardware may be a data processing device including a central processing unit (CPU) or other processor. Also, software driven by hardware may refer to a running process, an object, an executable, a thread of execution, a program, or the like. For example, the plant image generating unit 100 may refer to a combination of hardware for imaging an object and imaging it by controlling it, and converting the image into a form for subsequent processing by performing imaging and processing imaging.

The plant image generation unit 100 may photograph a target plant and generate a plant image I corresponding to a two-dimensional image. The plant image generator 100 may include an image sensor for photographing a plant image. The plant image generation unit 100 may generate a plant image I by photographing a target plant from the front. However, the present invention is not limited thereto, and in another embodiment, the plant image generating unit 100 may perform shooting in at least one of front, rear, left, right, and top surfaces of the target plant.

Here, the target plant photographed by the plant image generating unit 100 is a plant having a plurality of plant leaves, and each plant leaf may have a long and thin shape. The target plant may be monocotyledonous plants, but is not limited thereto.

Further, the space for photographing the target plant may be a space having a single color background. Specifically, it may be a space having a background of a color that contrasts with a plant leaf or is easily distinguishable from a color, forming a complementary color relationship, but is not limited thereto. The target plant may be photographed while sequentially passing through the photographing place through a conveyor system or the like, but is not limited thereto. 2A and 2B show exemplary plant images generated by the plant image generator 100. The plant image I may be composed of a plurality of pixels, and may have a resolution according to the number of pixels. The generated plant image I may be provided to the first plant region extractor 110 and the second plant region extractor 110.

The first plant region extracting unit 110 generates first plant region extraction data for distinguishing the plant region and the background region for each pixel of the plant image. The first plant region extraction data is data that stochastically divides the plant region and the background region in units of each pixel of the plant image I, and the probability that each of the plurality of pixels included in the plant image I corresponds to the plant region It may be calculated data. The first plant region extracting unit 110 may determine whether each of the plurality of pixels of the input plant image I corresponds to the plant region, and generate a probability value corresponding to the plant region as a result value.

The first plant region extraction unit 110 may generate first plant region extraction data through a machine learning model based on a convolutional neural network. The machine learning model based on the convolutional neural network is composed of a plurality of convolutional stages and a plurality of polling stages, and the value of the filter that can reflect the characteristics of the input image in the process of repeating each stage It can learn automatically and performs analysis and discrimination on images according to the learned filters. However, in the case of a conventional machine learning model based on a convolutional neural network, the resolution of the input image is reduced in the polling step of the input image. As a result, a result of a plurality of pixels of each input image based on the original resolution May not be suitable for the present invention.

Accordingly, the first plant region extracting unit 110 according to an embodiment of the present invention corresponds to a polling step and a transition down (TD) step of reducing the resolution of the feature map and a transition up (TU) of increasing the resolution of the feature map ) Comprises an encoder-decoder structure including a step. That is, the first plant region extraction data finally generated through the deconvolution process may be guaranteed to have the same resolution as the input plant image (I).

3A shows an overall operation process of a machine learning model based on a convolutional neural network performed by the first plant region extraction unit 110. The first plant region extraction unit 110 may be configured such that Conv, DB, TD, DB, TD, DB, TU, DB, TU, DB, Conv steps are sequentially performed as illustrated in FIG. It does not work.

In the block shown in FIG. 3A, Image I means plant image (I), and Lavel L is first plant region extraction data. In the block shown in Figure 3a, Conv represents the convolution step and creates a feature map for the plant image. The input plant image I through the convolution step may be formed as a plurality of feature maps according to the number of filters. The feature map generated through the convolution step may have the same size and resolution as the image input by zero padding.

TD represents a transition down step, and the feature map after TD may have a reduced resolution. The TU represents a transition up step, and the feature map after the TU can be increased in resolution at the same rate as the reduced rate in TD. Also, data input to the transition down may be skipped concatenated with data output from the transition up. That is, the data output from the transition up may be supplemented with data provided from the transition down. The loss of the original data generated in the transition down process may be compensated through a skip connetion to determine the pixels of the original image and generate result data.

DB stands for density block. The Dense Block is composed of a plurality of convolution steps, and feature maps generated through convolution in a density block are continuously accumulated, and the size of the feature maps in one density block is the same. These continuously accumulated points are indicated by small circular blocks (Concatenation) in FIGS. 3A and 3B.

3B is a block diagram showing the structure of a Dense Block according to an embodiment of the present invention. In the embodiment of the present invention, the Dense Block comprises two stages of convolution, but is not limited thereto. As the feature maps generated through convolution continue to accumulate and accumulate, the initial stage information may not be washed out, but may be continued with the later stage information, and the connection between each stage increases, so that the learning of the filter can be performed more efficiently. Can be.

According to the convolutional neural network structure illustrated in FIG. 3A, the first plant region extraction unit 110 may generate first plant region extraction data having the same resolution and size as the input image plant image (I). In addition, since it is determined whether it is a plant region or a background region for each pixel constituting the plant image I, even if the leaves of the plant are fine, it is possible to determine this. In addition, it may be possible to classify regions more accurately to perform judgment on each pixel through a learned filter that extracts characteristics of an image, such as comparison with surrounding pixels, not merely color differences.

Here, when the size (resolution) of the plant image I is larger than the maximum size (resolution) of the image that can be processed by the first plant region extractor 110, a pre-processing process for the plant image I may be required. have. However, if the resolution of the plant image I is downscaled to a processable image, pixel loss may occur. As pixel loss occurs in the fine leaf portion of the plant image I, unintended breaks or holes may occur when extracting the plant region. Therefore, the first plant region extracting unit 110 according to the present embodiment divides the plant image I into a size of a processable image to generate a plurality of sub-plant images. The plurality of sub-plant images may be the same size as the image that can be processed by the first plant region extraction unit 110, and the first plant region extraction unit 110 may extract the first sub-plant region extraction data for each sub-plant image. The first plant region extraction data for the original plant image (I) can be generated by generating each and combining them again.

However, the plurality of sub-plant images may include pixel regions overlapping with other sub-plant images. Since the value of the first sub-plant region data for the overlapping pixel region is a result of machine learning for another type of image, the resulting probability value may be different. The first plant region extracting unit 110 has the highest probability value for the overlapped pixel regions when the generated first sub-plant region extraction data includes overlapping pixel regions of the plurality of sub-plant images. The first plant region extraction data may be selected to combine the first plant region extraction data.

The first plant region extraction data generated by the first plant region extraction unit 110 is provided to the second plant region extraction unit 120.

The second plant region extraction unit 120 receives the plant image (I) from the plant image generation unit 100 and the first plant region extraction data from the first plant region extraction unit 110, respectively, and the plant image and the first Plant region extraction data is used to generate second plant region extraction data that maximizes interconnection between a plurality of pixels of a plant image. The second plant region extraction unit 120 may maximize and maximize the connectivity of the pixels of the plant image I through probabilistic graphical models. The probabilistic graphic model may be a conditional probability model called a density conditional random field (Dense Conditional Random Field) forming all of the interconnection relationship between a plurality of pixels. Here, maximizing or maximizing the connectivity means, as illustrated in FIG. 4, forming an interconnection graph between a plurality of pixels and determining whether each pixel is a plant region in consideration of the mutual influence between the connected pixels. do.

More specifically, the second plant region extraction unit 120 considers the probability value that each pixel based on the first plant region extraction data corresponds to the plant region and other mutual influence on the connection between the pixels, so that each pixel is connected to the plant region. The second plant region extraction data that determines whether or not it is applicable is generated.

Here, the mutual influence considered between the connected pixels may be determined in consideration of the distance between the connected pixels, the position difference, whether the R, G, and B values of the plant image I are similar, and whether the plant image color is similar. In consideration of the relationship between one pixel and another pixel connected to the one pixel, the probability value of whether each pixel corresponds to a plant region is expressed by a Gibbs energy function defined by Equation 1 below, and the result of this The value can be calculated by optimizing equation (1).

[Equation 1]

는 식물 이미지,

는 모델 파라미터(식물 영역인지 여부를 판별),

는 식물 이미지의 제i 화소,

는 식물 이미지의 제j 화소를 의미한다.here,

Plant images,

Is the model parameter (determining whether it is a plant area),

Is the i-th pixel of the plant image,

Means j pixel of the plant image.

)은 로지스틱 회귀분석 모델(logistic regression model)로 정의되며, 제1 식물 영역 추출 데이터에서 생성된 각 화소에 대한 확률 값(

)이 하기 수학식 2와 같이 입력될 수 있다.In the Gibbs energy function of Equation 1, the unary term (

) Is defined as a logistic regression model, and a probability value for each pixel generated from the first plant region extraction data (

) May be input as shown in Equation 2 below.

[Equation 2]

: 제1 식물 영역 추출 데이터에서 생성된 제i 화소의 결과 데이터

: Result data of the i-th pixel generated from the first plant region extraction data

)은 연결된 상이한 화소(

,

)의 영향성을 나타내는 수식으로, 가우시안 커널(Gaussian kernel)과 바이라테랄 커널(bilateral kernel)을 혼합하여 정의될 수 있다. Also, in the Gibbs energy function of Equation 1, pairwise interaction term (

) Is a different connected pixel (

,

) Is a formula representing the influence of the Gaussian kernel (Gaussian kernel) and may be defined by mixing a bilateral kernel (bilateral kernel).

))은 하기 수학식 3과 같이 정의될 수 있다.Gaussian kernel,

)) May be defined as in Equation 3 below.

[Equation 3]

는 고정된 특성 벡터로 하기 수학식 4와 같이 정의되고,

도 동일한 방식으로 정의될 수 있다.here,

Is defined as Equation 4 below as a fixed characteristic vector,

Can also be defined in the same way.

[Equation 4]

및

는

의 x, y 위치에 대한 Potts compatibility function으로

을 만족하며,

는 x위치에 대한 표준편차에 해당하는 파라미터 값,

는 y위치에 대한 표준편차에 해당하는 파라미터 값에 해당한다.)(here,

And

The

Potts compatibility function for x and y position of

Satisfying,

Is the parameter value corresponding to the standard deviation for the x position,

Is the parameter value corresponding to the standard deviation for y position.)

))을 통해 위치에 따른 영향성이 고려될 수 있다. In other words, the Gaussian kernel (Gaussian kernel,

)) Can be taken into account.

)의 바이라테랄 커널(bilateral kernel,

)은 하기 수학식 5와 같이 정의된다. pairwise interaction term (

) Of the bilateral kernel (bilateral kernel,

) Is defined as in Equation 5 below.

[Equation 5]

는 고정된 특성 벡터로 하기 수학식 6와 같이 정의되고,

도 동일한 방식으로 정의될 수 있다.here,

Is a fixed feature vector defined by Equation 6 below,

Can also be defined in the same way.

[Equation 6]

는 식물 이미지(I)의 제i 화소(

)의 R, G, B 값이며,

는 x위치에 대한 표준편차에 해당하는 파라미터 값,

는 y위치에 대한 표준편차에 해당하는 파라미터 값,

,

및

은 각각 R, G, B 값에 대한 표준 편차에 해당하는 파라미터 값이다.)(here,

Is the i-th pixel of the plant image (I)

), R, G, B values,

Is the parameter value corresponding to the standard deviation for the x position,

Is the parameter value corresponding to the standard deviation for y position,

,

And

Are parameter values corresponding to standard deviations for R, G, and B values, respectively.)

A color difference between pixels and a color difference considering distances between pixels may be considered through a bilateral kernel.

By combining Equation 3 and Equation 5, the pairwise interaction term in the Gibbs energy function of Equation 1 may be defined as Equation 7 below.

[Equation 7]

)를 손실 함수(loss function)로, L-BFGS(limited-memory Broyden-Fletcher-Goldfarb-Shanno)을 최적화 알고리즘으로 사용하여 수학식 1의 Gibbs energy function을 하기 수학식 8과 같이 최적화하여 각 화소에 대한 결과 값을 산출하여 제2 식물 영역 추출 데이터를 생성한다. The second plant region extracting unit 120 has a log likelihood loss (

) As the loss function and L-BFGS (limited-memory Broyden-Fletcher-Goldfarb-Shanno) as the optimization algorithm. The result value is calculated for the second plant region extraction data.

[Equation 8]

,

,

은 화소(

)의 결과 값, 제2 식물 영역 추출 데이터이다.)(here,

Silver pixel (

), The second plant area extraction data.)

 The second plant region extraction data is data indicating whether each pixel of the plant image I is a plant region. 5A is second plant region extraction data as a result of extracting a plant region from the plant image of FIG. 2A, and FIG. 5B is second plant region extraction data as a result of extracting a plant region from the plant image of FIG. 2B. The second plant region extraction data is a binary image in which the plant region is displayed in white and the background region is displayed in black.

The plant region extraction system according to an embodiment of the present invention distinguishes a plant region and a background region from a plant image through a convolutional neural network, but the result data is input through a deconvolution process. It is guaranteed to have the same resolution as the image.

In addition, through a conditional probability model called a density conditional random field, it is possible to generate result data that maximizes the connectivity of pixels in a plant image.

Therefore, a plant region with elongated leaves can be more precisely extracted from the plant image.

Hereinafter, a method for extracting a plant region according to an embodiment of the present invention will be described. 6 is a flowchart of a method for extracting plant regions according to an embodiment of the present invention.

Referring to Figure 6, the plant leaf identification method according to an embodiment of the present invention includes a plant image generation step (S100), the first plant area data extraction step (S110), the second plant area data extraction step (S120) do.

Here, the plant region extraction system for performing each step described above may be the plant region extraction system 10 of FIG. 1 described above, and detailed description thereof will be omitted. Also, FIGS. 1 to 5 may be referred to for the description of this embodiment.

First, a plant image is generated (S100).

The plant region extraction system 10 includes a plant image generation unit 100, a first plant region extraction unit 110, and a second plant region extraction unit 120. The step of generating a plant image (I) by photographing the target plant may be performed in the plant image generator 100.

The plant image generation unit 100 may photograph a target plant and generate a plant image I corresponding to a two-dimensional image. The plant image generator 100 may include an image sensor for photographing a plant image. Here, the target plant photographed by the plant image generating unit 100 is a plant having a plurality of plant leaves, and each plant leaf may have a long and thin shape. The target plant may be monocotyledonous plants, but is not limited thereto. The plant image I may be composed of a plurality of pixels, and may have a resolution according to the number of pixels. The generated plant image I may be provided to the first plant region extractor 110 and the second plant region extractor 110.

Subsequently, the first plant region extraction data is extracted (S110).

The extraction of the first plant region data may be performed by the plant region extraction unit 110. The first plant region extracting unit 110 generates first plant region extraction data for distinguishing the plant region and the background region for each pixel of the plant image. The first plant region extraction data is data that stochastically divides the plant region and the background region in units of each pixel of the plant image I, and the probability that each of the plurality of pixels included in the plant image I corresponds to the plant region It may be calculated data. The first plant region extracting unit 110 may determine whether each of the plurality of pixels of the input plant image I corresponds to the plant region, and generate a probability value corresponding to the plant region as a result value.

The first plant region extraction unit 110 may generate first plant region extraction data through a machine learning model based on a convolutional neural network. The convolutional neural network-based machine learning model includes a convolution step (Conv) for generating a feature map for the plant image, a transition down (TD) step for reducing the resolution of the feature map, and a feature map It includes a transition up (TU) step of increasing the resolution, the data input to the transition down may be skipped (skip connetion) with the data output from the transition up.

In addition, the convolutional neural network-based machine learning model includes a density block including a plurality of convolution steps, and feature maps generated through convolution within the density block are cumulative. As the initial stage information is not erased, it can be continued with the later stage information.

The first plant region extraction unit 110 may generate first plant region extraction data having the same resolution and size as the input image plant image (I). In addition, since it is determined whether it is a plant region or a background region for each pixel constituting the plant image I, even if the leaves of the plant are fine, it is possible to determine this. In addition, it may be possible to classify regions more accurately to perform judgment on each pixel through a learned filter that extracts characteristics of an image, such as comparison with surrounding pixels, not merely color differences.

Here, the step of generating the first plant region extraction data includes: dividing the plant image into a size of the processable image to generate a plurality of sub-plant images; Generating first sub plant region extraction data for each of the plurality of sub plant images; And combining the generated first sub plant region extraction data to generate the first plant region extraction data. The step of generating the first plant region extraction data by combining the generated first sub plant region extraction data may include generating pixels in which the generated first sub plant region extraction data overlaps the plurality of sub plant images. When the region is included, the first plant region extraction data may be combined by selecting the first sub-plant region extraction data having the highest probability value for the overlapping pixel region.

Subsequently, second plant region extraction data is generated (S120).

Generation of the second plant region extraction data is performed by the second plant region extraction unit 120. The second plant region extraction unit 120 may maximize and maximize the connectivity of the pixels of the plant image I through probabilistic graphical models. The probabilistic graphic model may be a conditional probability model called a density conditional random field (Dense Conditional Random Field) forming all of the interconnection relationship between a plurality of pixels. Here, maximizing or maximizing the connectivity means forming an interconnect graph between a plurality of pixels and determining whether each pixel is a plant region in consideration of the mutual influence between the connected pixels.

In the generating of the second plant region extraction data, all of the pixels according to the first plant region extraction data form a probability value corresponding to the plant region and an interconnection graph between the plurality of pixels, and between the connected pixels. In consideration of the mutual influence, the second plant region extraction data for determining whether each pixel is a plant region may be generated. The mutual influence may be determined in consideration of a distance difference between connected pixels and similarity of R, G, and B values of the plant image. The mutual influence may be determined by considering the distance difference between the connected pixels and whether the R, G, and B values of the plant image are similar. Taking into account the relationship between one pixel and another pixel connected to the one pixel, the probability value of whether each pixel corresponds to a plant region is expressed as a Gibbs energy function, and optimization of the second plant region extraction data is calculated. Can be.

The operation by the method for extracting plant regions according to the above-described embodiments may be at least partially implemented by a computer program and recorded in a computer-readable recording medium. The program for implementing the operation by the plant region extraction method according to the embodiments is recorded, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer are stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. In addition, the computer readable recording medium may be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment will be readily understood by those skilled in the art to which this embodiment belongs.

Although described above with reference to embodiments, the present invention should not be construed as being limited by these embodiments or the drawings, and those skilled in the art will think and scope of the present invention described in the following claims It will be understood that various modifications and changes can be made to the present invention without departing from the scope.

10: Plant area extraction system
100: plant image generating unit
110: first plant region extraction unit
120: second plant area extraction unit

Claims (17)

A plant image generating unit that photographs a target plant to generate a plant image;
A first plant region extraction unit for generating first plant region extraction data that separates a plant region and a background region for each pixel of the plant image through a machine learning model based on a convolutional neural network; And
A plant area extraction system comprising a second plant area extraction unit that generates second plant area extraction data that maximizes interconnection between a plurality of pixels of the plant image by utilizing the plant image and the first plant area extraction data. According to claim 1,
The convolutional neural network-based machine learning model of the first plant region extractor,
A convolution step (Conv) for generating a feature map for the plant image, a transition down (TD) step for reducing the resolution of the feature map, and a transition up (TU) step for increasing the resolution of the feature map,
The plant region extraction system, characterized in that the data input to the transition down is skipped (skip connetion) with the data output from the transition up. According to claim 2,
The convolutional neural network-based machine learning model of the first plant region extractor includes a density block including a plurality of convolution steps,
The feature map generated through convolution within the density block is accumulated so that the initial stage information is not erased, but the plant region extraction system is characterized in that it continues to be connected to the late stage information. According to claim 1,
When the plant image is larger than the size of the image that can be processed by the first plant region extracting unit, the first plant region extracting unit divides the plant image into the size of the processable image to generate a plurality of sub-plant images,
First sub plant region extraction data for each of the plurality of sub plant images are respectively generated,
A plant area extraction system, characterized in that the first plant area extraction data is generated by combining the generated first sub-plant area extraction data. According to claim 4,
When the first plant region extraction data of each of the first plant region extraction data includes overlapping pixel regions of the plurality of sub plant images,
And selecting the first sub-plant region extraction data having the highest probability value for the overlapping pixel regions and combining the first plant region extraction data. According to claim 1,
The second plant region extracting unit forms a probability value for each pixel corresponding to the first plant region extraction data and a graph of interconnection between the plurality of pixels, and considers the mutual influence between connected pixels And generating the second plant region extraction data for determining whether each pixel is a plant region. The method of claim 6,
The mutual influence is determined by considering the distance difference between the connected pixels and whether the R, G, and B values of the plant image are similar. According to claim 1,
The plant is a monocotyledonous plant (Monocotyledonous Plants), characterized in that the plant area extraction system. A plant region extraction system, photographing a target plant to generate a plant image;
Generating, by the plant region extraction system, first plant region extraction data that separates a plant region and a background region for each pixel of the plant image through a machine learning model based on a convolutional neural network; And
The plant region extraction system, using the plant image and the first plant region extraction data, generating a second plant region extraction data that maximizes the interconnection between a plurality of pixels of the plant image Extraction method. The method of claim 9,
The convolutional neural network-based machine learning model,
A convolution step (Conv) for generating a feature map for the plant image, a transition down (TD) step for reducing the resolution of the feature map, and a transition up (TU) step for increasing the resolution of the feature map,
The method of extracting a plant region, characterized in that data input to the transition down is skipped concatenated with data output from the transition up. The method of claim 10,
The convolutional neural network-based machine learning model includes a density block including a plurality of convolution steps,
The feature map generated through convolution in the density block is accumulated, and the initial stage information is not erased, but the plant region extraction method is characterized in that it is continued with the later stage information. The method of claim 9,
The step of generating the first plant region extraction data,
Dividing the plant image into a size of the processable image to generate a plurality of sub-plant images;
Generating first sub plant region extraction data for each of the plurality of sub plant images; And
And generating the first plant region extraction data by combining the generated first sub plant region extraction data. The method of claim 12,
The step of generating the first plant region extraction data by combining the generated first sub plant region extraction data may include overlapping the generated first sub plant region extraction data of the plurality of sub plant images. When a pixel region is included, a plant region extraction method characterized by selecting first sub-plant region extraction data having the highest probability value for the overlapping pixel region and combining the first plant region extraction data. The method of claim 9,
In the generating of the second plant region extraction data, all of the pixels according to the first plant region extraction data form a probability value corresponding to the plant region and an interconnection graph between the plurality of pixels, and between the connected pixels. A method of extracting plant regions, characterized in that the second plant region extraction data for determining whether each pixel is a plant region in consideration of mutual influence is generated. The method of claim 14,
The mutual influence is determined by considering the distance difference between connected pixels and whether the R, G, and B values of the plant image are similar. The method of claim 8,
The plant is a monocotyledonous plant (Monocotyledonous Plants), characterized in that the plant region extraction method. A computer program stored in a medium to execute the plant area extraction method according to any one of claims 9 to 16 in combination with hardware.

Images