KR102245337B1 - Learning method for classifying crop using weight parameter based on confusion matrix, method for classifying crop using the same, computer-readable recording medium for excuting the method, and crop classifying device - Google Patents

Abstract

Provided are a crop classification learning method to which a weight is applied based on an error matrix, a crop classification method using the same, a computer-readable recording medium recording a program for executing the method, and a crop classification apparatus. The crop classification apparatus includes an image input unit; an individual discrimination unit; a weight determining unit; and a final determination unit.

Description

A crop classification learning method applying weights based on an error matrix, a crop classification method using the same, a computer-readable recording medium recording a program to execute the method, and a crop classification device {LEARNING METHOD FOR CLASSIFYING CROP USING WEIGHT PARAMETER BASED ON CONFUSION MATRIX, METHOD FOR CLASSIFYING CROP USING THE SAME, COMPUTER-READABLE RECORDING MEDIUM FOR EXCUTING THE METHOD, AND CROP CLASSIFYING DEVICE}

The present invention relates to a crop classification learning method applying weights based on an error matrix, a crop classification method using the same, a computer-readable recording medium recording a program for executing the method, and a crop classification apparatus. In detail, the present invention provides a method for classifying crops to which a weight is applied based on an error matrix, a method for classifying crops based on colors at various positions of the crop through multi-image recognition using the learning method, and a method for classifying the crops. It relates to a computer-readable recording medium in which a program for execution is recorded, and a device for sorting crops.

The sugar content or nutritional value of a crop is directly related to the quality level of the crop, and consumers sometimes select excellent crops by visually classifying the condition of the crop in order to select a crop that has excellent taste, flavor, and texture. In addition, sellers want to sell crops with higher quality and higher demand from consumers at higher prices, even if they are grown together. Therefore, the maturity classification of crops is a necessary step before consumers and sellers trade with each other.

In particular, in the case of ripened fruits such as tomatoes, kiwis, bananas, etc., it is very important to selectively classify crops having appropriate maturity at a specific time because the degree of change in maturity from seller to consumer through distribution process is large. For example, for crops with a relatively long distribution process, it may be desirable to start distribution with a relatively low maturity, and for crops with a relatively short distribution process, it may be desirable to start distribution with a relatively high maturity. have.

In general, one factor that can be considered in the maturity classification is the color or hue seen from the appearance of the crop. As the crop matures, the color of the bark changes.Depending on the type of crop, there are crops that change color from the top of the crop, while there are crops that change color from the belly of the crop, and the crop's stripe line is darkened. There are crops for which maturity can be judged.

For example, in the case of tomatoes, antioxidants such as ascorbate, lycopene, beta-carotene, rutin and caffeic acid increase as maturation progresses, and malic acid Organic acids such as (malic acid) and citric acid are reduced, and glucose and fructose are accumulated, resulting in stronger sweetness and higher maturity.

Along with this, the hardness of the pulp decreases due to the decrease in pectin quality, and the color of the surface changes to red with the decrease of chlorophyll and the increase of lycopene. Therefore, as the hardness of the tomato is low and the surface color is closer to the red color, it can be estimated that the tomato has a higher maturity. As such, the method of classifying maturity based on the ratio of color distribution on the surface of the crop is most widely used because it is non-destructive and has convenience in applying it to industries.

Japanese Patent Laid-Open No. 2017-134585, (Patent Document 2) Japanese Patent No. 6632014

Evaluating Tomato Ripeness Using a Neural Network, JOURNAL OF SHITA, 8(3), 1996, (Non-Patent Document 2) Extraction methods of color and shape features for tomato grading, J.SHITA, 18(2), 2006

Previously, the classification of crop maturity such as tomatoes was dependent on classification by the naked eye. For example, it is common for a person to classify the maturity and grade the quality by comparing the color distribution and the ratio of the tomato surface with a color chart. An example of a color chart for the classification of tomatoes is a six-level classification as defined by the United States Department of Agriculture (USDA). The six-level classification according to the U.S. Department of Agriculture includes six levels of tomato maturity according to the green, tannish-yellow, pink, and red color combinations, and the percentage of the area they occupy on the surface of the tomato. It is divided into.

The six stages of tomato maturity classification of the US Department of Agriculture are shown in Table 1 below.

Stage number Stage name image Explanation One Green

The surface of the tomato is completely green
in color. The shade of green may vary from
light to dark 2 Breaker

There is a definite "break" in color from
green to tannish-yellow, pink or red on not
more than 10% of the surface 3 Turning

More than 10%, but not more than 30%, of the
surface, in the aggregate, shows a definite
change in color from green to tannish-yellow,
pink, red, or a combination thereof 4 Pink

More than 30%, but not more than 60%, of the
surface, in the aggregate, shows pink or red
in color 5 Light red

More than 60% of the surface, in the
aggregate, shows pinkish-red or red, provided
that not more than 90% of the surface is red 6 red

More than 90% of the surface, in the
aggregate, is red

However, this visual screening method requires training for the working manpower because the dependence on the subjectivity or expertise of the working manpower is very high. In addition, there was a problem that the speed of work was slow due to the labor-intensive nature, and the standards of maturity classification were inconsistent and the accuracy of the maturity classification result was degraded due to the subjective intervention of the workforce as the working time elapsed. In addition, due to the above-described limitations of the visual selection method, there is a limitation that is difficult to apply to a mass production system of crops.

Due to the limitation of such visual screening method, research on a system for evaluating the quality of crops and determining maturity has recently been conducted using computer vision technology. For example, Patent Literature 1 and Patent Literature 2 disclose a technique of extracting a crop surface color by computer vision technology and comparing the color with a predefined color database to determine the maturity of a crop.

Computer vision technology-based discrimination systems mostly acquire images of crops, preprocessors for noise filtering of collected RGB images, feature area detectors through image segmentation, and determine whether or not there are defects on the surface of crops and whether they are matured. It consists of a classifier.

In this conventional technique, it is common to classify maturity by extracting features of the relative ratio of color distribution of most crops. However, the conventional technique extracts color from a part of a crop and classifies the purity, that is, it is only based on the color or color ratio of the crop at a specific point in time. In addition, in order to realize this, it is necessary to fully understand the changes in the appearance characteristics of the specific crop according to the progress of maturation of a specific crop, so the expertise of the expert who designed the computer program is important, and factors that the designer has not already considered are reflected at all. There is a problem that cannot be done.

In addition, conventional computer vision technology is applied because changes in the characteristics of crops may appear differently depending on the crop, depending on the variety of the same crop, or the unique characteristics of the production unit, such as the production area or cultivation area, even if the same variety. There was an extremely limited limit to the following.

For example, among tomato crops, one cultivar may have a high probability of turning red from the stem-end region, and another cultivar may have a high probability of turning red from the flowe-end region. For another example, in the case of tomato crops, depending on the light direction or temperature (corresponding to human body temperature), it may turn red from near the stem-end, or turn red from near the flower-end, and the variety Therefore, since it may turn red in the form of a vertical band, the results of maturity classification may vary depending on the time point. In addition, when crops are classified using computer vision technology, different maturity classification results may be derived depending on the degree of light reflection or the direction of light in the image acquisition process.

In order to compensate for this, it may be considered not only to consider the relative color ratio at a specific point in time, but to classify the maturity by using a plurality of images acquired at different locations and a color change pattern based on a plurality of images. For example, images of near the apex (i.e., the first viewpoint) and the navel (i.e., the second viewpoint) of the crop, or an additional third viewpoint are all acquired and based on the classification information obtained at each viewpoint. If the final maturity of the crop is classified as a result, the above problems can be somewhat solved.

However, in the method of classifying crops based on images acquired at each time point, it becomes a question of how to classify the final maturity of the crop when the classification results between view points conflict. For example, if the maturity of the object is classified as the first stage as a result of the determination at the first point of time, and the maturity of the object is classified as the second stage as a result of the judgment at the second point, the final maturity of the crop individual is classified into which stage. There is a need for research on whether to do it.

Accordingly, the problem to be solved by the present invention is to determine the maturity of the crop by applying a deep learning model based on object detection and based on the input image of multiple viewpoints. It provides a method of classifying crops.

In addition, the probability-based score vector for each maturity class of each maturity class acquired from the multi-view input image is fused based on weight, but the weight for each viewpoint uniquely learned according to the crop to be identified is derived from the deep learning model and optimized. It is to provide a method of classifying crops that can maximize the use of spatial feature information of crops that may appear differently for each crop or variety by calculating the weights.

In particular, it provides a method of classifying crops with improved accuracy by using a vector not a simple scalar value as a weight, and further extracting only a vector for a predicted value, not a vector for the total value, based on the learned information.

Another problem to be solved by the present invention is that in applying a deep learning model based on object detection based on input images of multiple views, the reliability or classification accuracy of individual viewpoint images acquired from input images of multiple views can be improved. It is to provide a classification learning method.

Another problem to be solved by the present invention is to consider geometric and spatial feature information of colors by maturity stages distributed over all areas of a crop object, and maximize the use of spatial feature information of crops that may appear differently for each crop or variety. It is to provide a classification program for crops that are available.

Another problem to be solved by the present invention is to consider geometric and spatial feature information of colors by maturity stages distributed over all areas of a crop object, and maximize the use of spatial feature information of crops that may appear differently for each crop or variety. It is to provide a sorting device for crops that are available.

The problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

,

, …,

로 이루어진 제1 벡터를 출력 및 저장하는 단계; 및 제2 오차 행렬로부터 M개의 원소를 갖는 M개의 벡터를 포함하는 제2 벡터를 출력 및 저장하는 단계로서,

,

, …,

로 이루어진 제2 벡터를 출력 및 저장하는 단계를 포함한다.The method for classifying and learning crops according to an embodiment of the present invention for solving the above problem includes the steps of receiving a plurality of viewpoint images including a first viewpoint and a second viewpoint of a target object, respectively; Classification of reference data at the first and second viewpoints using an object detection-based deep learning model, and a probability-based first score vector for the first viewpoint and a probability-based second score for the second viewpoint Outputting a vector; Deriving a first error matrix and a second error matrix from the first score vector and the second score vector, respectively, the steps of: deriving a first error matrix and a second error matrix each having a matrix of M × M; A first step of outputting and storing the first vector containing the M vector has M elements from a first error matrix,

,

,… ,

Outputting and storing a first vector consisting of; And a second step of outputting and storing the second vector containing the M vector has M elements from the two-error matrix,

,

,… ,

And outputting and storing a second vector consisting of.

In this case, the first error matrix and the second error matrix may be a matrix having a horizontal axis as a prediction value and a vertical axis as a true value.

In addition, the first vector and the second vector may each include a column vector having M elements.

,

, …,

로 이루어진 제1 열벡터(column vector) 및

,

, …,

로 이루어진 제2 열벡터가 미리 정의되어 작물을 분류하는 방법으로서, 대상 객체의 제1 시점 및 제2 시점을 포함하는 복수 시점 영상을 각각 입력받는 단계; 딥러닝 모델을 이용하여 제1 시점에 관한 확률 기반 제1 스코어 벡터를 출력하는 단계로서, 최대 스코어가 i번째 클래스(i는 M 이하의 양의 정수)인 제1 스코어 벡터를 출력하는 단계; 딥러닝 모델을 이용하여 제2 시점에 관한 확률 기반 제2 스코어 벡터를 출력하는 단계로서, 최대 스코어가 j번째 클래스(j는 M 이하의 양의 정수)인 제2 스코어 벡터를 출력하는 단계; 상기 복수의 제1 벡터와 복수의 제2 벡터, 및 상기 제1 스코어 벡터와 제2 스코어 벡터에 기반하여 상기 제1 시점 및 상기 제2 시점에 관한 제1 가중치 벡터 및 제2 가중치 벡터를 포함하는 복수의 가중치 벡터를 결정하는 단계; 및 상기 복수의 스코어 벡터와 상기 복수의 가중치 벡터에 기반하여 최종 기준 데이터 레벨을 결정하는 단계를 포함한다.A method for classifying crops according to an embodiment of the present invention for solving the above other problems

,

,… ,

A first column vector consisting of and

,

,… ,

A method of classifying crops by defining a second row vector consisting of, comprising: receiving a plurality of viewpoint images including a first viewpoint and a second viewpoint of a target object, respectively; Outputting a probability-based first score vector for a first viewpoint using a deep learning model, the method comprising: outputting a first score vector whose maximum score is an i-th class (i is a positive integer less than or equal to M); Outputting a second score vector based on probability for a second viewpoint using a deep learning model, comprising: outputting a second score vector whose maximum score is a j-th class (j is a positive integer less than or equal to M); Including a first weight vector and a second weight vector for the first view and the second view based on the plurality of first vectors and the plurality of second vectors, and the first score vector and the second score vector Determining a plurality of weight vectors; And determining a final reference data level based on the plurality of score vectors and the plurality of weight vectors.

The first weight vector for the first view may be determined using Equation A below, and the second weight vector for the second view may be determined using Equation B below.

<Equation A>

<Equation B>

및

(여기서, i 및 j는 각각 M 이하의 양의 정수)는 각각 i번째 및 j번째 클래스에서 최대 스코어를 갖는 열벡터일 수 있다.Again, above

And

(Here, i and j are positive integers less than or equal to M , respectively) may be column vectors having maximum scores in the i-th and j-th classes, respectively.

Here, the following equation may be satisfied.

<Equation>

은 시점 n에 대한 가중치 벡터이다.here,

Is the weight vector for time n.

In addition, the determining of the final reference data level may be performed by including an algorithm using the following equation.

<Equation>

는 최종 결정 융합 벡터이고, N은 각 개별 시점의 총 개수를 의미하고, n은 각 개별 시점의 순번을 의미하고,

는 시점 n에서의 확률 기반 스코어 벡터이고,

은 시점 n에 대한 가중치 벡터이고,

는 성분별 곱(element wise multiplication)을 의미한다.here,

Is the final decision fusion vector, N means the total number of each individual time point, n means the order of each individual time point,

Is the probability-based score vector at time n,

Is the weight vector for time n,

Means element-wise multiplication.

A crop classification apparatus according to an embodiment of the present invention for solving the another problem includes an image input unit for receiving each of a plurality of viewpoint images of a target object; An individual discrimination unit for classifying reference data at each viewpoint of the object using an object detection-based deep learning model and outputting a plurality of probability-based score vectors for each viewpoint regarding the reference data level for the object; A weight determining unit for determining a plurality of weight vectors at each viewpoint based on a plurality of column vectors derived from an error matrix of the probability-based score vector and a maximum score of the probability-based score vector; And a final determination unit determining a final reference data level based on the plurality of probability-based score vectors and the plurality of weight vectors.

In the recording medium on which a program according to an embodiment of the present invention is recorded for solving the above another problem, a program for performing the above-described method for classifying crops may be recorded.

Details of other embodiments are included in the detailed description.

According to embodiments of the present invention, the final maturity is determined by fusing the probability-based score vector for each maturity class derived from each of the input images of multiple viewpoints based on weights, thereby overcoming the limitations of the conventional computer vision technology and Using spatial feature information, it is possible to more accurately and quickly determine the maturity of a crop.

In addition, the weight of each individual viewpoint image derived from the deep learning model can be given differently according to the viewpoint, and accordingly, according to the type of crop, according to the variety, and even according to the unique characteristics of a specific crop. It is possible to derive a fusion vector, and even if there is a contradiction or conflict in the classification result between multiple time points, it is possible to more accurately and quickly determine the maturity of the crop.

Effects according to the embodiments of the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a flowchart illustrating a method of learning crop classification based on deep learning according to an embodiment of the present invention.
2 is a schematic diagram of an apparatus for performing the learning method of FIG. 1.
3 illustrates a first error matrix derived from the first individual determination unit of FIG. 2.
4 illustrates a second error matrix derived from the second individual determination unit of FIG. 2.
5 illustrates a first precision table derived from the first error matrix of FIG. 3.
6 illustrates a second precision table derived from the second error matrix of FIG. 4.
7 is a flow chart showing a crop classification method according to an embodiment of the present invention.
8 is a schematic diagram of an apparatus for performing the method of FIG. 7.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only the embodiments make the disclosure of the present invention complete, and those skilled in the art to which the present invention pertains. It is provided to fully inform the person of the scope of the invention, and the invention is only defined by the scope of the claims. That is, various changes may be made to the embodiments proposed by the present invention. The embodiments described below are not intended to be limited to the embodiments, and should be understood to include all changes, equivalents, and substitutes thereto.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

In this specification,'and/or' includes each and every combination of one or more of the recited items. In addition, the singular form includes the plural form unless specifically stated in the text. As used herein,'comprises' and/or'comprising' does not exclude the presence or addition of one or more other elements other than the mentioned elements. Numerical ranges indicated using'to' represent numerical ranges including the values listed before and after them as lower and upper limits, respectively. "About" or "approximately" means a value or numerical range within 20% of the value or numerical range described thereafter.

In addition, terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a method of learning crop classification according to an embodiment of the present invention. 2 is a schematic diagram of an apparatus capable of performing the learning method of FIG. 1.

1 and 2, the method for classifying and learning crops according to an embodiment of the present invention includes receiving multiple viewpoint images of a target object (S100), and probability-based score vectors at each viewpoint based on the input images. Outputting (S200), outputting a plurality of learning vectors from a confusion matrix of the score vector (S300), and storing a learning result (S400).

The step S100 of receiving multiple viewpoint images may be a step of acquiring a plurality of viewpoint images photographing different viewpoints of at least one crop. That is, the step of receiving a multi-view image (S100) may include receiving a first image of a first view (S110) and a step of receiving a second image of a second view (S120).

The step S100 of receiving multiple viewpoints may be implemented by the image input unit 100 including the first image input unit 110 and the second image input unit 120. 1 and 2 illustrate a target object, for example, a top-view image photographed near the top of a tomato and a bottom-view image photographed near the navel of the target object as an image input interface. For example, the image input unit 100 receives two separate viewpoints, but the present invention is not limited thereto. In another embodiment, the image input unit 100 includes three or more image input units, and in this case, it is of course possible to additionally acquire a side view image or the like photographing the side of the target object.

The image input unit 100 may include a camera module (not shown) and a wired/wireless communication interface (not shown) to obtain an individual viewpoint image of the target object. The image input unit 100 may be configured in a plurality by being separated by hardware according to the number, or images of each individual viewpoint may be input by software from one hardware. For another example, the image input unit 100 may sequentially photograph each viewpoint by one piece of hardware. That is, in the present specification, the'individual viewpoint' means not only the case where the camera acquires the image of the target object but also the viewpoint according to the position of the target object.

For example, when a viewpoint is defined based on the position of the camera, an image interface acquired by a camera installed above (i.e., a first image input unit) is used as an image of a first viewpoint and a camera installed below (i.e., a second image). The image interface acquired by the input unit) may be defined as an image of a second view. For another example, when the viewpoint is defined based on the location of the target object, the image interface photographed above the target object is defined as the image from the first viewpoint, and the image interface photographed below the target object is defined as the image from the second viewpoint. can do.

Next, the step of outputting the probability-based score vector at each viewpoint based on the input image (S200) is to the individual determination unit 200 based on the individual viewpoint image of the target object input through the image input unit 100. By performing classification of the target object, for example, it may be a step of classifying the mastery class of the target object or learning to classify it. Outputting a probability-based score vector (S200) may include outputting a first score vector for a first viewpoint (S210) and outputting a second score vector for a second viewpoint (S220). . The individual determination unit 200 may be implemented through a classification software algorithm and a processor on which the software is executed.

In an exemplary embodiment, the classification model performed by the individual determination unit 200 may include a deep learning model based on object detection. The individual determination unit 200 uses a deep learning model based on object detection by inputting individual viewpoint images obtained from the first image input unit 110 and the second image input unit 120, respectively, and a probability-based score for each mastery class. You can output vectors. The individual discrimination unit 200 may be separated by hardware and configured as a plurality in accordance with the number of individual image input units 100, or may output a score vector at each individual viewpoint in software from one hardware. . 2 illustrates a case where the individual determination unit 200 includes a first individual determination unit 210 and a second individual determination unit 220.

The probability-based score vector for each maturity class output from the individual discrimination unit 200 may include a discrete probability distribution data form that comprehensively expresses a color information pattern of a target object appearing in an individual viewpoint image.

According to the present invention, by using a model based on object detection, it is possible to classify a target object by considering not only the color ratio of the target object but also the characteristic of a geometric color distribution. In addition, the reason why the individual determination unit 200 uses the deep learning model based on objection detection instead of the image classification method in the present embodiment is the possibility that a plurality of target objects will be included in one image. It was considered.

That is, the individual discrimination unit 200 may be designed to classify the state of the object, that is, the maturity of the crop, rather than classifying it as a crop in classifying the class of the object through the object detection operation. The individual discrimination unit 200 according to the present invention can perform the class classification of the object and the position estimation at the same time because the deep learning model used as the core of the mastery classification operates based on object search. Even if a plurality of tomatoes are included in an image acquired by the image input unit 110 or an image acquired by each of the second image input unit 120, maturity may be classified by dividing each tomato object.

In other words, each individual determination unit 200, that is, the first individual determination unit 210 and the second individual determination unit 220, both when the image viewpoint of the target object is constant and the image viewpoint of the target object is not constant. Therefore, it is possible to simultaneously perform a plurality of estimating the location of the target object, estimating the point of view of the target object, estimating the position of the bounding box of the target object, classifying the color of the target object, and classifying the maturity of the target object, so that it can be used for tomato maturity classification.

Various deep learning models may be used as the maturity determination model of the individual determination unit 200 including the first individual determination unit 210 and the second individual determination unit 220. For example, R-convolutional neural network (R-region-based convolutional neural network, R-) designed with a depth and structure suitable for classifying the maturity of objects in finding various relationships and patterns from collected data. Object detection models such as CNN) model or YOLO (you only look once) model and individual instance segmentation models such as Mask R-CNN model can be used.

For example, the R-CNN model calculates the candidate region of the bounding box surrounding the target object of the input image through a separate first search unit or a region proposal generator, and calculates the estimation result of the first search unit. Based on the result, it is possible to have a two-step search structure that performs object classification and bounding box regression by passing the result to the second search unit. Preferably, the individual determination unit 200 according to the present embodiment may derive a result of a high mean average precision (mAP) and a fast processing speed of 5FPS level by using the faster R-CNN.

As another example, the YOLO model is a single stage detector, and the estimation of the bounding box from the raw data input of the pixel level of the input image and the classification of the maturity class of the target object are performed in one convolutional neural network ( CNN) stream can be used. In other words, it is possible to classify real-time streams without bottlenecks by designing a single-step searcher of the YOLO model having a fast search speed and a high level of mAP among single-step searchers.

을 구하기 위해 각각의 후보 바운딩 박스에 대한 출력 벡터를 원소로 갖는 집합

을 출력할 수 있다. 구체적으로, 제1 개별 판별부(210)는 벡터

을 구하기 위해 집합

을 출력하고, 제2 개별 판별부(220)는 벡터

를 구하기 위해 집합

를 출력할 수 있다.As a non-limiting example, the individual determination unit 200 is an output vector

The set containing the output vector for each candidate bounding box as an element to obtain

Can be printed. Specifically, the first individual determination unit 210 is a vector

Set to save

And the second individual determination unit 220 is a vector

Set to save

Can be printed.

은 하기 수식 1과 같이 정의될 수 있다.At this time, the candidate set

May be defined as in Equation 1 below.

<Equation 1>

는 각 숙도 단계에 대한 우도(likelihood)

을 원소로 가지며, 하기 수식 2와 같이 정의될 수 있다.Here, K is the number of candidate bounding boxes, and an output vector for each candidate bounding box

Is the likelihood for each maturity level

Has as an element, and may be defined as in Equation 2 below.

<Equation 2>

은 하기 수식 3과 같이 정의될 수 있다.Where M is the number of classes to be classified, and

May be defined as in Equation 3 below.

<Equation 3>

는 k번째 후보 경계 상자 내 객체(obj)인 작물이 존재한다는 조건 하에, 그 작물의 숙도 단계가 c _m일 조건부 확률이고, C_k는 k 번째 후보 경계 상자 내에 객체인 작물이 존재할 확률(confidence score)로 정의한다.here

Is the conditional probability that the crop's maturity level is c _m , under the condition that there is a crop that is an object (obj) in the k-th candidate bounding box, and C _k is the probability that the crop, which is an object, exists in the k-th candidate bounding box (confidence score). ).

를 생성하고, 각 출력 벡터는 M개의 원소를 가지므로

의 총 개수는 K×M개일 수 있다.Therefore, for the input image of the viewpoint n, the individual determination unit 200 outputs the vector for each of K candidate bounding boxes.

And each output vector has M elements, so

The total number of may be K × M.

을 포함하는 후보 바운딩 박스의 인덱스 k를

라 하면, 개별 판별부(200)의 출력 벡터

는 하기 수식 4와 같다.Of all the elements printed here,

The index k of the candidate bounding box containing

If d, the output vector of the individual discrimination unit 200

Is as shown in Equation 4 below.

<Equation 4>

을 포함하는 후보 바운딩 박스의 인덱스

는 NMS(non-maximum suppression) 알고리즘을 통해 구할 수 있다.Here, the largest of the total K × M elements

Index of the candidate bounding box containing

Can be obtained through a non-maximum suppression (NMS) algorithm.

Meanwhile, the first individual determination unit 210 and the second individual determination unit 220 may learn different deep learning models for each individual viewpoint, or learn one deep learning model without distinction for individual viewpoints.

In an exemplary embodiment, the first individual determination unit 210 and the second individual determination unit 220 learn the same deep learning model, but are pre-trained based on public databases (COCO, ImageNet, etc.). trained) By fine-tuning the backbone network architecture for multiple view images of tomatoes, the deep learning model of the maturity process at individual viewpoints can be learned and appropriate model initial parameters can be set. Since the final decision mastery level depends on the classification result of the first individual determination unit 210 and the second individual determination unit 220, a learning strategy must be established in a direction to maximize the performance of the deep learning model.

To do this, it is necessary to expand the size of the dataset used to train the deep learning model. The dataset may include a top view image and a bottom view image for each tomato object, that is, a pair of RGB image data collected from a bidirectional view and annotation data corresponding to each image. In this case, the annotation data may include information on the maturity stage of the tomato object in the image, the location and size of the bounding box.

As an in-depth non-limiting example, the method performed by the inventor of the present invention in the process of completing the present invention is as follows.

In other words, in the image collection process, tomato objects of the Daphnis variety are divided into six levels of maturity based on the U.S. Department of Agriculture color chart, and placed in the chamber of the camera system. It proceeded by storing the captured raw image data. At this time, the collected image was captured through a JAI 3CCD color camera as a 1624×1280 pixel resolution, 48-bit, 2-channel TIF image. The captured image was converted into an 8-bit 3-channel JPEG file for deep learning and used. In addition, in order to improve the performance of the deep learning model by securing as many primitive features as possible to express the maturity level, a large-scale data set consisting of at least 500 individuals for each maturity level was constructed. The large-scale dataset of tomato individuals collected in this way is used for learning a deep learning model constituting a maturity classification method or a maturity classification device according to the present invention. The learning algorithm follows a mini-batch-based stochastic gradient descent (SGD), and parameters set by the present inventor are as follows.

-max training steps: 50,000

-mini-batch/subdivisions size: 32/8

-learning rate: 0.01 (scale 0.1 at step 25,000, 35,000)

-weight decay: 0.0005

-learning momentum: 0.9

In addition, in order to improve the performance of the deep learning model, it is necessary to find the hyperparameter setting of the model that can best detect the explicit features of each stage of maturity in addition to the expansion of the used dataset pool. The present invention has defined the network depth, that is, the number of layers and the number of scales of the feature map finally outputted by the network, as optimization targets in consideration of a bias-variance tradeoff among several types of hyper parameters. To this end, the inventors of the present invention performed a process of classifying individual viewpoint maturity using a CNN structure capable of extracting a feature map of single/multi scale based on various depth structures and analyzed the result. The candidates carried out by the present inventors were the following conditions.

-total stride=32 single scale, darknet-19-based network model

-total stride=8, 16, 32 triple scale, darknet-19 based network model

-total stride=32 single scale, darknet-107 based network model

-total stride=8, 16, 32 triple scale, darknet-107 based network model

In addition, among the several CNN models used as candidates, the model with 107 convolutional layers and set hyperparameters to output feature maps of three scales through three up-sampling is in accordance with the results of the individual viewpoint maturity classification process. As the size of the micro-average ROC curve was the largest, the average accuracy was found to be the best. This means that under the condition of classifying tomato maturity at individual viewpoints using the tomato image dataset collected in the present embodiment, a model capable of outputting a feature map with a deep structure and various scales can be an efficient selection.

Next, the step of outputting a plurality of learning vectors from the error matrix according to each viewpoint (S300) may be a step of deriving an error matrix from the score vector for each mastery class, and extracting a column vector (learning vector) from the error matrix. . For example, the step of outputting a plurality of learning vectors (S300) may be arranged as a set of column vectors divided by mastery classes according to the viewpoint, and the result is stored in the learning vector storage unit 400 as described later. Can be saved. 2 illustrates that the step of outputting a learning vector (S300) is performed by the learning vector calculation unit 300 including the first learning vector calculation unit 310 and the second learning vector calculation unit 320, but the present invention This is not limited thereto, and of course, in some embodiments, the step S300 of outputting the learning vector may be configured and performed together with the individual determination unit 200.

In an exemplary embodiment, the learning vector by the learning vector operation unit 300 may be determined from a confusion matrix derived from a learning result of the deep learning model of the individual determination unit 200. As a non-limiting example, an error matrix at a first point in time when a 5-step classification is performed through deep learning, that is, a first error matrix derived from the first individual determination unit 210 is exemplarily shown in FIG. 3, and The error matrix at the second time point in which the five-stage classification is performed through running, that is, the second error matrix derived from the second individual discriminating unit 220 is exemplarily shown in FIG. 4.

3 illustrates a first error matrix derived from the first individual determination unit 210 of FIG. 2. 4 illustrates a second error matrix derived from the second individual determination unit 220 of FIG. 2. In FIGS. 3 and 4, the vertical axis denotes a true value input in the learning process, and the horizontal axis denotes a class prediction value calculated by the deep learning model. In this case, the classes to be classified is a first error matrix and the second error matrix may have a matrix of the respective M × M when M clear up.

That is, the error matrices of FIGS. 3 and 4 are actually 126 tomatoes having a first stage maturity, 197 tomatoes having a second stage maturity, 250 tomatoes having a third stage maturity, and 141 tomatoes having a fourth stage maturity. It shows the result of performing the classification by the individual determination unit 200 using 116 tomatoes having a dog, five-stage maturity.

Referring further to FIG. 3, it is exemplified that the first individual discrimination unit 210 actually classifies 118 out of 126 tomatoes having a maturity level of 1 into a first step, and classifies 8 into a second step. In addition, it is exemplified that 12 out of 197 tomatoes that actually have 2 stages of maturity are classified into 1 stage, 159 are classified into 2 stages, 25 are classified into 3 stages, and 1 are classified into 4 stages. . In addition, it is exemplified that out of 250 tomatoes with 3 stages of maturity, 8 were classified into 2 stages, 228 were classified into 3 stages, and 14 were classified into 4 stages. In addition, it is exemplified that 3 out of 141 tomatoes with 4 stages of maturity were classified into 3 stages and 138 were classified into 4 stages. In addition, it is exemplified that one of 116 tomatoes with 5 levels of maturity was actually classified into 4 stages and 115 of them were classified into 5 stages.

Likewise, the error matrix of FIG. 4 represents a second error matrix, which is a result of classification by the second individual discrimination unit 220 based on an image with a different viewpoint for the same tomato as tested in FIG. 3. That is, FIG. 4 is a result of classifying values of an image captured at a different viewpoint than that of FIG. 3 for the same tomato set by the same method.

Referring further to FIG. 4, the second individual determination unit 220 actually classifies 102 out of 126 tomatoes having a maturity level of 1 into 1 stage, 15 into 2 stages, and 9 into 3 stages. Illustratively. In addition, it is exemplified that 17 out of 197 tomatoes that actually have a two-stage maturity were classified as one, 172 as two, and eight were classified as three. In addition, 12 out of 250 tomatoes that actually have 3 stages of maturity are classified into 1 stage, 13 are classified into 2 stages, 214 are classified into 3 stages, 7 are classified into 4 stages, and 4 It exemplifies that is classified into 5 stages. In addition, it is exemplified that 22 out of 141 with 4 levels of mastery are actually classified into 3 levels, 105 are classified into 4 levels, and 14 are classified into 5 levels. In addition, it is exemplified that out of 116 tomatoes with 5 levels of maturity, 7 are classified into 3 stages, 12 are classified into 4 stages, and 97 are classified into 5 stages.

Although the present invention is not limited thereto, the difference between the above results of the first individual discrimination unit 210 and the second individual discrimination unit 220, that is, the difference in the error matrix, is due to the inherent characteristics of the target crop. I can. For example, when there is a high probability that the color of the tomato, which is a target crop, changes from the vicinity of the top, the first error matrix may provide a result that is closer to the actual one as in the above example. As another example, the difference in the above results may be due to the characteristics of the classifier. For example, it may be because the first individual determination unit 210 has a high possibility of classifying steps 1 to 3 more accurately, and the second individual determination unit 220 has a high possibility of classifying steps 4 and 5 more accurately. .

Meanwhile, a precision table can be derived using an error matrix. In a non-limiting example, the first and second correct tables derived from the first error matrix of FIG. 3 and the second error matrix of FIG. 4 are exemplarily shown in FIGS. 5 and 6, respectively. In an exemplary embodiment, the individual determination unit 200 and/or the vector operation unit 300 may output an accuracy table as shown in FIGS. 5 and 6 from the error matrix of FIGS. 3 and 4.

5 illustrates a first accuracy table derived from the first error matrix of FIG. 3. 6 illustrates a second accuracy table derived from the second error matrix of FIG. 4.

The correct table means the ratio of the positives (TP) to the sum of the false positives (FP, false positives, false positives) and the positives (TP) (TP/(TP+FP)). That is, the correct table means the ratio of positives among the predicted values. In some other embodiments, a recall table may be used rather than a precision table. The above table refers to the ratio of the positive positive (TP) to the sum of the false negative (FN, false negative, false negative) and the positive positive (TP) (TP/(TP+FN)). That is, the above table means the ratio of positives among true values.

First, referring to FIG. 5 further, the first-stage positive image among the positive image (TP) values that cross the first error matrix of FIG. 3 in a diagonal direction is 118 out of 130 predicted values, so it has a precision of about 90.8%. . On the other hand, 12 of the 130 predictions predicted for the first stage are actually second stage maturity, so the false positive error (FP) of the second stage tomato as the first stage is 9.2%.

는 [90.8, 9.2, 0, 0, 0] 값을 갖는 열벡터(column vector)일 수 있다.Therefore, among the values predicted for the first level of maturity among the images acquired at the first point in time, a vector in which the values of the positive images and the positive errors are distributed in the order of actual maturity can be derived, and this can be defined as the 1-1 learning vector. have. In this case, the 1-1 learning vector

May be a column vector having a value of [90.8, 9.2, 0, 0, 0].

Likewise, among the positive values that cross the first error matrix of FIG. 3 in a diagonal direction, the second-stage positive images are 159 out of 175 predicted values, and thus have an accuracy of about 90.9%. On the other hand, out of the 175 predicted values predicted for stage 2 maturity, 8 were actually stage 1 maturity, and 8 were actually stage 3 maturity, so the false positives of erroneously judging the tomatoes of stage 1 and 3 as stage 2 were 4.6% and 4.6, respectively. %to be.

는 [4.6, 90.9, 4.6, 0, 0] 값을 갖는 열벡터일 수 있다.Therefore, among the values predicted for the second level of maturity among the images acquired at the first point in time, a vector listing the values in which the positive images and the positive errors are distributed in the order of actual maturity can be derived, and this can be defined as a 1-2 learning vector. have. In this case, the 1-2 learning vector

May be a column vector having a value of [4.6, 90.9, 4.6, 0, 0].

In addition, among the positive image values that cross the first error matrix of FIG. 3 in a diagonal direction, the third-stage positive image is 228 out of 256 predicted values, and thus has an accuracy of about 89.1%. On the other hand, 25 of the 256 predicted values predicted by 3rd level maturity are actually 2nd level maturity, and 3 are actually 4th level maturity. 1.2%.

는 [0, 9.8, 89.1, 1.2, 0] 값을 갖는 열벡터일 수 있다.Therefore, among the values predicted at the third level of maturity among the images acquired at the first point in time, a vector in which the values of the positive images and the positive errors are distributed in the order of actual maturity can be derived, and this can be defined as the 1-3 learning vector. have. In this case, the 1-3 learning vector

May be a column vector having a value of [0, 9.8, 89.1, 1.2, 0].

In addition, among the positive image values crossing the first error matrix of FIG. 3 in a diagonal direction, the fourth step positive image is 138 out of 154 predicted values, and thus has an accuracy of about 89.6%. On the other hand, out of the 154 predicted values predicted by the 4th stage, 1 is actually 2nd stage, 14 is actually 3rd stage, and 1 is actually 5th stage. The positive errors that were incorrectly judged in step 4 were 0.6%, 9.1% and 0.6%, respectively.

는 [0, 0.6, 9.1, 89.6, 0.6] 값을 갖는 열벡터일 수 있다.Therefore, among the values predicted at the 4th level of maturity among the images acquired at the first point in time, a vector listing the values in which the positive images and the positive errors are distributed in the order of actual maturity can be derived, and this can be defined as the 1-4 learning vector. have. In this case, the 1-4 learning vector

May be a column vector having a value of [0, 0.6, 9.1, 89.6, 0.6].

는 [0, 0, 0, 0, 100] 값을 갖는 열벡터일 수 있다.Lastly, 115 of the 115 predicted with 5 levels of maturity are positive. Therefore, it is possible to derive a vector in which the values in which the positive images and the positive errors are distributed among the values predicted at the fifth level of maturity among the images acquired at the first point in time, in the order of the actual maturity, can be derived, and this can be defined as the 1-5 learning vector. have. In this case, learning vectors 1-5

May be a column vector having a value of [0, 0, 0, 0, 100].

Next, referring to FIG. 6 further, among the positive positive images (TP) values crossing the second error matrix of FIG. 4 in a diagonal direction, the first-stage positive images are 102 out of 131 predicted values, so about 77.9% precision is obtained. Have. On the other hand, out of the 131 predicted values for stage 1 maturity, 17 are actually 2 stage maturity, and 12 are actually 3 stage maturity, so 13.0% and 9.2% falsely judged tomatoes in stages 2 and 3 as stage 1 .

는 [77.9, 13.0, 9.2, 0, 0] 값을 갖는 열벡터일 수 있다. Therefore, among the values predicted for the first level of maturity among the images acquired at the second point in time, a vector listing the values in which the positive images and the positive errors are distributed in the order of actual maturity can be derived, and this can be defined as the 2-1 learning vector. have. In this case, the 2-1 learning vector

May be a column vector having a value of [77.9, 13.0, 9.2, 0, 0].

Likewise, among the positive positive images (TP) values crossing the second error matrix of FIG. 4 in a diagonal direction, the second-stage positive images are 172 out of 200 predicted values, and thus have a precision of about 86.0%. On the other hand, out of 200 predicted values for stage 2 maturity, 15 are actually stage 1 maturity, and 13 are actually stage 3 maturity, so the false positives of erroneously judging the tomatoes of stage 1 and stage 3 as stage 2 are 7.5% and 6.5%, respectively. to be.

는 [7.5, 86.0, 6.5, 0, 0] 값을 갖는 열벡터일 수 있다.Therefore, it is possible to derive a vector in which the values in which the positive images and the positive errors are distributed among the values predicted with the second level of maturity among the images acquired at the second point in time, in the order of actual maturity, can be derived, and this can be defined as a 2-2 learning vector. have. In this case, the 2-2 learning vector

May be a column vector having a value of [7.5, 86.0, 6.5, 0, 0].

In addition, among the positive positive images (TP) values crossing the second error matrix of FIG. 4 in a diagonal direction, the third-stage positive images are 214 out of 260 predicted values, and thus have a precision of about 82.3%. On the other hand, out of the 260 predicted values predicted by 3rd level maturity, 9 are actually 1st level, 8 are actually 2nd level, 22 are actually 4th level, and 7 are actually 5th level. The affirmative errors that incorrectly judged the tomatoes of stages 4 and 5 as stage 3 were 3.5%, 3.1%, 8.5%, and 2.7%, respectively.

는 [3.5, 3.1, 82.3, 8.5, 2.7] 값을 갖는 열벡터일 수 있다.Therefore, among the values predicted at the 3rd level of maturity among the images acquired at the second point in time, a vector in which the values of the positive and false positives are distributed in the order of actual maturity can be derived, and this can be defined as the 2-3 learning vector. have. In this case, the 2-3 learning vector

May be a column vector having a value of [3.5, 3.1, 82.3, 8.5, 2.7].

In addition, among the positive positive images (TP) values crossing the second error matrix of FIG. 4 in a diagonal direction, the fourth-stage positive images are 105 out of 124 predicted values, so they have a precision of about 84.7%. On the other hand, out of the 124 predicted values for stage 4 maturity, 7 are actually 3 stage maturity, and 12 are actually 5 stage maturity, so 5.6% and 9.7% of the false positive errors of misjudged tomatoes in stages 3 and 5 as stage 4, respectively. to be.

는 [0, 0, 5.6, 84.7, 9.7] 값을 갖는 열벡터일 수 있다. Therefore, among the values predicted at the fourth level of maturity among the images acquired at the second point in time, a vector that lists the values of the positive images and the positive errors in the order of actual maturity can be derived, and this can be defined as the 2-4 learning vector. have. In this case, the 2-4 learning vector

May be a column vector having a value of [0, 0, 5.6, 84.7, 9.7].

Finally, among the positive positive images (TP) values crossing the second error matrix of FIG. 4 in a diagonal direction, the fifth-stage positive images are 97 out of 115 predicted values, and thus have a precision of about 84.3%. On the other hand, of the 115 predictions predicted for the 5th level, 4 are actually 3rd level, and 14 are actually 4th level, so the false positives of incorrectly judging the 3rd and 4th stage tomatoes as 5th stage are 3.5% and 12.2%, respectively. to be.

는 [0, 0, 3.5, 12.2, 84.3] 값을 갖는 열벡터일 수 있다.Therefore, it is possible to derive a vector listing the values in which the positive images and the positive errors are distributed among the values predicted at the 5th level of maturity among the images acquired at the second point in time, in order of actual maturity, and this can be defined as the 2-5 learning vector. have. In this case, the 2-5 learning vector

May be a column vector having a value of [0, 0, 3.5, 12.2, 84.3].

), 제1-2 학습 벡터(

), 제1-3 학습 벡터(

), 제1-4 학습 벡터(

) 및 제1-5 학습 벡터(

)를 포함하는 제1 학습 벡터(

,

, …,

)는 M개의 벡터로 이루어지고, 제1-1 학습 벡터(

), 제1-2 학습 벡터(

), 제1-3 학습 벡터(

), 제1-4 학습 벡터(

) 및 제1-5 학습 벡터(

)는 각각 M개의 원소를 가질 수 있다.That is, the 1-1 learning vector (

), the 1-2 learning vector (

), the 1-3 learning vector (

), the 1-4 learning vector (

) And the 1-5 learning vector (

A first learning vector containing (

,

,… ,

) Consists of M vectors, and the 1-1 learning vector (

), the 1-2 learning vector (

), the 1-3 learning vector (

), the 1-4 learning vector (

) And the 1-5 learning vector (

) May each have M elements.

), 제2-2 학습 벡터(

), 제2-3 학습 벡터(

), 제2-4 학습 벡터(

) 및 제2-5 학습 벡터(

)를 포함하는 제2 학습 벡터(

,

, …,

)는 M개의 벡터로 이루어지고, 제2-1 학습 벡터(

), 제2-2 학습 벡터(

), 제2-3 학습 벡터(

), 제2-4 학습 벡터(

) 및 제2-5 학습 벡터(

)는 각각 M개의 원소를 가질 수 있다.Also, the 2-1 learning vector (

), the 2-2 learning vector (

), the 2-3 learning vector (

), the 2-4 learning vector (

) And the 2-5 learning vector (

A second learning vector containing (

,

,… ,

) Consists of M vectors, and the 2-1 learning vector (

), the 2-2 learning vector (

), the 2-3 learning vector (

), the 2-4 learning vector (

) And the 2-5 learning vector (

) May each have M elements.

These column vectors are summarized as follows.

및

(여기서, m은 M 이하의 정수)는 각각 m번째 클래스에서 최대 스코어를 갖는 열벡터일 수 있다.As can be seen above

And

(Here, m is an integer less than or equal to M ) may be a column vector having a maximum score in each m-th class.

If the maturity of the tomato measured at the first time point and the maturity of the tomato measured at the second time point are different from each other, it may be difficult to determine whether the information from the first time point is correct or the information from the second time point is accurate. I can. Therefore, it is decided whether to give a higher weight by trusting the maturity classification of the measured tomato at the first time point or to give the weight higher by more trusting the maturity classification of the tomato measured at the second time point. It may be desirable to determine the final level of. Therefore, the accuracy of the first time point and the second time point may differ from each other in terms of the accuracy of the tomato prediction value at which of the predicted values for which tomato maturity of the first time and the second time point. The considered weight can be obtained from the first learning vector and the second learning vector. This will be described later together with the method of classifying crops.

Next, the vector storage unit 400 stores the learning vector values output from the confusion matrix, that is, 2M vector results including the first learning vector and the second learning vector. For example, when the predicted mastery classes are the same, the plurality of learning vectors outputted by the error matrix uses the distribution of the actual mastery classes as a learning result, and the learning result is stored in the vector storage unit 400. Vectors stored in the vector storage unit 400 may be used to calculate a weight for determining the importance or priority of each time point in classifying the maturity of a crop in the future.

Hereinafter, a method for classifying crops according to the present invention and an apparatus therefor will be described. However, a description of the same or similar configuration as the above-described method for learning the classification of crops will be omitted, and this will be clearly understood by those of ordinary skill in the art from the accompanying drawings.

7 is a flow chart showing a crop classification method according to an embodiment of the present invention. 8 is a schematic diagram of an apparatus for performing the method of FIG. 7.

7 and 8, the method of classifying crops according to an embodiment of the present invention includes receiving multiple viewpoint images of a target object (S500), and calculating a probability-based score vector at each viewpoint based on the input image. Outputting (S600), determining a weight vector from the learned training vectors (S700), and determining a final reference data level (S800).

The step S500 of receiving multiple viewpoint images may be a step of acquiring a plurality of viewpoint images photographing different viewpoints of at least one crop. That is, the step of receiving a multi-view image (S500) may include a step of receiving a first image of a first view (S510) and a step of receiving a second image of a second view (S520).

Further, the step S500 of receiving multiple viewpoints may be implemented by the image input unit 500 including the first image input unit 510 and the second image input unit 520. 7 and 8 show a case in which the image input unit 500 receives two separate viewpoints in order to implement a top view image photographed near the top of the target object and a bottom view image photographed near the navel of the target object as an image input interface. Is exemplified.

In addition, since the step of receiving the image (S500) and the image input unit 500 have been described above together with the learning method, a redundant description will be omitted.

Next, the step of outputting the probability-based score vector at each viewpoint based on the input image (S600) is to the individual determination unit 600 based on the individual viewpoint image of the target object input through the image input unit 500. By performing classification of the target object, for example, it may be a step of classifying the mastery class of the target object.

Outputting the probability-based score vector (S600) may include outputting a first score vector for a first viewpoint (S610) and outputting a second score vector for a second viewpoint (S620). . The individual determination unit 600 may be implemented through a classification software algorithm and a processor on which the software is executed.

In an exemplary embodiment, the classification model performed by the individual determination unit 600 may include a deep learning model based on object detection or a deep learning model based on individual instance sementation. The individual discrimination unit 600 uses a deep learning model based on object detection by inputting individual viewpoint images obtained from the first image input unit 510 and the second image input unit 520, respectively, and a probability-based score for each mastery class. You can output vectors. 8 illustrates a case where the individual determination unit 600 includes a first individual determination unit 610 and a second individual determination unit 620.

In addition, since the step of outputting the score vector (S600) and the individual determination unit 600 have been described above together with the learning method, a redundant description will be omitted.

Next, the step of determining the weight vector according to each viewpoint (S700) is based on the individual viewpoint image of the target object input through the image input unit 500 and the classification of the object by the individual determination unit 600, It may be a step of determining a weight for determining the importance or priority of the viewpoint. Determining the weight vector (S700) may include calculating a first weight vector for a first viewpoint (S710) and calculating a second weight vector for a second viewpoint (S720).

및

가 하기와 같은 경우를 예로 하여 설명하나, 본 발명이 이에 제한되지 않음은 물론이다.Hereinafter, each score vector derived from the first individual determination unit 610 and the second individual determination unit 620

And

The following case is described as an example, but the present invention is not limited thereto.

가 출력되고, 제2 시점에서는

가 출력될 수 있다. 제1 개별 판별부(610) 및 제2 개별 판별부(620)는 최대 스코어를 갖는 클래스를 선택하여 해당 시점에서의 숙도로 판별할 수 있다. 위와 같은 예시에서, 제1 시점의 제1 개별 판별부(610)는 최대 스코어로 80을 갖는 클래스 1(첫번째 원소), 즉 1단계 숙도로 판별하고, 제2 시점의 제2 개별 판별부(620)는 최대 스코어로 80을 갖는 클래스 2(두번째 원소), 즉 2단계 숙도로 판별할 수 있다.That is, at the first point in time for one tomato object

Is output, and at the second point

Can be output. The first individual determination unit 610 and the second individual determination unit 620 may select a class having a maximum score and determine the degree of mastery at a corresponding time point. In the above example, the first individual determination unit 610 at the first time point determines the class 1 (the first element) having a maximum score of 80, that is, the first level of maturity, and the second individual determination unit 620 at the second time point. ) Can be identified as class 2 (the second element) with a maximum score of 80, that is, level 2 maturity.

을 바탕으로 제1 가중치 벡터를 결정할 수 있다. 이 경우 제1 가중치 결정부(710)가 결정하는 제1 가중치 벡터는 하기 수식 5를 통해 도출될 수 있다.In an exemplary embodiment, the first weight determination unit 710 includes the learned first learning vector and the second learning vector and the vector.

A first weight vector may be determined based on. In this case, the first weight vector determined by the first weight determiner 710 may be derived through Equation 5 below.

<Equation 5>

Here, the symbol "./" means element-wise division.

,

)를 고려하되, 제1 개별 판별부(610)가 출력한 벡터

가 객체의 숙도 클래스를 1단계로 판단하였기 때문에 위에서 예시한 10개의 학습 벡터들 중에서, 1단계로 예측했던 결과에 대한 오차 행렬로부터 학습된 제1-1 학습 벡터

및 제2-1 학습 벡터

를 이용하여 제1 가중치 벡터를 연산 및 결정할 수 있다.That is, in calculating the first weight vector for the first viewpoint, the learning vector for both the first viewpoint and the second viewpoint (

,

), but the vector output by the first individual determination unit 610

Is the 1st learning vector learned from the error matrix for the result predicted by the 1st step among the 10 training vectors illustrated above, since the mastery class of the object is determined as 1st level.

And the 2-1 learning vector

The first weight vector may be calculated and determined using.

를 바탕으로 제2 가중치 벡터를 결정할 수 있다. 이 경우 제2 가중치 결정부(720)가 결정하는 제2 가중치 벡터는 하기 수식 6을 통해 도출될 수 있다.In addition, the second weight determination unit 720 and the learned second learning vector and the second learning vector

A second weight vector may be determined based on. In this case, the second weight vector determined by the second weight determiner 720 may be derived through Equation 6 below.

<Equation 6>

,

)를 고려하되, 제2 개별 판별부(620)가 출력한 벡터

가 객체의 숙도 클래스를 2단계로 판단하였기 때문에 위에서 예시한 10개의 학습 벡터들 중에서, 2단계로 예측했던 결과에 대한 오차 행렬로부터 학습된 제1-2 학습 벡터

및 제2-2 학습 벡터

를 이용하여 제2 가중치 벡터를 연산 및 결정할 수 있다.That is, in calculating the second weight vector for the second viewpoint, the learning vector for both the first viewpoint and the second viewpoint (

,

), but the vector output by the second individual determination unit 620

Is determined as the maturity class of the object in step 2, from among the 10 training vectors exemplified above, the 1-2 learning vector learned from the error matrix for the result predicted in step 2

And 2-2 learning vector

The second weight vector may be calculated and determined using.

In calculating the weight vector, the inventors of the present invention do not simply combine the overall learning result and reliability of the first point of view and the overall learning result of the second view and the reliability of the weight vector, but derive each point of view in the process of classifying the actual crops. The present invention was accomplished by focusing on the superiority of the classification result in determining the weight by extracting only the column vector for the generated score vector class.

If the explanation exemplified above is generalized, it can be expressed as follows.

,

, …,

로 이루어진 제1 학습 열벡터(column vector) 및

,

, …,

로 이루어진 제2 학습 열벡터가 미리 정의되고, 작물의 분류 결과 최대 스코어를 갖는 i번째 클래스를 갖는 제1 스코어 벡터

및 최대 스코어를 갖는 j번째 클래스를 갖는 제2 스코어 벡터

가 출력된 경우, 제1 가중치 벡터(

) 및 제2 가중치 벡터(

)는 각각 하기 수식 7 및 수식 8을 이용하여 결정될 수 있다.In other words, from the results of learning through deep learning

,

,… ,

A first learning column vector consisting of and

,

,… ,

A second learning column vector consisting of is predefined, and a first score vector having the i-th class having the maximum score as a result of classification of the crop

And a second score vector having the j-th class with the maximum score.

Is output, the first weight vector (

) And the second weight vector (

) May be determined using Equation 7 and Equation 8 below, respectively.

<Equation 7>

<Equation 8>

는 제1 시점에서 얻은 학습 벡터로부터 유래한 것이고,

는 제2 시점에서 얻은 학습 벡터로부터 유래한 것일 수 있다. 또, i 및 j는 각각 제1 시점 및 제2 시점의 분류 결과 최대 스코어를 갖는 클래스의 번호이다.As described above

Is derived from the learning vector obtained at the first time point,

May be derived from the learning vector obtained at the second point in time. In addition, i and j are the numbers of the class having the maximum score as a result of the classification at the first time point and the second time point, respectively.

In some embodiments, the weight vectors may satisfy Equation 9 below.

<Equation 9>

은 시점 n에 대한 가중치 벡터이다. 다만, 본 발명이 수식 9에 제한되는 것은 아니며, 다른 실시예에서 가중치 벡터들의 합은 1일 수도 있다. 여기서 벡터의 합이 1이라 함은,

가 5개의 원소를 갖는 열벡터인 경우

임을 의미한다. here,

Is the weight vector for time n. However, the present invention is not limited to Equation 9, and the sum of weight vectors may be 1 in another embodiment. Here, the sum of vectors is 1,

Is a 5 element column vector

Means

) 및 제2 가중치 벡터(

)를 결정할 수 있다.Through the above process, the first weight vector (

) And the second weight vector (

) Can be determined.

Next, in the step of determining the final reference data level (S800), the probability-based score vector output from the individual determination unit 600 and the weight vector calculated from the weight determination unit 700 are fused to the final classification of the target object. Determining a class, for example, may be a step of determining a final mastery level or a final mastery class of the target object. The final determination unit 800 may be implemented through a classification software algorithm and a processor on which the software is executed.

In an exemplary embodiment, the final determination unit 800 includes the first individual determination unit 610, the second individual determination unit 620, the first weight determination unit 710, and the second weight determination unit 720, respectively. The final reference data level can be determined by using the output vector as an input. For example, the final determination unit 800 does not perform one-hot encoding on the output vector of each deep learning stream of the first individual determination unit 610 and the second individual determination unit 620. It can be used as it is in the form of the original probability distribution.

The final discrimination unit 800 does not consider only the output vector at either the first or the second viewpoint in order to comprehensively consider the color distribution pattern of the maturity stage that continuously develops over the entire area of the target object. The score vector and the weight vector at both the viewpoint, that is, the first viewpoint and the second viewpoint, are fused. The conventional maturity classification technique simply selects an indexer having a maximum value among elements in a vector. In this case, if the determined maturity is misclassified, the variance of the scores for each maturity level of the deep learning stream output vector tends to increase. This may be due to the fact that the characteristics related to the maturity of the crop embedded in the individual viewpoint images are not clear enough to accurately determine maturity.

The present invention is to minimize such misclassification, and in the process of determining the maturity of a target object, for example, a crop, rather than using only a single viewpoint image, the final maturity is considered by considering all the color distribution patterns of the maturity stages that continuously develop over the entire object area. Can be considered.

In addition, weights can be calculated together to reflect whether the probability-based score vector based on which stream is a result closer to the facts among the deep learning streams. For example, a tomato variety may have a high probability of starting a color distribution pattern change from near the apex, and any other tomato variety may have a high probability of starting a color distribution pattern change from a navel.

For another example, in the case of tomato crops, depending on the light direction or temperature (corresponding to human body temperature), it may turn red from the stem-end or red from the flower-end, depending on the variety. Since it may turn red in the form of a vertical band, the results of maturity classification may vary depending on the time point.

In this case, the fact that the degree to which the probability-based score vector at each time point is reflected in the final maturity level determination in the two different breeds or two different cases may be different, resulting in a result closer to the fact. To this end, the present invention can achieve the above object by calculating and determining the weight vector as described above and reflecting the same in the final reference data determination.

의 최대 스코어를 갖는 클래스 및

의 최대 스코어를 갖는 클래스를 활용하여 가중치 벡터를 결정하기 때문에 그렇지 않은 경우에 비해 분류에 대한 신뢰도를 향상시킬 수 있다.In particular, in the step of determining the weight (S700), unlike the direct use of the predetermined weight based on the learning result, the present invention is a result derived from the individual determination unit 600 in the process of classifying the actual crop, for example

Class with a maximum score of and

Since the weight vector is determined by using the class with the maximum score of, the reliability for classification can be improved compared to the case where it is not.

In an exemplary embodiment, the final determination unit 800 may determine the final reference data level through Equation 10 below.

<Equation 10>

는 최종 결정 융합 벡터로서, 최종 기준데이터 레벨을 의미한다. 또, N은 각 개별 시점의 총 개수를 의미하는 것으로 2 이상의 정수이다. 즉, 도 7 및 도 8 등과 같이 개별 시점이 2개인 경우, N은 2이다. 또한 n은 각 개별 시점의 순번 내지는 특정 시점을 의미하는 것으로 1 이상 N 이하의 정수이다. here,

Denotes the final decision fusion vector, and denotes the final reference data level. In addition, N refers to the total number of each individual time point, and is an integer of 2 or more. That is, when there are two separate viewpoints as shown in FIGS. 7 and 8, N is 2. In addition, n is an integer of 1 or more and N or less, which means the order or specific time of each individual time point.

은 시점 n에서의 가중치로서 벡터(예컨대, 열벡터)일 수 있다. 예시적인 실시예에서,

은 전술한 가중치 결정부(700)로부터 연산 또는 결정될 수 있다. 구체적으로,

은 제1 시점에 관한 제1 가중치 결정부(710)로부터 도출되고,

는 제2 시점에 관한 제2 가중치 결정부(720)로부터 도출될 수 있으나 본 발명이 이에 제한되는 것은 아니다.

May be a vector (eg, a column vector) as a weight at time n. In an exemplary embodiment,

May be calculated or determined from the above-described weight determination unit 700. Specifically,

Is derived from the first weight determining unit 710 for the first viewpoint,

May be derived from the second weight determining unit 720 for the second viewpoint, but the present invention is not limited thereto.

는 시점 n에서의 확률 기반 스코어 벡터로서, 전술한 개별 판별부(600)로부터 출력될 수 있다. 구체적으로,

은 제1 시점에 관한 제1 개별 판별부(610)로부터 출력되고,

는 제2 시점에 관한 제2 개별 판별부(620)로부터 출력될 수 있으나 본 발명이 이에 제한되는 것은 아니다.In addition,

Is a probability-based score vector at time n, and may be output from the individual determination unit 600 described above. Specifically,

Is output from the first individual determination unit 610 for the first time point,

May be output from the second individual determination unit 620 for the second viewpoint, but the present invention is not limited thereto.

는 성분별 곱(element wise multiplication)을 의미한다.And

Means element-wise multiplication.

이고,

이며,

이고,

인 경우

이고, 최종 판별부(800)는 숙도 단계를 최대 스코어를 갖는 클래스 1, 즉 1단계로 분류할 수 있다.As in the above example

ego,

Is,

ego,

If

And, the final determination unit 800 may classify the maturity stage into a class 1 having a maximum score, that is, a first stage.

는 하기 수식 9와 같이 정의될 수 있다.As another non-limiting example, the vector output from the final determination unit 800

May be defined as in Equation 9 below.

<Equation 11>

의 원소 각각은 대상 객체, 즉 작물의 단계별 우도(likelihood)로서 이산 확률 분포로 나타낼 수 있다. 또, 최종 판별부(800)의 출력 벡터로서 최종적인 숙도 단계를 판별하기 위해 하기 수식 12를 사용할 수 있다.vector

Each of the elements of can be expressed as a discrete probability distribution as the target object, that is, the likelihood of each crop. In addition, the following Equation 12 may be used to determine the final maturity stage as an output vector of the final determination unit 800.

<Equation 12>

의 원소 중 가장 큰 확률 값의 인덱스 값 내지는 클래스 레이블로서 1 이상 M 이하의 값을 가질 수 있다.Here, y is a final maturity level discrimination value determined from input images of multiple views for one target object, and the pick function is a function that returns the m-th value in the list, where m is an integer between 1 and M. Again, the output vector

The index value of the largest probability value or the class label among the elements of may have a value of 1 or more and M or less.

As described above, according to the final determination unit 800 of the present invention, in the process for determining the final maturity for the multi-view video input of the target object, even if there is an error in some of the maturity classification results at individual points of view, the correctly classified result Classification accuracy can be improved by correcting it.

In addition, different weight vectors are assigned at each individual time point, but by learning the weights reflecting the accuracy calculated from the error matrix, the importance or priority at each time point can be given differently. Among specific varieties, it is possible to improve the accuracy of classification of crops grown under specific conditions. Moreover, when the first error matrix from the first view point and the second error matrix from the second view point are not reflected in the weight as a whole, but are classified into a specific mastery level among the score vectors derived from the individual discrimination unit 600. By summing only the column vectors to derive the weight vector, the importance or priority at each time point can be more accurately reflected.

The crop classification method according to the present invention, a program therefor, a recording medium in which the program is recorded, and a crop classification device may be applied to various crops such as tomatoes, peaches, apples, pears, paprika or peppers.

In addition, the present invention may be implemented by a computer-readable recording medium in which a program for executing the above-described method for sorting crops is recorded. In addition, the crop classification apparatus according to the present invention includes an input interface for inputting crop images, a memory, and a processor, and a program or software may be stored in the memory and executed by the processor.

The embodiments of the present invention have been described above, but these are only examples and do not limit the present invention, and those of ordinary skill in the field to which the present invention pertains, within the scope not departing from the essential characteristics of the embodiments of the present invention. It will be appreciated that various modifications and applications not illustrated above are possible.

Therefore, the scope of the present invention should be understood to include modifications, equivalents, or substitutes of the technical idea exemplified above. For example, each component specifically shown in the embodiment of the present invention can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present invention defined in the appended claims.

In addition, each component on the configuration diagram or block diagram of the accompanying drawings may be implemented by software or hardware such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). However, each component in the configuration diagram or block diagram may be implemented in an addressable storage medium as well as software and hardware, and may be configured to execute one or more processors.

Each component on the configuration diagram or block diagram may mean a module, segment, or part of code including one or more executable instructions for executing a specified logical function. Therefore, the functions provided by the components on the configuration diagram or block diagram may be implemented by a plurality of more subdivided components, or a plurality of components on the configuration diagram or block diagram may be implemented by one integrated component. Of course.

That is, one or more components may be selectively combined and operated within the scope of the object of the present invention. In addition, all components can be implemented as one independent hardware, and some or all of the components are selectively combined to have a program module that performs some or all functions combined in one or a plurality of hardware. It can also be implemented as a computer program. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may be stored in a computer-readable recording medium or a storage medium, and read and executed by a computer, thereby implementing an embodiment of the present invention. Examples of the recording medium include magnetic recording media, optical recording media, and the like.

100: video input unit
110: first image input unit
120: second image input unit
200: individual discrimination unit
210: first individual discrimination unit
220: second individual discrimination unit
300: learning vector operator
310: first learning vector operator
320: second learning vector operator
400: learning vector storage

Claims (9)

Receiving a plurality of viewpoint images each including a first viewpoint and a second viewpoint of the target object;
Classification of reference data at the first and second viewpoints using an object detection-based deep learning model, and a probability-based first score vector for the first viewpoint and a probability-based second score for the second viewpoint Outputting a vector;
Deriving a first error matrix and a second error matrix from the first score vector and the second score vector, respectively, wherein the first error matrix and the second error matrix are a horizontal axis, a prediction value, and a vertical axis. Deriving a first error matrix and a second error matrix each having a matrix of M × M and a matrix having a true value;
A first step of outputting and storing the first vector containing the M vector has M elements from a first error matrix,

,

,… ,

Outputting and storing a first vector consisting of; And
A first step of outputting and storing the second vector containing the M vector has M elements from the two-error matrix,

,

,… ,

Including the step of outputting and storing a second vector consisting of,
The first vector and the second vector each includes a column vector (column vector) having M elements of the crop classification learning method. delete

,

,… ,

A first column vector consisting of and

,

,… ,

As a method for classifying crops by defining a second heat vector consisting of,
Receiving a plurality of viewpoint images each including a first viewpoint and a second viewpoint of the target object;
Outputting a probability-based first score vector for a first viewpoint using a deep learning model, the method comprising: outputting a first score vector whose maximum score is an i-th class (i is a positive integer less than or equal to M);
Outputting a second score vector based on probability for a second viewpoint using a deep learning model, comprising: outputting a second score vector whose maximum score is a j-th class (j is a positive integer less than or equal to M);
Including a first weight vector and a second weight vector for the first view and the second view based on the plurality of first vectors and the plurality of second vectors, and the first score vector and the second score vector Determining a plurality of weight vectors; And
And determining a final reference data level based on the plurality of score vectors and the plurality of weight vectors. The method of claim 3,
The first weight vector for the first viewpoint is determined using Equation A below, and the second weight vector for the second viewpoint is determined using Equation B below.
<Equation A>

<Equation B>

The method of claim 4,
remind

And

(Here, i and j are each a positive integer less than or equal to M ) is a method for classifying crops that is a column vector having a maximum score in the i-th and j-th classes, respectively. The method of claim 4,
Classification method of crops satisfying the following formula.

(here,

Is the weight vector for time n) The method of claim 4,
The step of determining the final reference data level is a method of classifying crops performed by including an algorithm using the following equation.

(here,

Is the final decision fusion vector, N means the total number of each individual time point, n means the order of each individual time point,

Is the probability-based score vector at time n,

Is the weight vector for time n,

Means element-wise multiplication) An image input unit receiving a plurality of viewpoint images of the target object, respectively;
An individual discrimination unit for classifying reference data at each viewpoint of the object using an object detection-based deep learning model and outputting a plurality of probability-based score vectors for each viewpoint regarding a reference data level for the object;
A weight determining unit determining a plurality of weight vectors at each viewpoint based on a plurality of column vectors derived from an error matrix of the probability-based score vector and a maximum score of the probability-based score vector; And
A final determination unit for determining a final reference data level based on the plurality of probability-based score vectors and the plurality of weight vectors,
The error matrix includes a first error matrix and a second error matrix derived from score vectors in a plurality of view images, and the first error matrix and the second error matrix each have a horizontal axis as a predicted value and a vertical axis as an actual value. An apparatus for classifying crops having M x M matrices, wherein a first column vector derived from the first error matrix and a second column vector derived from the second error matrix each have M elements. A computer-readable recording medium on which a program for performing the method according to claim 3 is recorded.

Images