KR20240032288A - Apparatus and method for constructing a hybrid green tea learning dataset consisting of a quantitative green tea learning dataset generation module and a qualitative green tea learning dataset generation module - Google Patents

Abstract

In the present invention, the problem of not satisfying the leap into the global industry and the diverse consumption of domestic tea lovers due to the tendency not to scientifically approach the quality of tea and its enjoyment, and the existing green tea quality evaluation methods are used to standardize the quality of green tea products. The problem is that it is an obstacle to the development of the domestic tea industry as it has not been selected as a recognized and standardized quality evaluation method for evaluation for grading, and the variables that determine the taste of green tea have so far only been identified through visual inspection by experts and after drinking. In order to improve the problem of not being able to provide customized tea to various consumers with the current method due to the absence of accumulated data and the existence of deviations among experts depending on the analog method, a qualitative green tea learning data set creation module (100 ), By constructing the quantitative green tea learning data set creation module 200, it is possible to provide qualitative green tea learning data sets and quantitative green tea learning data sets related to green tea taste and quality that have not existed before, and can be used for smart apps and artificial intelligence deep learning. It can be easily applied to green tea-related promotion, distribution, and export system construction, and is raised as one of the reasons for the low global market share compared to the scale of domestic tea consumption and the high-quality tea produced in Boseong, Jeollanam-do and Halla, Jeju. It overcomes the limitations of qualitative methods for quality control, and as basic data necessary for the scientific tea production process and distribution, its utilization can be increased by 1.5 to 3 times compared to the existing one, and it is possible to increase the use of tea to consumers who actually enjoy tea. By adding consumers' taste spectrum for low, medium, and high quality green teas to the quantitative green tea learning data set, we can provide green tea taste and quality with realism and liveliness, increasing green tea consumption by 80% compared to before. When evaluating the quality of green tea based on taste, aroma, tea color, appearance of dried tea leaves, and shape of dried leaves, the priorities and weights that green tea consumers consider important for each evaluation factor are objectively and systematically set. , A hybrid green tea learning data set construction device and method consisting of a quantitative green tea learning data set creation module and a qualitative green tea learning data set creation module that can create a new consumer-oriented green tea consumption market rather than a supplier-oriented green tea consumption market. The purpose is to provide.

Description

Apparatus and method for constructing a hybrid green tea learning dataset consisting of a quantitative green tea learning dataset generation module and a qualitative green tea learning dataset generation module}

The present invention is applied to building a green tea-related promotion, distribution, and export system linked to smart apps and artificial intelligence deep learning. More specifically, it collects green tea learning data from green tea-related web and app sites, and collects the collected data. Based on data, a qualitative green tea learning dataset on green tea taste and quality is created, and a quantitative green tea learning dataset is created based on tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data. This relates to a hybrid green tea learning data set construction device and method consisting of a quantitative green tea learning data set creation module and a qualitative green tea learning data set creation module.

In Korea, green tea and black tea are in the spotlight as drinking products enjoyed by people of all ages.

On the other hand, the green tea industry, which is enjoyed for health and preference (taste, aroma, etc.), is facing limitations in growing globally beyond the domestic market.

In particular, compared to the tastes and high level of interest of producers and consumers in green tea, there is a lack of consensus on its quality or common quality.

Accordingly, due to the tendency to not approach the quality and enjoyment of tea scientifically, it has not been able to satisfy the leap into the global industry and the diverse consumption of domestic tea lovers.

In addition, existing green tea quality evaluation methods are becoming an obstacle to the development of the domestic tea industry as they have not been selected as a recognized and standardized quality evaluation method for evaluating the quality of green tea products for standardization or grading.

In addition, because the variables that determine the taste of green tea have so far only relied on experts' visual observation and differentiation after drinking, there is no accumulated data, and there are differences among experts depending on the analog method, so the current method is not suitable for various consumers. There were limitations in providing customized cars to customers.

1. Domestic Patent Publication No. 10-2321735 (2021.10.29) 2. Domestic Patent Publication No. 10-1949154 (2019.02.12)

In order to solve the above problems, the present invention can provide a qualitative green tea learning data set and a quantitative green tea learning data set regarding the taste and quality of green tea that do not exist before, and the size of domestic tea consumption and the tea produced in Boseong, Jeollanam-do and Halla, Jeju. It overcomes the limitations of qualitative methods for tea quality control, which are raised as one of the reasons for the low global market share compared to high-quality tea, and increases utilization as basic data necessary for the scientific tea production process and distribution, and enables actual tea By adding consumers' taste spectrum for low, medium, and high green tea, which classifies the quality of tea for consumers, to the quantitative green tea learning data set, we can provide green tea taste and quality with a sense of realism and liveliness. , a quantitative green tea learning data set that can objectively and systematically set the priorities and weights that green tea consumers value for each evaluation factor when evaluating the quality of green tea based on aroma, tea color, appearance of dried tea leaves, and shape of fresh leaves. The purpose is to provide a hybrid green tea learning data set construction device and method consisting of a generation module and a qualitative green tea learning data set generation module.

In order to achieve the above objective, a hybrid green tea learning data set construction device consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module according to the present invention is provided.

A qualitative green tea learning data set generation module 100 that collects green tea learning data, labels the collected data, increases the amount of data, and generates a qualitative green tea learning data set through verification;

This is achieved by being composed of a quantitative green tea learning data set generation module 200 that generates a quantitative green tea learning data set based on tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data.

As described above, in the present invention

First, it can provide qualitative green tea learning data sets and quantitative green tea learning data sets on green tea taste and quality that have never existed before, making it easy to build green tea-related promotion, distribution, and export systems linked to smart apps and artificial intelligence deep learning. It can be applied and applied.

Second, overcome the limitations of qualitative methods for tea quality control, which is one of the reasons for the low global market share compared to the scale of domestic tea consumption and high-quality tea produced in Boseong, Jeollanam-do and Halla, Jeju, and develop a scientific tea production process and distribution. As basic data needed, its usability can be increased by 1.5 to 3 times compared to existing data.

Third, by adding consumers' taste spectrum for low, medium, and high green tea, which classifies tea quality for consumers who actually enjoy tea, to the quantitative green tea learning dataset, we provide green tea taste and quality with a sense of realism and liveliness. This can increase green tea consumption by 80% compared to before.

Fourth, when evaluating the quality of green tea based on taste, aroma, tea color, appearance of dried tea leaves, and shape of fresh leaves, the priorities and weights that green tea consumers consider important for each evaluation factor are objectively and systematically set, and supplier-centered It is possible to create a new consumer-centered green tea consumption market, rather than the green tea consumption market of .

Hereinafter, a preferred embodiment according to the present invention will be described with accompanying drawings.

Figure 1 is a block diagram showing the components of a hybrid green tea learning data set construction device 1 consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module according to the present invention, and Figure 2 It relates to a configuration diagram showing a hybrid green tea learning data set construction device (1) consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module according to the present invention, which is a qualitative green tea learning data set. It consists of a generation module (100) and a quantitative green tea learning data set generation module (200).

First, the qualitative green tea learning data set generation module 100 according to the present invention will be described.

The qualitative green tea learning data set generation module 100 collects green tea learning data, labels the collected data, increases the amount of data, and generates a qualitative green tea learning data set through verification. It plays a role.

It consists of data that has attributes in the form of categories or orders.

Additionally, in order to create a green tea taste/quality recognition model, green tea taste/quality image data is needed.

In the present invention, green tea learning data is collected from green tea-related web and app sites to construct green tea taste and quality images into a learning data set containing detailed information on green tea taste, green tea quality, tea leaf harvest time, green tea cultivation area, tea type, and green tea product. A qualitative green tea learning dataset is constructed to collect and create an optimal green tea taste and quality recognition recognition model.

The qualitative green tea learning data set generation module 100 is largely divided into a Green Tea Images Annotation Tool type GUI program engine unit 110 and an augmentation unit 120, as shown in FIG. 3. , It consists of a qualitative green tea learning data set generation control unit 130.

First, the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit 110 according to the present invention will be described.

The GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit 110 collects green tea learning data, labels the collected data, and runs the overall task of configuring the learning data in one program. It plays a role in forming learning data about green tea.

First, since you need to know information about the learning data to recognize green tea taste and quality image objects, you need to know the annotation format like existing programs.

Additionally, the annotation step can only be performed if there is green tea taste and quality image data.

However, it takes a lot of time to individually collect images of the taste and quality of domestic green tea.

Therefore, in the present invention, the overall work to construct a qualitative green tea learning data set is performed, including collecting green tea learning data, labeling the collected data, and an augmentation function to increase the amount of data. It consists of a GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit, which is a GUI (Graphic User Interface) program that can be done in one program.

The GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit is additionally configured with a crawling module and semi-automated object recognition function that are not present in other labeling programs.

In addition, in order to implement a crawling module for data collection, the Requests module is used based on Selenium to retrieve the desired green tea taste and quality image.

In addition, Tensorflow (111) and Keras-RetinaNet module (112) are configured to recognize green tea learning data for semi-automated labeling work.

The Tensorflow 111 serves as an open source software library for data flow programming for various tasks.

The Keras-RetinaNet module 112 loads a pre-trained coco model and performs object detection using it. This uses the characteristics of the pre-trained model as the initial value of the weight and performs transfer learning to create a new model using the learning data.

The interface of the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit consists of the Tkinter unit built into Python by default.

In other words, in order to form the interface window and operation buttons that make up the GTIAT (Green Tea Images Annotation Tool) type GUI program engine part, a function (command= It is composed of the form self.module).

As shown in FIG. 4, the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit 110 includes a crawling/scraping module 113 and a Selenium module 114.

The crawling/scraping module 113 serves to collect data desired by the user from web/app pages.

This is mainly done by Python-based libraries.

This is configured by selecting either the requests module or BeautifulSoup.

The Selenium module 114 serves to collect data on the web/app browser by utilizing the driver of the web/app browser.

This makes it easy to access in-depth data.

Originally, Selenium was a module for testing web and app browsers, but as interest in data collection gradually grew, it was used as a data collection module.

This is provided in the form of a Python library, and is installed in the environment through the above command.

Also, if you have installed the library, access the browser's site and install the driver.

The present invention is configured to use the Chrome driver.

The driver is provided as an .exe file. Place the driver file in the environment you are using.

Once you have completed configuring the environment to collect data through Selenium, use Phthon to configure functions that allow you to connect to web and app sites with images of the desired green tea taste and quality and access the data.

Before collecting data, be sure to check whether the page of the green tea-related web or app site can be crawled and scraped.

For example, "http://www.example.com/robots.txt"

If you connect by adding '/robots.txt' to the web/app page address, you will see the allowable range for collecting data on the web/app page.

The Disallow part refers to the part of the web/app site where crawling and scraping are not allowed.

After checking the crawling range, we retrieve green tea taste and quality images from green tea-related websites and app sites.

By identifying and entering the HTML and CSS tags and properties that the web/app page consists of in Selenium's 'find_element(s)' function, you can access the desired data.

Since the green tea taste and quality images of high-resolution green tea on green tea-related web and app sites are located in a complex structure, identify and input the html and CSS elements where the desired green tea taste and quality image data is located.

The 'execute_script' function allows you to move to the last page if the web/app page consists of multiple pages.

In the present invention, it is a green tea-related web/app site, where users can purchase green tea by period, region, and customer taste evaluation on famous domestic sites ('Green Tea World', 'Green Tea Nara', 'Seon Dawon', and 'Daehan Dawon'). It is designed to collect images of green tea taste and quality.

By linking with the 'Button' function of the widget in the Tkinter section, the GTIAT program provides images of the taste and quality of green tea currently being sold, or customers It is designed to provide an image of the taste and quality of green tea evaluated by taste.

In order to collect green tea taste and quality image data, we implemented a module targeting green tea-related web and app sites, but in order to allow users to also retrieve green tea taste and quality image data from other web and app sites, we used a keyword search method. It is configured to download and link the Crawler module.

The keyword search crawler is the icrawler library, and if installed in the appropriate environment, you can search for green tea taste and quality images from Google, Baidu, Bing, and Flickr.

In addition, the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit 110 is configured to include a search type crawler in conjunction with a widget to enable Google green tea taste and quality image search.

Second, the augmentation unit 120 according to the present invention will be described.

The augmentation unit 120 increases the amount of green tea taste and quality image data in order to increase the learning effect in addition to the learning data about green tea formed through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit. It plays a role.

This is configured to provide various effects on green tea taste and quality images by utilizing OpenCV and various APIs.

Transformation that rotates or enlarges the original green tea taste and quality image is called traditional augmentation, and it shows good performance in classification problems.

In other words, in order to avoid the green tea taste and quality image being significantly outside the image, it is configured to convert 0, 15, 30, 150, 165, 180, 195, 210, 330, and 345 degrees.

The empty space created by rotation is filled with 0. This transformation increases the size of the qualitative green tea learning dataset by 10 times.

The application of green tea taste and quality image augmentation is structured within the program so that users can directly apply the augmentation effect to the green tea taste and quality images collected through the crawling module in the GTIAT program.

The reason for augmenting green tea taste and quality image data is to further increase the performance of the learning model by inflating the amount of data.

In fact, in the performance evaluation of the VGG model, the error rate was reduced to 3-5% even when the same photo was input in a different size.

This is a concept that is added from the source of data, and in particular, it can improve performance as it has the effect of providing patterns for green tea taste and quality images.

In particular, general augmentation, such as rotation or enlargement of the original data, maximizes performance in simple classification.

After collecting from the green tea-related web and app site crawling module that implemented the qualitative green tea learning data set and performing augmentation, the storage structure of the green tea taste and quality image data is as shown in figure.

First, the green tea taste and quality images collected from the green tea-related web/app site folder selected by the user in the GTIAT program are saved.

At this time, when green tea is saved according to the name of product, period, region, fermentation, and manufacturing process, it is composed of 4 to 20 folders.

The 'Thumbnail' folder stores small-sized green tea taste and quality image data as thumbnail green tea taste and quality images on green tea-related web and app sites.

The 'Big' folder is the same type of green tea taste/quality image as the thumbnail green tea taste/quality image, but is stored as a green tea taste/quality image with much higher resolution.

The 'Detail' folder connects to the detailed page where green tea is posted on the green tea sales site and stores high-resolution images of the taste and quality of green tea in a different direction and form than the thumbnail.

Finally, the 'Aug' folder randomly applies various augmentation effects to the green tea taste and quality images in the 'Big' folder.

Third, the qualitative green tea learning data set generation control unit 130 according to the present invention will be described.

The qualitative green tea learning data set generation control unit 130 includes learning data about green tea formed through a GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit and green tea taste and quality images increased through an augmentation unit. After verifying the data, it controls it to generate a qualitative green tea learning dataset.

The qualitative green tea learning data set is a collection of data that matches the contents of one database table or one statistical data matrix.

It consists of columns and rows.

The column represents a specific variable, and the row is configured to match a given member.

Then, the values of each variable are listed, and each value is called data.

It constitutes data for one or more members and matches the number of rows.

The data set is a set of data objects, and the data objects include records, points, vectors, patterns, cases, It consists of events, samples, observations, entities, etc.

In the present invention, the green tea taste/quality object and the green tea object in an abnormal state are set as data objects.

The data object consists of several attributes, and the attribute refers to a characteristic or feature that can define differences between data objects.

Attributes are called variables, characteristics, fields, features, dimensions, etc.

The qualitative green tea learning dataset is configured by selecting one of “original data”, “original data + rotation, crop, gray-scale”, and “original data + Noise + Weather”.

And, the original data consists of a column, 'folder' indicating the folder name, 'filename' indicating the original data name, 'source' indicating version information, and 'size' indicating video size information, 'header' representing scenario information, 'event' representing event information, 'object' representing green tea taste/quality object, 'database' representing the database version name, and 'annotation' representing the name of the annotation structure. and ‘width’, which represents the width value of the image or video, ‘height’, which represents the height value of the image or video, ‘depth’ which represents the channel value of the image or video, and ‘duration’ which represents the number of frames per second. , 'frames' indicating the total number of frames, 'location' indicating the cultivation place and location of green tea, 'weather' indicating the weather, 'time' indicating the time zone, and disease name indicating abnormal condition of green tea. 'eventname', 'objectname' indicating the green tea object in an abnormal state, 'position' indicating the location of the green tea in an abnormal state, 'positionframe' indicating the location information of the green tea object, and the location of the green tea object in the position frame. 'keypoint' indicating the position, 'x' indicating the x-axis coordinate of the keypoint, 'y' indicating the y-axis coordinate of the keypoint, 'start' indicating the start point of the frame, and ' It consists of ‘end’.

The qualitative green tea learning data set generation control unit requires labeling in order to learn the stored green tea taste and quality image learning data into the network.

As shown in FIG. 6, the qualitative green tea learning data set generation control unit 130 includes an annotation control unit 131 for green tea taste and quality recognition, and a RetinaNet-type green tea taste and quality recognition control unit 132. ), an annotation format forming control unit 133, and a qualitative green tea learning data verification control unit 134.

[Annotation control unit (131) for green tea taste and quality recognition]

The annotation control unit 131 for green tea taste and quality recognition forms a bounding box shape for the green tea taste and quality image data and then controls it to generate an annotation.

This is done in the form of Pascal VOC or YOLO format.

Especially in the case of a large amount of data, it consumes a lot of labor and time, and mistakes may occur during the annotation process, resulting in incorrect data.

Accordingly, the present invention is configured to recognize green tea taste and quality image data through a green tea recognition network and then automatically store annotation information appropriate for the network format.

[RetinaNet type green tea taste and quality recognition control unit (132)]

The RetinaNet-type green tea taste and quality recognition control unit 132 converts the weight file learned through RetinaNet into hdf5 format that can handle large volumes, and then recognizes the green tea taste and quality. It plays a role in controlling the

This is the hdf5 format developed as a tool for storing large amounts of complex data.

This format is possible when you want to use the system platform independently, and it is capable of processing large amounts of complex data.

You can handle this hdf5 format by installing the h5py module in Python.

In other words, the green tea taste and quality image data set is constructed through hdf5 format.

Using this format, which is capable of processing large amounts of complex data, Python provides Keras-based RetinaNet as a module.

Additionally, a Convert module is constructed that converts the weight file learned through Retina Net into hdf5 format that can handle large amounts of data.

Additionally, if you learn the desired data with Retina Net and input the weight file into the given conversion module, it will be converted to hdf5 file format.

Then, the learning results are read through the Tensorflow-based load model (load_mode1()) function, and if the reading function is included in the Tkinter program, the green tea recognition network is linked to the labeling tool. This happens.

The RetinaNet uses ResNet as a backbone network and configures two task-specific subnetworks, which have a one-stage structure. It consists of

Here, the ResNet seeks to know the overall features of the first input green tea taste and quality image, and uses FPN as the structure, using a multi-scale convolutional feature pyramid. Create a Feature Pyramid.

For reference, when detecting green tea taste/quality images of various sizes, FPN is configured to find objects while resizing the efficient calculation of green tea taste/quality image size, using a top-down method. Extract features.

Here, top-down feature extraction extracts features through the feature map at each level, but by reusing the features calculated at the higher level, feature extraction at the lower level extracts multi-scale features. Enables efficient extraction.

Because of this, the detection time for an object can also be shortened.

Looking at the structure of RetinaNet, one role is to classify the class of the anchor box, and the other role is to predict the distance between the anchor box and ground-truth object boxes.

In Equation 1, the y term is a value of 1 or -1 and indicates the ground truth class, and the p term means the class probability predicted by the model for the class with 'y=1', so it is 0 or 1. It has the value of

And, if the terms are organized as in Equation 2, it corresponds to the top graph among the graphs shown in FIG. 7. So, if the loss becomes large, it can become a problem as it overwhelms the losses of low (rare) classes.

The most common way to address class imbalance is to use a weight alpha (α) ranging from 0 to 1.

Alpha (α) is applied to class 1 and (1-α) to class -1.

In Equation 3, alpha (t) for class t is defined by the α balance CE loss equation as follows.

However, if the imbalance problem becomes serious during the learning process of the class dense detector, the CE Loss overwhelming phenomenon occurs.

This will have a huge impact on the gradient.

In Equation 3, alpha (α) serves as a balance term that adjusts the importance of positive and negative samples to be balanced.

At this time, as shown in Equation 4, the balance term (1-pt)γ was added to the alpha term, that is, the cross entropy loss.

When Pt is a small value, as the balance term approaches 1, the value of the term becomes almost 0, so ultimately, it does not affect the loss function.

[Annotation format forming control unit (133)]

The annotation format forming control unit 133 recognizes green tea by linking the RetinaNet learned from the COCO dataset with the Tkinter GUI program, forms it in the form of a bounding box, and forms the annotation format. It plays a role in generating and controlling the qualitative green tea learning data set.

Naturally, in order to perform the annotation process, coordinate information must be stored in the recognized object.

In the coordinate information, absolute coordinates in the form of Pascal VOC and coordinates in the YOLO method to be learned to create a domestic vehicle classifier must be calculated.

First, in order to define the coordinates recognized by Retina Net, predicted values such as bounding box, confidence score, and label recognized through the 'predict_on_batch()' function of the Keras function are stored in Python's 'zip( )' function to group them into a single sequence and then put them in a list so that coordinate information can be obtained.

Here, the coordinate method of the YOLO model represents a ratio and corresponds to a decimal point, so the coordinates of the bounding box are defined in the form of a float.

After defining the bounding box in the order of x1, y1, x2, and y2, coordinate information is stored in each predefined list.

Since it is an absolute value coordinate format in Pascal VOC format, all you have to do is convert the defined float form to an integer form at the end.

Appropriate annotation information for the YOLO model is defined as the ratio of the width and height from the midpoint coordinates of the bounding box x,y and the midpoint coordinates w,h, so it is configured to apply a conversion formula according to the format.

Then, it is returned in the form of a tuple and is configured to store annotation information suitable for the YOLO format.

In addition, annotation information is additionally stored, and is configured to store annotation information in accordance with the format of RetinaNet.

RetinaNet's annotation information is stored as absolute coordinates, but annotation information for all data sets is included in one .CSV file, and the file path where the data is located is additionally entered into the annotation information.

When generating quantitative green tea learning data, the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit 110 according to the present invention provides objects in Pascal VOC format, YOLO format, and RetinaNet model format. Annotation information is stored in three ways to provide users with a variety of network training possibilities.

[Qualitative green tea learning data verification control unit (134)]

The qualitative green tea learning data verification control unit 134 serves to verify and control the qualitative green tea learning data generated by the annotation format forming control unit.

This evaluates how well learning is done when learning, and consists of a verification dataset (Development Dataset).

The development dataset is used to evaluate the performance of the learned model during training, and the results are reflected in the parameters.

The purpose of the development dataset is to tune the parameters among the parameters learned from the training data so that the optimal parameters can be found to fit well with the actual data.

In the present invention, as shown in Figure 8, the qualitative green tea learning data set and the verification data set are configured so as not to overlap, and are configured at a ratio of 7:3 or 8:2.

Next, the quantitative green tea learning data set generation module 200 according to the present invention will be described.

The quantitative green tea learning data set generation module 200 serves to generate a quantitative green tea learning data set based on tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data.

Here, quantitative refers to quantitative data, where observed values have numerical properties.

As shown in FIG. 9, it consists of a YOLO (You Look Only Once) object recognition module 210, an R-CNN (Regions with CNN features) algorithm module 220, and a quantitative green tea learning dataset generation control unit 230. do.

First, the YOLO (You Look Only Once) object recognition module 210 according to the present invention will be described.

The YOLO (You Look Only Once) object recognition module 210 recognizes green tea taste and quality element objects present in the green tea taste and quality image from tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data. , It plays a role in recognizing the green tea taste/quality element object by learning the location of the corresponding green tea taste/quality element object.

It is configured to learn more than 9,000 classes that combine green tea taste and quality Imagenet and COCO data sets extracted from tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data.

The YOLO (You Look Only Once) object recognition module consists of a single pipeline network structure.

And, for quick processing, the original green tea taste and quality image is divided into S × S pieces in the form of a grid of the same size.

The more confident (Confidence Score) there is a green tea taste/quality element object inside each grid, the more bounding boxes are created.

Grids that are not included in the bounding box are deleted by determining that there are no green tea taste/quality element objects through the midpoint of the grid.

Additionally, each grid cell is configured to display in color which class the object within the Proposal Bounding Box in the area belongs to.

The YOLO (You Look Only Once) object recognition module is configured to predict the number of bounding boxes designated as a predefined shape and calculate a confidence score based on this.

Here, the designated bounding box is like an anchor box, and has an important influence on the YOLO model in the learning problem and contributes to high accuracy.

Second, the R-CNN (Regions with CNN features) algorithm module 220 according to the present invention will be described.

The R-CNN (Regions with CNN features) algorithm module 220 derives regions in which green tea objects are estimated to be present among recognized green tea objects through the YOLO (You Look Only Once) object recognition module, and in each region After deriving CNN characteristics, each region plays a role in classifying classes related to green tea taste and quality.

As shown in Figure 10, it is composed of a selective search unit 221 and a convolutional neural network unit 222.

The Selective Search unit 221 does not distinguish objects from tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data, but similar texture, color, and search results from green tea taste and quality images. Using a selective search method that connects adjacent pixels with intensity, it searches for candidate areas among green tea objects that are likely to contain green tea taste and quality elements.

The tea leaf data by period refers to data about tea leaves by period.

The above data for tea quality experts refers to data evaluated by tea quality experts according to tea leaves and types by period.

Here, tea includes both green tea and black tea.

The consumer preference spectrum data refers to data on various preference spectra that have been directly tasted and evaluated by consumer discriminators for each quality (low, medium, high) determined by a tea quality expert.

The tea production area data refers to data about the tea production area.

This consists of green tea quality (high, middle, low) classification data and black tea quality (high, middle, low) classification data by tea production area (by amount of sunlight, temperature deviation, etc.).

During the production process of green tea and black tea, the production location, climate, daily temperature range, precipitation such as dry and rainy seasons, and frequency of fine dust are affected. The attribute is defined operationally as 1 for the first leaf of the year, 2 for the second tea leaf, and 3 for the third tea leaf in the year (e.g., the first young tea leaf usually tastes better). value is given), green tea harvesting period (there are cases where it is divided into 4 and subdivided into seasons, etc.), and finally, tea leaf color (range and density of light green and purple), leaf size, leaf width, and leaf size. The object of collection is the shape, level of fluff, etc.

Lastly, the number of tea leaves, aroma, color, and level of taste are affected during the brewing process at the stage of consuming black and green tea.

In other words, we collect a variety of data that can be commonly used in the life cycle of tea, such as the production process, identification of produced tea, and experience during the consumption process, and among them, key field values (tea leaves) that can determine the quality of tea at the production stage. Based on the color, tea harvest time, production area, shape and width of tea leaves, and level of fluff), labeling is composed of three correct answers: low, medium, and high, which is the expert's quality evaluation.

In addition, a consumer evaluation group is recruited (to ensure diversity), and data is included to evaluate the taste of tea as perceived by general consumers.

The convolutional neural network unit 222 serves to classify objects divided into each region into classes related to green tea taste and quality through a convolutional neural network.

This performs a pooling operation. Through this convolutional neural network and pooling operation, the features of the input green tea taste and quality image are extracted.

[Quantitative green tea learning data set generation control unit (230)]

The quantitative green tea learning data set generation control unit 230 verifies the classes related to green tea taste and quality classified through the R-CNN (Regions with CNN features) algorithm module and then controls it to generate a quantitative green tea learning data set. It plays a role.

It is comprised of a quantitative green tea learning data verification control unit 231.

The quantitative green tea learning data verification control unit 231 serves to verify and control the quantitative green tea learning data.

This evaluates how well learning is done when learning, and consists of a verification dataset (Development Dataset).

The development dataset is used to evaluate the performance of the learned model during training, and the results are reflected in the parameters.

The purpose of the development dataset is to tune the parameters among the parameters learned from the training data so that the optimal parameters can be found to fit well with the actual data.

In the present invention, as shown in FIG. 11, the quantitative green tea learning data set and the verification data set are configured so as not to overlap, and are configured at a ratio of 7:3 or 8:2.

Through this, the harvest time of tea, region, quality standards for green tea and black tea (e.g., dark green with a hint of purple is a low quality for green tea, whereas black tea is a commercial quality, which is a conflicting standard), color of tea, and whether or not the tea is fluffy. , Quantitative learning data that collected tea based on the weight of the tea (reference), degree and distribution of light green tea leaf color, temperature deviation and amount of sunlight in the tea production area, and classified the collected tea into high, medium, and low through experts. Three can be formed.

Hereinafter, a method for constructing a hybrid green tea learning data set consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module according to the present invention will be described.

Figure 15 is a flow chart showing a hybrid green tea learning data set construction method consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module according to the present invention.

[Step of generating a qualitative green tea learning dataset (S10)]

First, through the qualitative green tea learning data set creation module, green tea learning data is collected, the collected data is labeled, the amount of data is increased, and a qualitative green tea learning data set is created through verification. .

As shown in FIG. 16, green tea learning data is collected through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit, labeling the collected data is performed, and the overall work for configuring the learning data is performed. is run in one program to form learning data about green tea (S11).

Next, in the augmentation unit, the amount of green tea taste and quality image data is increased in addition to the learning data about green tea generated through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit (S12).

Next, in the qualitative green tea learning data set generation control unit, learning data about green tea formed through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit, and green tea taste and quality image data increased through the augmentation unit. After verifying, a qualitative green tea learning dataset is created (S13).

For this reason, as shown in Figure 13, the qualitative green tea learning data set includes classification data according to the green tea collection time and classification data according to the degree of green tea fermentation among the green tea taste and quality image data.

[Step of generating a quantitative green tea learning dataset (S20)]

Next, through the quantitative green tea learning data set creation module, a quantitative green tea learning data set is created based on tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data.

As shown in Figure 17, through the YOLO (You Look Only Once) object recognition module, the tea leaf data by period, tea quality expert data, consumer preference spectrum data, and tea production area data are used to identify the green tea taste and quality image. The green tea taste/quality element object and the location of the corresponding green tea taste/quality element object are learned to recognize the green tea taste/quality element object (S21).

Next, in the R-CNN (Regions with CNN features) algorithm module, regions where green tea objects are estimated to be present among the recognized green tea objects are derived through the YOLO (You Look Only Once) object recognition module, and CNNs are used in each region. After deriving the characteristics, each region is classified into classes regarding green tea taste and quality (S22).

Next, the quantitative green tea learning data set generation control unit verifies the classes related to green tea taste and quality classified through the R-CNN (Regions with CNN features) algorithm module, and then generates a quantitative green tea learning data set (S23).

As a result, as shown in Figure 14, the quantitative green tea learning data set includes green tea quality grade (high, middle, low) classification data, green tea dried leaf shape, flavor, hot water extract, and scoring criteria data for taste. do.

[Application stage (S30)]

Finally, as shown in Figure 12, the qualitative green tea learning data set and quantitative green tea learning data set are applied to building green tea-related promotion, distribution, and export systems linked to smart apps and artificial intelligence deep learning.

1: Hybrid green tea learning data set construction device
100: Qualitative green tea learning data set creation module
110: GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit
120: Augmentation department
130: Qualitative green tea learning data set generation control unit
200: Quantitative green tea learning data set creation module
210: YOLO (You Look Only Once) object recognition module
220: R-CNN (Regions with CNN features) algorithm module
230: Quantitative green tea learning data set generation control unit

Claims (9)

A qualitative green tea learning data set generation module 100 that collects green tea learning data, labels the collected data, increases the amount of data, and generates a qualitative green tea learning data set through verification;
Quantitative green tea learning, which consists of a quantitative green tea learning data set generation module 200 that generates a quantitative green tea learning data set based on tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data. A hybrid green tea learning dataset construction device consisting of a dataset creation module and a qualitative green tea learning dataset creation module.
The method of claim 1, wherein the qualitative green tea learning data set generation module (100)
A GTIAT (Green Tea Images Annotation Tool) type that collects green tea learning data, labels the collected data, and runs the overall task of constructing the learning data in one program to form learning data about green tea. A GUI program engine unit 110,
In order to increase the learning effect in addition to the learning data about green tea formed through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit, an augmentation unit 120 that increases the amount of green tea taste and quality image data is provided. ,
After verifying the green tea learning data formed through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine section and the increased green tea taste and quality image data through the augmentation section, a qualitative green tea learning data set was created. A hybrid green tea learning data set construction device consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module, characterized by consisting of a qualitative green tea learning data set generation control unit 130 that controls the generation.
The method of claim 2, wherein the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit 110
Tensorflow (111), which serves as an open source software library for data flow programming for various tasks,
A Keras-RetinaNet module 112 that loads a pre-trained coco model and performs object detection using it,
A crawling and scraping module 113 that collects data desired by users from web and app pages,
A quantitative green tea learning data set creation module and qualitative green tea learning data, which are comprised of a Selenium module 114 that collects data on a web/app browser using the driver of the web/app browser. A hybrid green tea learning data set construction device consisting of a set generation module.
The method of claim 2, wherein the qualitative green tea learning data set generation control unit 130
An annotation control unit 131 for green tea taste and quality recognition that forms a bounding box for green tea taste and quality image data and then controls it to generate an annotation;
A RetinaNet-type green tea taste and quality recognition control unit (132) that converts the weight file learned through RetinaNet into hdf5 format that can handle large volumes, and then controls it to recognize the taste and quality of green tea. and,
RetinaNet learned from the COCO dataset is linked with the Tkinter GUI program to recognize green tea, form a bounding box, and then form an annotation format to create and control a qualitative green tea learning dataset. Annotaion type forming control unit 133,
Generating a quantitative green tea learning data set, characterized in that it consists of a qualitative green tea learning data verification control unit 134 that verifies and controls the qualitative green tea learning data generated through the annotation format forming control unit. A hybrid green tea learning data set construction device consisting of a module and a qualitative green tea learning data set creation module.
The method of claim 1, wherein the quantitative green tea learning data set generation module 200
Green tea taste and quality element objects that exist in the green tea taste and quality image and the location of the corresponding green tea taste and quality element object are learned from tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data, A YOLO (You Look Only Once) object recognition module 210 that recognizes taste and quality element objects,
Through the YOLO (You Look Only Once) object recognition module, regions that are estimated to contain green tea objects are derived from among the recognized green tea objects, CNN characteristics are derived from each region, and then each region has information on green tea taste and quality. An R-CNN (Regions with CNN features) algorithm module 220 that classifies classes,
It consists of a quantitative green tea learning data set generation control unit 230 that verifies the classes related to green tea taste and quality classified through the R-CNN (Regions with CNN features) algorithm module and then controls it to generate a quantitative green tea learning data set. A hybrid green tea learning data set construction device consisting of a quantitative green tea learning data set creation module and a qualitative green tea learning data set creation module.
The method of claim 5, wherein the R-CNN (Regions with CNN features) algorithm module 220
Selective search connects adjacent pixels with similar texture, color, and intensity from green tea taste and quality images without distinguishing subjects from tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data. A Selective Search unit 221 that searches for candidate areas among green tea objects that are likely to contain green tea taste and quality factors using a Search method;
A quantitative green tea learning data set generation module and a qualitative green tea learning data set, which are composed of a convolutional neural network unit 222 that classifies objects divided into each area into classes related to green tea taste and quality through a convolutional neural network. A hybrid green tea learning data set construction device consisting of a generation module.
Through the qualitative green tea learning data set creation module, green tea learning data is collected, the collected data is labeled, the amount of data is increased, and a qualitative green tea learning data set is created through verification (steps) S10) and,
A step (S20) of generating a quantitative green tea learning data set based on tea leaf data by period, data for tea quality experts, consumer preference spectrum data, and tea production area data through the quantitative green tea learning data set generation module;
A quantitative green tea learning data set characterized by applying the qualitative green tea learning data set and the quantitative green tea learning data set to building a green tea-related promotion, distribution, and export system linked to smart apps and artificial intelligence deep learning (S30). A hybrid green tea learning dataset construction method consisting of a creation module and a qualitative green tea learning dataset creation module.
The method of claim 7, wherein the step (S10) of generating the qualitative green tea learning data set is
Through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit, green tea learning data is collected, the collected data is labeled, and the overall work for configuring the learning data is run in one program to provide green tea. A step (S11) of forming learning data about
In the augmentation unit, a step (S12) of increasing the amount of green tea taste and quality image data in addition to the learning data about green tea generated through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit;
In the qualitative green tea learning data set generation control unit, the learning data about green tea formed through the GTIAT (Green Tea Images Annotation Tool) type GUI program engine unit and the increased green tea taste and quality image data through the augmentation unit are verified. Then, a hybrid green tea learning data set construction method consisting of a quantitative green tea learning data set generation module and a qualitative green tea learning data set generation module, comprising the step of generating a qualitative green tea learning data set (S13).
The method of claim 7, wherein the step (S20) of generating the quantitative green tea learning data set is
Through the YOLO (You Look Only Once) object recognition module, green tea taste and quality element objects that exist in the green tea taste and quality image are identified from tea leaf data by period, tea quality expert data, consumer preference spectrum data, and tea production area data, and the corresponding A step (S21) of learning the location of the green tea taste/quality element object and recognizing the green tea taste/quality element object,
In the R-CNN (Regions with CNN features) algorithm module, regions where green tea objects are estimated to be present among recognized green tea objects are derived through the YOLO (You Look Only Once) object recognition module, and CNN characteristics are derived from each region. After deriving, each region is classified into classes regarding green tea taste and quality (S22),
The quantitative green tea learning data set generation control unit verifies the classes related to green tea taste and quality classified through the R-CNN (Regions with CNN features) algorithm module, and then generates a quantitative green tea learning data set (S23). A hybrid green tea learning dataset construction method consisting of a quantitative green tea learning dataset creation module and a qualitative green tea learning dataset creation module.