KR102610506B1 - Apparatus, method and program for identifying defects in agricultural products based on artificial intelligence - Google Patents

Abstract

This disclosure relates to an artificial intelligence-based agricultural product defect identification device, which determines the type and location of each defect present in agricultural products in a received image based on an artificial intelligence model and provides a device corresponding to the determined defect type and defect location. By assigning a weight of 1, a defect score can be calculated for each defect.

Description

Apparatus, method and program for identifying defects in agricultural products based on artificial intelligence}

This disclosure relates to a device that identifies defects in agricultural products based on artificial intelligence.

Recently, various methods have been used to select agricultural products.

Representative examples include sorting by weight or size, and with the advancement of science and technology, technologies for quantitatively sorting sugar content or color have also been developed.

In particular, technology for screening using machine vision includes color and feature detection technology using RGB/NIR cameras. This is a very sensitive system numerically, so it is not easy to configure the system, and in order to detect various defects, each defect There are problems such as the need for a camera with the right optical characteristics, which increases the number of cameras, and it is difficult to distinguish between defects or even between the fruit's top and navel.

In addition, it can only be applied to fruits that have been clearly studied optically spectrally, so it is currently only applied to well-known fruits such as apples, tangerines, and tomatoes.

Although there are these attempts and advances in technology related to machine vision, there are limits to obtaining a three-dimensional effect of agricultural products by photographing them with a general camera.

It is expected that agricultural product selection technology using artificial intelligence models will be improved if the problem of not reflecting this three-dimensional effect is resolved, but currently, such technology is not made public.

Republic of Korea Patent Publication No. 10-2023-0082836, (2023.06.09)

The purpose of the embodiment disclosed in this disclosure is to provide an artificial intelligence-based agricultural product defect identification device.

In addition, the purpose of the embodiment disclosed in the present disclosure is to provide a defect identification device for agricultural products that can calculate a defect score for each defect by assigning a weight corresponding to the type and location of each defect existing in the agricultural product.

In addition, the embodiment disclosed in the present disclosure is an agricultural product defect identification device that can calculate a defect score optimized for the type of agricultural product and packaging and distribution method by applying defect calculation weights for each area of the agricultural product according to the packaging and distribution method of the agricultural product. The purpose is to provide.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

An artificial intelligence-based agricultural product defect identification device according to an embodiment of the present disclosure to solve the above-described problem includes: a receiving unit that receives an image containing agricultural products; A storage unit storing an artificial intelligence model in which a method for determining defects in at least one agricultural product is learned; And based on the artificial intelligence model, the type and location of each defect present in the agricultural product in the received image are determined, and a first weight corresponding to the determined defect type and defect location is assigned to each defect. Includes a processor that calculates the score.

In addition, the processor sets an axis for generating a 3D coordinate system according to the type and shape of the agricultural product based on the artificial intelligence model, partitions an area for the agricultural product in the image based on the set axis, and divides each of the partitioned regions into an area for the agricultural product in the image. Based on the defect weight for the region, the first weight corresponding to the location of each defect may be assigned.

In addition, the processor receives an image including depth information and color information about the agricultural product through the receiving unit, and creates a three-dimensional modeling image of the agricultural product in the received image based on the depth information and the color information. generating, converting the 3D modeling image into a 2D planar image, and determining the type and location of each defect based on the converted 2D planar image.

In addition, the processor calculates the defect score for each defect based on the size of each defect and a first weight corresponding to the type of agricultural product, the type and location of each defect, and the calculated defect score. can be combined to calculate the final defect score for the agricultural product.

In addition, it includes a second weight for each region set according to at least one of a packaging method and a distribution method of the at least one agricultural product, and the processor sets the second weight corresponding to the location of each defect according to the type of the agricultural product. The defect score can be calculated by further assigning .

In addition, the processor calculates the final defect score for the agricultural product based on the defect score calculated for each defect, and the more types of defects included in the agricultural product, the higher the final defect score can be calculated. .

In addition, the storage unit stores the probability of occurrence of each type of defect in each area for each type of the at least one agricultural product, and the processor is configured to store the first defect determined for the specific area as a result of analyzing the image in the specific area. If the probability of occurrence of the second defect in the specific area is different from the probability of occurrence of the second defect in the specific area by more than a preset first probability, or if the probability of the second defect occurring in the specific area is more than the preset second probability, the defect occurs in the first probability. It is determined that the second defect is not a defect, and the defect score can be recalculated according to the determined result.

In addition, it further includes a communication unit that communicates in conjunction with a agricultural product sales server, wherein the processor is configured to determine the defect score calculated for each defect included in the agricultural product included in the sales unit according to the sales unit of the agricultural product. The price for the above sales unit of agricultural products can be calculated.

In addition, an artificial intelligence-based agricultural product defect identification method according to an embodiment of the present disclosure for solving the above-described problem is a method performed by a agricultural product defect identification device, and includes the steps of receiving an image containing agricultural products; Determining the type and location of each defect present in the agricultural product in the received image based on an artificial intelligence model learned to determine a defect in at least one agricultural product; and calculating a defect score for each defect by assigning a first weight corresponding to the determined defect type and defect location.

In addition to this, a computer program stored in a computer-readable recording medium for execution to implement the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the above-described problem solving means of the present disclosure, it provides the effect of providing an artificial intelligence-based agricultural product defect identification device.

According to the means for solving the above-described problem of the present disclosure, it provides the effect of calculating a defect score for each defect by assigning a weight corresponding to the type and location of each defect existing in agricultural products.

According to the means for solving the above-described problem of the present disclosure, by applying defect calculation weights for each area of agricultural products according to the packaging and distribution method of agricultural products, it provides the effect of calculating a defect score optimized for the type and packaging and distribution method of agricultural products. .

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram illustrating a conventional wire frame image modeling method.
Figure 2 is a block diagram of an artificial intelligence-based agricultural product defect identification device according to an embodiment of the present disclosure.
Figure 3 is a flowchart of an artificial intelligence-based agricultural product defect identification method according to an embodiment of the present disclosure.
Figure 4 is a diagram illustrating the photographing unit photographing rotating agricultural products.
Figure 5 is a diagram illustrating the setting of a first region of interest and a second region of interest on an image of agricultural products.
Figure 6 is a diagram illustrating the creation of a 3D coordinate system using a connecting line between a first region of interest and a second region of interest.
Figure 7 is a diagram illustrating setting a point group for agricultural products in an image.
Figure 8 is a diagram illustrating the creation of a mesh by connecting each point.
Figure 9 is a diagram illustrating an image of agricultural products to which color information has not yet been applied.
Figure 10 is a diagram illustrating application of the color corresponding to each mesh based on color information.
Figure 11 is a diagram illustrating converting a 3D image into a 2D flat image using a map projection method.
FIG. 12 is a diagram illustrating the conversion of the 3D modeling image of FIG. 10 into a 2D flat image.
FIG. 13 is a diagram illustrating the display of the defective area of agricultural products by reflecting the defective area determined based on FIG. 12 in the image of FIG. 10.
Figure 14 is a diagram illustrating a defect score weight set for each area of agricultural products according to the packaging/distribution method of agricultural products.

Like reference numerals refer to like elements throughout this disclosure. The present disclosure does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present disclosure pertains is omitted. The term 'unit, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'unit, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part "includes" a certain component, this means that it may further include other components, rather than excluding other components, unless specifically stated to the contrary.

Throughout the specification, when a member is said to be located “on” another member, this includes not only cases where a member is in contact with another member, but also cases where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the present disclosure will be described with reference to the attached drawings.

In this specification, the 'agricultural product defect identification device according to the present disclosure' includes all various devices that can perform computational processing and provide results to the user. For example, the device for identifying defects in agricultural products according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take the form of any one.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The server device is a server that processes information by communicating with external devices, and may include an application server, computing server, database server, file server, game server, mail server, proxy server, and web server.

The portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS, GSM, PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, Smart Phone, etc. All types of handheld bases It may include wireless communication devices and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD).

Functions related to artificial intelligence according to the present disclosure are operated through a processor and a storage unit. The processor may consist of one or multiple processors. At this time, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. One or more processors control input data to be processed according to predefined operation rules or artificial intelligence models stored in the storage unit. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weights, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, during the learning process, a plurality of weights may be updated so that the loss or cost value obtained from the artificial intelligence model is reduced or minimized. Artificial neural networks may include deep neural networks (DNN), such as Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or deep Q-network, etc., but are not limited to the examples described above.

According to an exemplary embodiment of the present disclosure, a processor may implement artificial intelligence. Artificial intelligence refers to a machine learning method based on artificial neural networks that allows machines to learn by imitating human biological neurons. Methodology of artificial intelligence includes supervised learning, in which the answer (output data) to the problem (input data) is determined by providing input data and output data together as training data according to the learning method, and only input data is provided without output data. Unsupervised learning, in which the solution (output data) to the problem (input data) is not determined, and rewards are given from the external environment whenever an action is taken in the current state, learning is directed to maximizing these rewards. It can be divided into reinforcement learning. In addition, artificial intelligence methodologies can be divided according to the architecture, which is the structure of the learning model. The architecture of widely used deep learning technology can be divided into convolutional neural networks, recurrent neural networks, transformers, and generative adversarial neural networks.

The device may include an artificial intelligence model. An artificial intelligence model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. Artificial intelligence models may be composed of neural networks (or artificial neural networks) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to an overall model in which artificial neurons (nodes), which form a network through the combination of synapses, change the strength of the synapse connection through learning and have problem-solving capabilities. Neurons in a neural network can contain combinations of weights or biases. A neural network may include one or more layers consisting of one or more neurons or nodes. By way of example, a device may include an input layer, a hidden layer, and an output layer. The neural network that makes up the device can infer the result to be predicted from arbitrary input by changing the weights of neurons through learning.

The processor creates a neural network, trains or learns a neural network, performs operations based on received input data, generates information signals based on the results, or retrains the neural network. Neural network models include CNN, R-CNN, RPN, RNN, S-DNN, S-SDNN, Deconvolution Network, DBN, RBM, Fully Convolutional Network, LSTM Network, Classification such as GoogleNet, AlexNet, VGG Network, etc. It may include, but is not limited to, various types of models such as Network. The processor may include one or more processors to perform operations according to the models of the neural network. For example, a neural network is a deep neural network. It can be included.

Neural networks include CNN, RNN, perceptron, multi-layer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE ( Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), Depp Belief Network (DBN), Deep Convolutional Network (DCN), Deconvolutional Network (DN), Deep Convolutional Inverse Graphics Network (DCIGN), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It may include Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), Kohonen Network (KN), and Attention Network (AN). Those skilled in the art will understand that it is not limited to this and may include any neural network.

According to an exemplary embodiment of the present disclosure, the processor may support a Convolution Neural Network (CNN), a Region with Convolution Neural Network (R-CNN), a Region Proposal Network (RPN), a Recurrent Neural Network (RNN), such as GoogleNet, AlexNet, VGG Network, etc. ), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC/QA, Text Analysis, Dialog System, GPT-3 , GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, and Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation, and Data Creation for ResNet data intelligence. , but is not limited to this. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Figure 1 is a diagram illustrating a conventional wire frame image modeling method.

Referring to Figure 1, in the past, the general shape of the agricultural product was modeled and the captured image was overlaid on the surface of the wire frame. Since there is a limitation from the user's point of view in having to check multiple three-dimensional 2D images, simple photography was used. There was no difference from the image.

In addition, because modeling is carried out in a way that a pre-generated wire frame overlays the image, three-dimensional information such as deformities, compressions, cracks, etc. are expressed in a distorted form compared to existing 2D images and have nothing to do with volume measurement. There is a downside.

Due to these problems, even if defects in agricultural products are displayed on the image, it is difficult for users viewing the image to know the actual shape and characteristics of the defects.

Below, with reference to other drawings, a detailed description of how the agricultural product defect identification device 100, method, and program according to an embodiment of the present disclosure solves the above-mentioned problems will be described.

Figure 2 is a block diagram of an artificial intelligence-based agricultural product defect identification device 100 according to an embodiment of the present disclosure.

The agricultural product defect identification system 10 may include an artificial intelligence-based agricultural product defect identification device 100 and a photographing unit 200.

In an embodiment of the present disclosure, the artificial intelligence-based agricultural product defect identification device 100 may include a photographing unit 200, and may receive image data captured by the external photographing unit 200 through the receiving unit 120. It may be possible.

When the agricultural product defect identification device 100 receives images taken from the outside and processes them according to an algorithm, in this embodiment, the agricultural product defect identification device 100 may be implemented in the form of a server including a server device, At this time, the receiving unit 120 may be configured to include a communication unit.

The agricultural product defect identification device 100 according to an embodiment of the present disclosure includes a processor 110, a receiver 120, and a storage unit 130.

The processor 110 includes a storage unit 130 that stores data for an algorithm for controlling the operation of components within the device or a program that reproduces the algorithm, and performs the above-described operations using the data stored in the storage unit 130. It may be implemented with at least one processor 110 that performs. At this time, the storage unit 130 and the processor 110 may each be implemented as separate chips. Alternatively, the storage unit 130 and the processor 110 may be implemented as a single chip.

In addition, the processor 110 may control any one or a combination of the above-described components in order to implement various embodiments according to the present disclosure described in the drawings below on the present device.

In addition to operations related to the application program, the processor 110 may typically control the overall operation of the device. The processor 110 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the storage unit 130. .

Additionally, the processor 110 may control at least some of the components of the device to run an application program stored in the storage unit 130. Furthermore, the processor 110 may operate in combination with at least two or more of the components included in the device to run the application program.

The processor 110 can execute instructions, commands, and algorithms stored in the storage unit 130 and measure external defects in agricultural products based on artificial intelligence by using an artificial intelligence model.

The receiving unit 120 is configured to receive image data of agricultural products photographed through the photographing unit 200, and depending on the embodiment, may be configured to include a communication unit capable of wired/wireless communication with the outside.

The communication unit may include one or more modules that connect the artificial intelligence-based agricultural product defect identification device 100 to one or more networks.

The communication unit may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module. there is.

Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits communication signals. In addition, the wireless communication module may further include a signal conversion module that modulates a digital control signal output from the processor 110 through a wireless communication interface into an analog wireless signal under the control of the processor 110.

The short-range communication module is for short-range communication and includes Bluetooth®, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, and NFC ( Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology can be used to support short-distance communication.

The storage unit 130 can store data supporting various functions of the device. The storage unit 130 may store a plurality of application programs (application programs or applications) running on the device, data for operating the device, and commands. At least some of these applications may exist for basic functions of the device. Meanwhile, the application program may be stored in the storage unit 130, installed on the device, and driven to perform an operation (or function) by the processor 110.

The storage unit 130 can store data supporting various functions of the device and a program for the operation of the processor 110, and can store input/output data (e.g., music files, still images, videos, etc. ) can be stored, and a plurality of application programs (application programs or applications) running on the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The storage unit 130 is of a flash memory type, hard disk type, solid state disk type, SDD type (Silicon Disk Drive type), and multimedia card micro type. card micro type), card type memory (e.g. SD or XD storage unit 130, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM) , it may include at least one type of storage medium among electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. Additionally, the storage unit 130 is separate from the main device, but may be a database connected by wire or wirelessly.

Additionally, the storage unit 130 may be provided with a plurality of processes for the agricultural product defect identification device 100.

In some embodiments, the apparatus 100 for identifying defects in agricultural products may further include a modeling unit and a display unit.

The processor 110 controls the modeling unit to generate a modeling image for the agricultural product, and when a defect in the agricultural product is recognized, the processor 110 controls the modeling unit to display the defect recognition result in the modeling image, thereby generating a final modeling image.

The display unit displays (outputs) information processed in the device. For example, the display unit may display execution screen information of an application program (eg, an application) running on the device, or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

The processor 110 displays the modeled image on the display unit, allowing the user to visually check the image of the agricultural product and displaying defects on the agricultural product.

The photographing unit 200 includes an RGB camera 210 and a depth camera 230 (eg, 3D camera, depth camera).

The camera processes image frames, such as still images or moving images, obtained by the image sensor in shooting mode. The processed image frame may be displayed on the display unit or stored in the storage unit 130.

The RGB camera 210 can acquire color information (RGB data) by photographing a target.

Depth camera 230 can obtain depth information (depth data) about the target by photographing the target.

In an embodiment of the present disclosure, the RGB camera 210 may refer to an image sensor, the depth camera 230 may refer to a depth sensor, and the photographing unit 200 may refer to a sensor unit.

In the embodiment of the present disclosure, the photographing unit 200 is not limited to the RGB camera 210 and the depth camera 230, respectively, and by photographing the target, color information (RGB data) and depth information (depth data) about the target are obtained. Any means or sensor that can obtain can be applied.

The artificial intelligence-based agricultural product defect identification device 100 and method according to an embodiment of the present disclosure acquire image data including color information and depth information for the agricultural product targeted for defect identification, and 3D model the target agricultural product based on this. Create an image, obtain a planar figure of the 2D image for the generated 3D modeling image, determine the location and type of defects in the agricultural product within the 2D plan, and perform 3D modeling based on the location information within the 2D plan. The location of the defect in the image is specified, a weight is applied according to the location of the specified defect, and a differential score is assigned to each defect to ultimately calculate the defect score for the agricultural product.

Below, the artificial intelligence-based agricultural product defect identification method according to an embodiment of the present disclosure will be described in detail with reference to the flowchart of FIG. 3 and the detailed example drawings of FIGS. 4 to 12.

Figure 3 is a flowchart of an artificial intelligence-based agricultural product defect identification method according to an embodiment of the present disclosure.

With reference to FIG. 3, an artificial intelligence-based method for identifying defects in agricultural products according to an embodiment of the present disclosure will be described.

The processor 110 receives an image containing agricultural products through the receiver 120. (S100)

In detail, the processor 110 receives images of agricultural products photographed through the photographing unit 200, and at this time, the image data includes depth information and color information about the agricultural products.

Figure 4 is a diagram illustrating the photographing unit 200 photographing rotating agricultural products.

The photographing unit 200 can generate image data or video data containing depth information and color information about the agricultural product by capturing at least one image or video of the agricultural product that rotates or moves while rotating as shown in FIG. 4. there is.

In the embodiment of the present disclosure, since image data also consists of a plurality of image frames, it will be referred to as image data below.

Figure 5 is a diagram illustrating the setting of a first region of interest 501 and a second region of interest 502 on an image of agricultural products.

FIG. 6 is a diagram illustrating the creation of a 3D coordinate system 601 using the connecting line between the first region of interest 501 and the second region of interest 502.

In the examples below, agricultural products in the image refer to images corresponding to agricultural products in the image.

Referring to FIG. 5, the processor 110 inputs image data obtained for agricultural products into an artificial intelligence model to create a first region of interest (ROI: Region Of Interest) 501 and a second region of interest (502) for agricultural products. can be set, and a line connecting the first region of interest 501 and the second region of interest 502 can be created.

Then, the processor 110 creates a 3D coordinate system 601 with the generated connection line as the Z axis.

In one embodiment, the artificial intelligence model may set a preset feature area for agricultural products as the first region of interest 501. For example, the first region of interest 501 may be the calyx of agricultural products. .

In one embodiment, the processor 110 identifies the first region of interest 501 corresponding to the calyx in the agricultural products in the image based on an artificial intelligence model, and selects the first region of interest 501 from the agricultural products in the image. The second region of interest 502 can be identified by calculating the longest distance.

In one embodiment, if the feature area preset for the corresponding agricultural product is not detected in the image received from the photographing unit 200, the processor 110 may request rephotography so that the feature area is included.

At this time, in the case of agricultural products that are not circular in shape, the longest axis cannot be the Z-axis, so a 3D coordinate system can be created by applying another method.

For example, the processor 110 may apply various methodologies depending on the target agricultural product, such as latitude and longitude.

The processor 110 can set the axis of the object using an artificial intelligence model. In detail, the processor 110 can set the axis of the object based on at least one of the type and shape of the object.

In the case of an object shaped like an oval and rugby ball, the axis can be set by setting the Z axis to the axis connecting the longest points from ROI 1 and ROI 2 detected by AI.

However, for flat-shaped objects such as flat apples, flat peaches, pumpkins, and some tangerines, the above method is not used. Instead, the opposite vertex, ROI 2, is found using AI, the shortest distance is formed as the z-axis, or the two longest distances are used. The z-axis can be formed using mathematical methods such as finding the intersection point.

In one embodiment, the processor 110 sets an axis for generating a 3D coordinate system according to the type and shape of the agricultural product in the image based on an artificial intelligence model, and divides the area for the agricultural product in the image based on the set axis. .

In one embodiment, the processor 110 may set at least one axis of latitude and longitude as the axis for generating the 3D coordinate system.

The agricultural product defect identification device 100 labels characteristic areas on agricultural products in the image, and when the labeled image is input as learning data to an artificial intelligence model, the artificial intelligence model can detect the corresponding part.

The processor 110 may generate a point group for agricultural products in the image.

Figure 7 is a diagram illustrating setting a point group for agricultural products in an image.

Referring to FIG. 7, the processor 110 may set/generate a point group including a plurality of points on the surface of the agricultural product in the image. At this time, the image to which the processor 110 sets/generates the point group may be an image containing only depth information without color information applied.

The processor 110 may measure depth information for each of a plurality of points based on depth information of image data received from the photographing unit 200.

In one embodiment, the processor 110 may use the depth information of the image data received from the photographing unit 200 to create/set a plurality of points with different depth information, and the points may be set to a preset size or It can also be set based on the actual size of the agricultural product.

In one embodiment, the processor 110 may create/set a plurality of points so that the points have a preset distance between them.

In one embodiment, the processor 110 calculates the area of the surface of the agricultural product based on the image data received from the photographing unit 200, determines the number of points to be created based on the calculated area, Points can also be created/set on the surface of agricultural products based on the determined number of points.

The artificial intelligence-based external defect measurement device for agricultural products according to an embodiment of the present disclosure can apply various algorithms for generating/setting points.

The processor 110 may generate a polygonal mesh by connecting the plurality of points set above.

Figure 8 is a diagram illustrating the creation of a polygonal mesh by connecting each point.

Figure 9 is a diagram illustrating an image of agricultural products to which color information has not yet been applied.

Referring to FIG. 8, the processor 110 may generate a polygonal mesh by connecting points set/created on the surface of the agricultural product in the image.

In one embodiment, the processor 110 may create a triangular mesh by connecting lines connecting adjacent points.

The processor 110 may generate a first 3D modeling image as shown in FIG. 9 based on the mesh created on the surface of the agricultural product in the image and the depth information of each of the plurality of points.

The processor 110 may generate a modeling image for agricultural products by assigning color information to each mesh based on color information included in the image data received in S100.

Figure 10 is a diagram illustrating application of the color corresponding to each mesh based on color information.

Since the processor 110 received image data captured in S100 to include color information of agricultural products, the processor 110 allocates/gives color information corresponding to each of the generated meshes based on the color information to create a second three-dimensional image for agricultural products. Modeling images can be created.

If the first 3D modeling image generated for agricultural products was an image to which only depth information was applied without color (RGB) applied, the second 3D modeling image is an image to which color (RGB) was applied to the first 3D modeling image.

In detail, the processor 110 may determine color information for each of the plurality of points based on the received image data.

Additionally, the processor 110 may generate a second 3D modeling image by assigning color information corresponding to each mesh created by connecting the points.

The processor 110 may convert the generated modeling image into a 2D planar image.

The processor 110 can analyze images containing agricultural products based on an artificial intelligence model. More specifically, the processor 110 can analyze 2D flat images based on an artificial intelligence model.

The processor 110 determines the type and location of each defect present in the agricultural product in the image received from S100. (S200)

Specifically, the processor 110 can determine the type and location of each defect present in agricultural products in a 2D flat image based on an artificial intelligence model.

In one embodiment, if a defect is recognized in agricultural products in the image as a result of analysis of the 2D flat image, the processor 110 may display the area of the recognized defect on the 3D modeling image.

Figure 11 is a diagram illustrating converting a 3D image into a 2D flat image using a map projection method.

FIG. 12 is a diagram illustrating the conversion of the 3D modeling image of FIG. 10 into a 2D flat image.

In an embodiment of the present disclosure, the apparatus 100 for identifying defects in agricultural products may reconstruct a 3D image (3D modeling image) of the agricultural product based on color information and depth information obtained for the agricultural product.

The processor 110 uses a depth map expressing the distance from the center point of the object in the reconstructed 3D modeling image, and a method suitable for unfolding a sphere or ellipsoid, such as Mercator projection, Gal-Peters projection, equirectangular projection, and dynamism mapping. You can use it to convert a 3D object into a 2D image.

That is, the processor 110 rotates the agricultural product, photographs the rotating agricultural product through the photographing unit 200, acquires an image containing color information and depth information about the agricultural product, and generates a 3D modeling image based on this. Of course, you can also obtain a 2D RGB-D development view.

In an embodiment of the present disclosure, the processor 110 collects the data and, if there is a defect visible in the RGB color and a defect visible in the depth map, labels it with the name of the defect using classification or bbox, segmentation method, and if there is no defect at all, the processor 110 collects the data. You can train an artificial intelligence model by labeling it as normal.

If necessary, the processor 110 can input data obtained by performing image differentiation using a Sobel Filter, Laplacian Filter, etc. as learning data.

The processor 110 can train the data set to an artificial intelligence model that can learn images, such as CNN, Trasformoer, etc.

Referring to FIG. 12, the processor 110 controls the modeling unit to convert the second 3D modeling image into a 2D plane image.

The processor 110 analyzes the 2D flat image based on an artificial intelligence model, determines whether a defect exists in the 2D flat image, and if a defect exists, determines the type of defect and includes it in the area determined to be a defect. Check the coordinates of the first meshes.

The processor 110 can analyze the 2D planar image with CNN and process the area recognized as a defect by boxing.

FIG. 13 is a diagram illustrating the display of the defective area of agricultural products by reflecting the defective area determined based on FIG. 12 in the image of FIG. 10.

If a defect is recognized in the agricultural product through the analysis of S600, the processor 110 can distinguish the area where the defect is recognized on the 3D modeling image.

For example, the processor 110 may provide a visual effect to distinguish an area where a defect is recognized from a normal area in a 3D image. The processor 110 may edit the 3D modeling image so that areas where defects are recognized are distinguished from normal areas in the 3D image.

Referring to FIG. 12 , when a defect is recognized in the agricultural product through analysis in S600, the processor 110 may display the area of the recognized defect on the second 3D modeling image.

In detail, the processor 110 displays the recognized defect area on a mesh corresponding to the coordinates of the first mesh on the second 3D modeling image.

The processor 110 determines the first meshes included in the area determined to be defective through the artificial intelligence model as defective areas, and the first meshes may be classified as corresponding to the type of defect determined through the artificial intelligence model. .

That is, the processor 110 can distinguish defects present in agricultural products by type by assigning defect types to the first meshes included in the same defect.

The processor 110 calculates a defect score for each defect by assigning a first weight corresponding to the type of defect and the location of the defect. (S300)

The processor 110 calculates a defect score by applying a second weight according to at least one of the packaging method and the distribution method. (S400)

The processor 110 calculates the final defect score based on the defect score calculated for each defect. (S500)

In the case of agricultural products, especially fruits, even if the defect is the same, the impact on the value (price, quality) of the agricultural product is different depending on which area of the entire agricultural product the defect exists in.

Conventional inventions only judged the location and type of defects in agricultural products, so they had the disadvantage of making it impossible to judge the value of agricultural products based on the judgment results.

The artificial intelligence-based agricultural product defect identification device 100 and method according to an embodiment of the present disclosure determines whether a defect exists in the agricultural product, and if a defect exists, determines the size of the defect, the type of the defect, and the location of the defect. Understand awareness.

In addition, the agricultural product defect identification device 100 calculates a defect score for each defect by applying a corresponding weight according to the location of the defect, thereby making it possible to accurately evaluate the value of the agricultural product.

In one embodiment, the processor 110 may calculate a defect score for each defect based on the type of agricultural product, the first weight corresponding to the type and location of each defect, and the size of each defect. .

The processor 110 analyzes the 2D planar image to determine the type of defect and the location of each defect, and confirms the location on the 3D modeling image of the mesh corresponding to the location of each defect to accurately determine the location of each defect. there is.

Additionally, the processor 110 may calculate the final defect score for the agricultural product based on the calculated defect score. Specifically, the processor 110 may calculate the final defect score for the agricultural product by adding up the calculated defect scores.

In an embodiment of the present disclosure, the lower the final defect score, the higher the quality of the agricultural product, and the higher the final defect score, the lower the quality of the agricultural product.

Figure 14 is a diagram illustrating a defect score weight set for each area of agricultural products according to the packaging/distribution method of agricultural products.

Referring to FIG. 14, the artificial intelligence-based agricultural product defect identification device 100 according to an embodiment of the present disclosure may calculate the final defect score by further reflecting a second weight considering the packaging method and distribution method of the agricultural product.

In an embodiment of the present disclosure, the weights used to calculate the defect score include a weight for the size of the defect for each type of agricultural product, a weight for the type of defect for each type of agricultural product, a weight for the location of the defect for each type of agricultural product, and a packaging method and distribution method for the agricultural product. It may include a weight for each region set according to at least one of the following.

In an embodiment of the present disclosure, the second weight may be set to a higher weight as the area is positioned upward in the packaging method.

In an embodiment of the present disclosure, the second weight may be set to a higher weight as the area is exposed to the outside in the packaging method.

For example, in the case of peaches, they are packaged with the stem facing down. This is to reduce pressure during shipping by increasing the area in contact with the bottom of the pressure-sensitive peach, and normal peaches also have unsightly appearance around the stem due to chafing during growth. The reason why scratches, etc. occur frequently, is because if you place the top down, these scratches are covered and it looks good when packaging.

Because of these characteristics, when sorting peaches, there are two things that can be automated that are different from existing sorting. One is that scratches close to the stem are treated with a lower score than the navel, and the other is that if automatic packaging is possible, the stem must be turned down. The point is that it has to be left.

In one embodiment, when an image of the packaged state of agricultural products is received, the processor 110 may analyze the images based on an artificial intelligence model and set a second weight according to the packaging and distribution method of the agricultural products.

In detail, when an image of a specific agricultural product being packaged is received, the processor 110 determines how the agricultural product is packaged in the received image, and divides the agricultural product into a plurality of areas based on the identified packaging method, A second weight can be set for each of the partitioned areas according to the identified packaging method.

In one embodiment, the processor 110 calculates the final defect score for the agricultural product based on the defect score calculated for each defect, but the more types of defects included in the agricultural product, the higher the final defect score can be calculated. there is.

That is, the more diverse the types of defects contained in the agricultural product, the processor 110 may determine that the agricultural product was grown in an environment vulnerable to various defects and calculate a relatively high final defect score.

For example, the final defect score may be calculated to be higher for a second agricultural product with 4 first defects and 1 second defect than for a first agricultural product with 5 first defects. (Assume that conditions such as size and number of defects are the same)

For example, a second agricultural product with 2 first defects, 2 second defects, and 1 third defect will have a higher final defect score than a first agricultural product with 5 first defects. You can. (Assume that conditions such as size and number of defects are the same)

In conclusion, by applying the first weight, the second weight, and the correction coefficient according to the diversity of defect types, the processor 110 determines the type of agricultural product, the first weight corresponding to the type and location of each defect, and the location of each defect. A defect score for each defect is calculated based on the corresponding second weight and the size of each defect, and a correction coefficient according to the defect score calculated for all defects included in the agricultural product and the diversity of defect types is applied to the agricultural product. The final defect score can be calculated.

The agricultural product defect identification system 10 according to an embodiment of the present disclosure may further include an agricultural product sales server.

The communication department can communicate in conjunction with the agricultural product sales server.

The processor 110 may calculate the price of the agricultural product according to the sales unit based on the defect score calculated for each defect included in the agricultural product included in the sales unit according to the sales unit of the agricultural product. At this time, the sales unit refers to the number of agricultural products sold as a set at one time.

Therefore, the artificial intelligence-based agricultural product defect identification device 100 according to an embodiment of the present disclosure can determine the price of the agricultural product by reflecting the final defect score calculated in real time.

In one embodiment, the first price for a sales unit without defects in the agricultural product is preset, and the processor 110 may adjust the first price by reflecting the final defect score calculated for the agricultural product. .

In one embodiment, the agricultural product sales server can use an artificial intelligence model to analyze the purchasing behavior pattern of a user who uses the agricultural product sales service, and the purchasing behavior pattern means the user's purchasing tendency.

For example, the artificial intelligence model determines the user's preferred agricultural products and preferred price range (e.g. high price, medium price, low price)

The processor 110 may recommend agricultural products to the user by matching the final defect score calculated for the agricultural products with the user's purchasing behavior pattern.

For example, agricultural products with a low final defect score can be recommended to users who want high-priced (high-quality) agricultural products, and agricultural products with a somewhat high final defect score can be recommended to users who want low-priced (low-quality) agricultural products.

In one embodiment, the artificial intelligence-based agricultural product defect identification device 100 may additionally proceed with a correction process for determining the type of defect.

The storage unit 130 stores the probability of occurrence of each type of defect in each area for each type of at least one agricultural product.

In one embodiment, the processor 110 determines the probability that a first defect (defect type) determined for a specific area will occur in the specific area as a result of analyzing the image and the probability that a second defect (defect type) will occur in the specific area. If the difference is greater than the preset first probability, the processor 110 may determine that the defect is a second defect rather than a first defect.

For example, assume that the preset first probability is 30%, the probability of a first defect occurring in the first area of the first agricultural product is 30%, and the probability of a second defect occurring is 85% or more. If the defect is determined to be the first defect in S200, the probability of the second defect occurring is 55%, which is higher than the first probability, so the artificial intelligence model can determine that the defect is the second defect.

In addition, if the processor 110 determines that the probability that a second defect occurs in a specific area is more than a preset second probability as a result of analyzing the image, the artificial intelligence model determines that the defect is a second defect rather than a first defect. You can.

In addition, if the type of defect in the agricultural product in the image changes through the above process, the processor 110 can recalculate the defect score according to the determined result.

The method according to an embodiment of the present disclosure described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a server, which is hardware.

The above-mentioned program is C, C++, JAVA, machine language, etc. that can be read by the processor (CPU) of the computer through the device interface of the computer in order for the computer to read the program and execute the methods implemented in the program. It may include code coded in a computer language. These codes may include functional codes related to functions that define the necessary functions for executing the methods, and include control codes related to execution procedures necessary for the computer's processor to execute the functions according to predetermined procedures. can do. In addition, these codes may further include memory reference-related codes that indicate at which location (address address) in the computer's internal or external memory additional information or media required for the computer's processor to execute the above functions should be referenced. there is. In addition, if the computer's processor needs to communicate with any other remote computer or server in order to execute the above functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes regarding whether communication should be performed and what information or media should be transmitted and received during communication.

The storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the user's computer. Additionally, the medium may be distributed to computer systems connected to a network, and computer-readable code may be stored in a distributed manner.

The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which this disclosure pertains.

Above, embodiments of the present disclosure have been described with reference to the attached drawings, but those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: Agricultural product defect identification system
100: Agricultural product defect identification device
110: processor
120: Receiving unit
130: storage unit

Claims (10)

A receiving unit that receives images containing agricultural products;
A storage unit storing an artificial intelligence model in which a method for determining defects in at least one agricultural product is learned; and
Based on the artificial intelligence model, determine the type and location of each defect present in the agricultural product in the received image,
A defect score for each defect is given a second weight corresponding to the location of each defect based on a first weight corresponding to the determined defect type and defect location and a weight for each area set according to the packaging method of the agricultural product. Including a processor that calculates,
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
The processor,
Based on the artificial intelligence model, set the axis for creating a 3D coordinate system according to the type and shape of the agricultural product,
Dividing an area for agricultural products in the image based on the set axis,
Characterized in assigning the first weight corresponding to the location of each defect based on the defect weight for each partitioned area,
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
The processor,
Receive an image including depth information and color information about the agricultural product through the receiving unit,
Generating a three-dimensional modeling image for agricultural products in the received image based on the depth information and the color information,
Converting the 3D modeling image into a 2D flat image,
Characterized in determining the type and location of each defect based on the converted 2D planar image.
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
The processor,
Calculating the defect score for each defect based on the type of agricultural product, a first weight corresponding to the type and location of each defect, and the size of each defect,
Calculate the final defect score for the agricultural product by adding up the calculated defect scores,
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
The processor,
When an image of the agricultural product being packaged is received, the manner in which the agricultural product is packaged is determined from the received image,
Characterized in that the agricultural products are divided into a plurality of areas based on the identified packaging method, and the second weight is set in each divided region according to the identified packaging method,
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
The processor,
Calculate the final defect score for the agricultural product based on the defect score calculated for each defect,
Characterized in that the more types of defects contained in the agricultural product, the higher the final defect score is calculated.
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
The storage unit,
The probability of occurrence of each type of defect in each area is stored for each type of the at least one agricultural product,
The processor,
As a result of analyzing the image, the probability that the first defect determined for a specific area will occur in the specific area and the probability that the second defect will occur in the specific area are different than a preset first probability, or the probability that the second defect will occur in the specific area is different than the preset first probability. 2 If the probability of occurrence of a defect is greater than the preset second probability, it is determined that the defect is the second defect rather than the first defect,
Characterized in that the defect score is recalculated according to the determined result,
Artificial intelligence-based agricultural product defect identification device.
According to paragraph 1,
It further includes a communication unit that communicates in conjunction with the agricultural product sales server,
The processor,
Characterized in calculating the price for the sales unit of the agricultural product based on the defect score calculated for each defect included in the agricultural product included in the sales unit according to the sales unit of the agricultural product,
Artificial intelligence-based agricultural product defect identification device.
A method performed by an agricultural product defect identification device,
Receiving an image containing agricultural products;
Determining the type and location of each defect present in the agricultural product in the received image based on an artificial intelligence model learned to determine a defect in at least one agricultural product; and
A defect score for each defect is given a second weight corresponding to the location of each defect based on a first weight corresponding to the determined defect type and defect location and a weight for each area set according to the packaging method of the agricultural product. Including the step of calculating,
Artificial intelligence-based agricultural product defect identification method.
A computer-readable recording medium combined with a hardware computer and storing a program for executing the method of claim 9.

Images