KR20220010374A - Growth analysis method based on smart edge device and apparatus and system therefor - Google Patents

Abstract

The present invention relates to a smart edge device-based growth analysis method and a device and system therefor. According to an embodiment of the present invention, the smart edge device-based growth analysis method comprises the steps of: acquiring and receiving an image photographed through a provided camera; detecting and classifying an object from the acquired image; acquiring a pre-trained deep learning model to be used in accordance with a classification processing result from an external database; detecting an object to be analyzed through machine learning based on the acquired deep learning model; analyzing a maturity level of the detected object; and determining a crop harvest time on the basis of the analyzed maturity level. Therefore, an optimal harvest time and a cultivation method can be suggested.

Description

Growth analysis method based on smart edge device and apparatus and system therefor

The present invention relates to crop growth monitoring technology, and more particularly, analyzes the real-time growth status and maturity of crops through deep learning based on images and spectral data collected through a three-dimensional depth camera and hyperspectral camera. To provide a growth analysis method based on a smart edge device equipped with an artificial intelligence algorithm and an apparatus and system for the same.

As cloud computing is recently activated, a large number of various data are being processed and analyzed in the cloud computing environment.

Cloud computing is a network environment that provides services such as server, storage, and software analysis through the Internet.

Recently, as the number of people and companies using cloud services increases exponentially, a situation in which the limit capacity that can be processed by the server and data sensor is exceeded frequently occurs, resulting in data processing delay. In addition, various security problems are also occurring in the communication process for cloud computing.

Edge computing technology is an introduced technology to solve data processing speed, capacity, and security issues in the cloud environment.

Edge computing means that an end device performs computing based on sensed data.

In cloud computing, where data processing is concentrated in data centers, etc., edge computing reduces the data processing burden in the cloud by performing some data processing on end devices, and consequently data processing delay can be prevented in advance. there are advantages to

Recently, as automation and unmanned cultivation technologies are introduced in the agricultural field, research on edge computing is being actively conducted.

An object of the present invention is to provide a growth analysis method based on a smart edge device, and an apparatus and system therefor.

Another object of the present invention is a smart edge device equipped with an artificial intelligence algorithm to analyze the real-time growth status and maturity of crops through deep learning based on images and spectral data collected through a three-dimensional depth camera and a hyperspectral camera. It is to provide a growth analysis method based on it, and an apparatus and system therefor.

Another object of the present invention is to provide a smart edge device-based growth analysis method capable of performing edge computing by mounting a camera on a GPU board having an AI chipset embedded therein, and an apparatus and system therefor.

Another object of the present invention is to provide a real-time object recognition and classification service using a low-power, high-performance image processing embedded solution equipped with a deep learning processor, thereby preventing excessive data traffic and processing delay in a cloud environment in advance. It is to provide a smart edge device-based growth analysis method, and an apparatus and system therefor.

Another object of the present invention is to provide a growth analysis method based on a smart edge device capable of real-time decision-making by performing edge computing through an artificial intelligence chipset built in even in a network failure, and an apparatus and system therefor.

Another object of the present invention is to provide a growth monitoring apparatus based on a small and inexpensive smart edge device that operates with low power without a heat sink.

Another object of the present invention is to provide a smart edge device-based growth monitoring apparatus capable of presenting an optimal harvest time and cultivation method based on environmental data and crop growth data collected in the field.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

A smart edge device-based growth analysis method according to an embodiment of the present invention includes the steps of acquiring and receiving an image photographed through a provided camera, detecting and classifying an object from the acquired image, and according to the classification processing result Acquiring a pre-trained deep learning model to be used from an external database, detecting an analysis target object through machine learning based on the obtained deep learning model, analyzing the maturity of the detected object, and the analyzed maturity It may include determining a crop harvest time based on the.

In an embodiment, the method may include generating an image file for the acquired image and transmitting the generated image file to a cloud server, wherein the image file may be stored in a database interworking with the server. .

In an embodiment, the camera may include a 3D DEPTH camera capable of capturing RGB images and DEPTH images, and a spectroscopic camera capable of capturing RGN images.

In an embodiment, the method may further include obtaining sugar content measurement information corresponding to the detected object, and the maturity level may be analyzed further based on the sugar content measurement information.

In an embodiment, the method includes: obtaining object information corresponding to the detected object based on image information obtained from the 3D DEPTH camera; and vegetation based on a reflectance calculated based on the image information obtained from the spectroscopic camera The method further comprises calculating an index value, wherein the object information includes at least one of size information, shape information, growth information, and chromaticity information, and the harvest time of the crop is further based on the object information and the vegetation index value. can be determined.

In an embodiment, the reflectivity includes reflectance Red and reflectance NIR in the near-infrared band for the red region of the visible light band, and the vegetation index value NDVI is expressed by the following formula:

NDVI = (NIR-Red)/(NIR+Red)

can be calculated by

In an embodiment, the method further comprises the steps of obtaining growth information and location information of a crop when the detected object is a seedling, and determining whether the seedling is defective based on the obtained growth information and location information And, the growth information may include at least one of a leaf length, a leaf width, and a leaf type index.

In an embodiment, the detecting of the analysis target object based on the obtained deep learning model includes dividing the entire coordinates of the obtained image into a plurality of grid cells, generating two bounding boxes per one grid cell, and the Calculating the reliability, which is the probability that an object is in the bounding box, classifying a class for identifying which object the image in each grid cell is included in, and based on the calculated reliability and the classified class It may include detecting the analysis target object.

In an embodiment, the method includes the steps of obtaining a growth profile from the external database, obtaining growth measurement information, analyzing a correlation between the growth profile and growth measurement information, and a growth model based on the correlation analysis result. The method may further include supplementing the setting information and storing the supplemented setting information in the external database, wherein the setting information may include at least one of temperature information, humidity information, carbon dioxide concentration information, and electrical conductivity.

The growth analysis system according to another embodiment of the present invention provides an image photographing system that provides RGB images, DEPTH images, and RGN images photographed through a 3D DEPTH camera and a spectroscopic camera, and an image file based on an image obtained from the image photographing system generates, classifies and detects objects from the acquired image, and acquires a pre-trained deep learning model to be used according to the classification processing result from an external database through an Ethernet communication module, and the acquired deep learning model may include an image analysis system that detects an object to be analyzed through machine learning based on , and analyzes a maturity level corresponding to the detected object, and a measurement information analysis system that determines a crop harvest time based on the analyzed maturity. .

In an embodiment, the image capturing system may transmit the generated image file to a server located in a cloud environment through the Ethernet communication module, and the image file may be stored in a database interworking with the server.

In an embodiment, the image analysis system may include a USB communication module for communication with the 3D DEPTH camera and a Wi-Fi communication module for communication with the spectroscopic camera.

In an embodiment, the apparatus may further include a sugar content measurement system for measuring a sugar content corresponding to the detected object, and the measurement information analysis system may further analyze the maturity level based on the measured sugar content.

In an embodiment, the image analysis system includes an object information extraction unit for extracting object information corresponding to the detected object based on the image obtained from the 3D DEPTH camera, and a reflectance calculated based on image information obtained from the spectroscopic camera and a vegetation index calculation unit for calculating a vegetation index value based on The harvest time of the crop may be determined.

In an embodiment, the reflectivity includes reflectance Red and reflectance NIR in the near-infrared band for the red region of the visible light band, and the vegetation index value NDVI is expressed by the following formula:

NDVI = (NIR-Red)/(NIR+Red)

can be calculated by

In an embodiment, when the detected object is a seedling, the measurement information analysis system further comprises a failure determining unit for determining whether the seedling is defective based on growth information and location information of the crop, wherein the growth information includes a leaf length and a leaf width. And it may include at least one of the leaf type index.

In an embodiment, the image analysis system divides the overall coordinates of the obtained image into a plurality of grid cells, generates two bounding boxes per one grid cell, and calculates the reliability, which is the probability that an object is in the bounding box, and , an object detector for classifying a class for identifying which object the image in each grid cell is included in, and detecting the analysis target object based on the calculated reliability and the classified class.

In an embodiment, the measurement information analysis system further comprises a growth model updater that supplements setting information of the growth model by analyzing the correlation between the growth profile and the growth measurement information, wherein the setting information includes temperature information, humidity information, and carbon dioxide concentration information. and at least one of electrical conductivity.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

The present invention has the advantage of providing a smart edge device-based growth analysis method and an apparatus and system for the same.

In addition, the present invention is based on a smart edge device equipped with an artificial intelligence algorithm that analyzes the real-time growth status and maturity of crops through deep learning based on images and spectral data collected through a three-dimensional depth camera and a hyperspectral camera. There is an advantage of providing a growth analysis method and an apparatus and system therefor.

In addition, the present invention has the advantage of providing a smart edge device-based growth analysis method capable of performing edge computing by mounting a camera on a GPU board with an artificial intelligence chipset, and an apparatus and system therefor.

In addition, the present invention provides a real-time object recognition and classification service using a low-power, high-performance image processing embedded solution equipped with a deep learning processor, thereby preventing excessive data traffic and processing delay in a cloud environment in advance. There is an advantage of providing a device-based growth analysis method and an apparatus and system therefor.

In addition, the present invention has the advantage of providing a smart edge device-based growth analysis method capable of real-time decision-making by performing edge computing through a built-in artificial intelligence chipset even in a network failure, and an apparatus and system for the same.

Another object of the present invention is to provide a growth monitoring apparatus based on a small and inexpensive smart edge device that operates with low power without a heat sink.

In addition, the present invention has the advantage of providing a growth monitoring apparatus based on a smart edge device capable of presenting an optimal harvest time and cultivation method based on environmental data and crop growth data collected in the field.

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a view for explaining the structure of a growth analysis system according to an embodiment of the present invention.
2 is a view for explaining an image capturing and storage procedure in the growth analysis system according to an embodiment of the present invention.
3 is a view for explaining a growth analysis procedure in the growth analysis system according to an embodiment of the present invention.
4 is a view for explaining the detailed structure of the growth analysis system according to an embodiment of the present invention.
5 is a perspective view of a growth analysis system according to an embodiment of the present invention.
6 is an exploded view of a growth analysis system according to an embodiment of the present invention.
7 is a perspective view showing the structure of the bracket of FIG.
8 shows YOLO test results for strawberry maturity labeling according to an embodiment.
9 is a flowchart illustrating a procedure for detecting an object from a crop image according to an embodiment of the present invention.
10 shows a growth information registration screen according to an embodiment of the present invention.
11 is a view for explaining a procedure for generating a growth information registration screen according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 6 .

1 is a view for explaining the structure of a growth analysis system according to an embodiment of the present invention.

Referring to FIG. 1 , the growth analysis system 100 may largely include an image capturing system 10 , an image analysis system 20 , and a measurement information analysis system 30 .

The growth analysis system 100 may access the cloud 40 environment.

The image analysis system 20 may control the operation of the image capturing system 10 .

The cloud 40 may include a photographed image server 41 and an analysis database 42 .

The image capturing system 10 is equipped with an RGB camera, a depth camera, an RGN (or NDVI) camera, a near infrared (NIR) camera, and a thermal imaging infrared (Infrared, IR) camera. An image of an object may be captured and provided to the image analysis system 20 . Here, the RGB camera and the depth camera may be integrated into one device. Hereinafter, for convenience of description, a camera in which an RGB camera and a depth camera are integrated will be referred to as a 3D depth camera.

A near-infrared (NIR) camera can be used to determine the sugar content (BRIX) of a fruit through spectral analysis.

Thermal imaging IR cameras can be used to measure the leaf temperature of crops.

Growth analysis system 100 according to an embodiment of the present invention may be configured to include at least one of the above-described cameras according to the purpose of its use.

In the following description, it will be described as an example that the image analysis system 20 includes an RGB/Depth camera (or 3D DEPTH camera) and an RGN camera (or a spectroscopic camera).

The image capturing system 10 and the image analysis system 20 may be connected by wire and/or wirelessly. For example, the RGB/Depth camera may be connected to the image analysis system 20 through USB, and the RGN camera may be connected to Wi-Fi.

The image photographed by the image capturing system 10 is an RGB image, which is a digital image expressed by mixing three primary colors of red, green, and blue, a depth image that is an image capable of measuring a distance, and It may include an RGN image that is an image obtained by photographing red, green, and near infrared (NIR) data. Here, the photographed image may be transmitted to the photographed image server 41 through the image analysis system 20 and registered in the analysis database 42 . The image analysis system 20 may be linked with the photographing image server 41 through Ethernet communication.

The image analysis system 20 may classify images through deep learning, and analyze the maturity level of crops based on the classified images.

The measurement information analysis system 30 may generate an optimized growth model by comparing and analyzing the growth profile generated by the growth expert and the growth measurement information.

The image analysis system 20 may calculate a normalized difference vegetation index (NDVI) using the NIR image (or data) taken by the RGN camera (or NIR camera or spectroscopic camera).

Here, NDVI is a vegetation index that can measure the health status of the crop by analyzing the reflectance of the visible light (especially the red region) and the reflectance of the near-infrared band for the crop to be observed - for example, a green plant. It can be calculated according to the formula of

NDVI = (NIR - Red) / (NIR + Red)

Here, NIR is the reflectance of the near infrared band, and Red is the reflectance of the red visible light band.

The image analysis system 20 may calculate and/or generate crop coordinates, maturity, NDVI distribution, and the like, based on the RGB image.

The measurement information analysis system 30 may calculate the growth model setting information for optimizing the existing growth model by analyzing the correlation between the growth profile value and the current environmental measurement value.

For example, the growth model setting information may include a temperature value, a humidity value, a carbon dioxide concentration value, a hydrogen ion concentration index (PH) value, an electrical conductivity (EC) value, and the like.

Information analyzed by the measurement information analysis system 30 may be stored in the analysis database 42 .

As an example, the analysis database 42 is, as shown in Table 1 below, growth analysis information, growth survey item information of the growth model system, growth survey item information for each variety, growth information header information, growth information detailed information, growth model Information, growth profile information, sensor measurement information of the environmental information system, etc. can be configured and maintained in the form of a table.

<table specification> Table ID Table name CRUD note GROWTHIMG_MASTER Growth analysis information R GROWTH_MASTER Growth survey item information R GROWTH_HEAD growth information header R GROWTH_DATA Growth information details R GROWTH_PROFILE growth profile R GROWTH_MODEL growth model R

Here, the growth analysis information may include image information, maturity information, and NDVI analysis information.

Based on the start date and end date of the growth profile information, it is matched with the investigation date of the growth information header, and the measurement date and measurement time of the sensor measurement information may be matched based on the start date, end date, start time, and end time of the growth model information.

In the growth analysis system 100 according to the embodiment, a program for controlling a camera provided in the image capturing system 10 may be loaded and executed. Here, the program may be run on the Ubuntu Linux system, but is not limited thereto.

2 is a view for explaining an image capturing and storage procedure in the growth analysis system according to an embodiment of the present invention.

Referring to FIG. 2 , the growth analysis system 100 may generate an RGB image file and a depth image file by performing RGB&DEPTH camera shooting, and then store it in the internal memory ( S201 and S202 ).

Growth analysis system 100 may generate an RGN image file by performing RGN camera shooting and then store it in the internal memory (S203 and S204).

The growth analysis system 100 may read the generated RGB image file, the depth image file, and the RGN image file from the internal memory and transmit it to the photographing image server 41 of the cloud 40 ( S205 and S206 ).

The image files transmitted to the photographed image server 41 may be stored in the analysis database 42 (S207).

In the embodiment of FIG. 2 , the growth analysis system 100 may control the RGB&DEPTH camera and the RGN camera to perform photographing at a predetermined period - for example, 1 hour.

As an example, file names corresponding to each of the RGB image, the depth image, and the RGN image may be stored as COLOR.jpg, DEPTH.jpg, and NDVI.jpg.

The growth analysis system 100 may convert the generated image file into Base64 format data and transmit it to the photographed image server 41 .

Here, Base64 refers to an encoding method that converts bit binary data (eg, an executable file, an image file, etc.) into a string consisting of only characters in the common ASCII region that is not affected by character codes.

When the image file is received, the photographed image server 41 may add a photograph name, photographing year, month, and photographing time corresponding to the received image file to the file name and store it.

3 is a view for explaining a growth analysis procedure in the growth analysis system according to an embodiment of the present invention.

The photographed image server 41 may perform maturity deep learning on the previously collected dataset to generate a learning model required for maturity analysis, and then store the learning model file in the analysis database 42 (S301 to S303). Here, the learning model may be generated for each crop type.

The growth analysis system 100 may read a pre-generated image file from an internal memory and perform a classification process (S303 to S304). For example, the growth analysis system 100 may read a corresponding image file and classify an object part on the corresponding image.

Growth analysis system 100 may obtain a learning model corresponding to the analysis target crop from the analysis database 42 (S305).

Growth analysis system 100 may perform maturity analysis on the classified object by performing machine learning based on the acquired learning model (S306). Here, the machine learning algorithm may be a You Only Look One (YOLO) method, but is not limited thereto.

YOLO is a real-time object detection algorithm, a machine learning algorithm developed with inspiration from the point that a person instantly recognizes the location and type of an object when looking at a picture.

For example, if the crop to be analyzed is a strawberry, a task of displaying the object to be recognized by the YOLO Mark as a box is performed to collect the strawberry image required for image learning and perform machine learning using the YOLO algorithm. can This corresponds to a preprocessing process for accurate learning, and when the boxed work is performed, as shown in Table 1 below, boxes may be displayed with different labeling according to the maturity stage - that is, maturity level.

The growth analysis system 100 may transmit maturity analysis information to the photographed image server 41 (S307). Here, the maturity analysis information may include coordinate information, size information, shape information, chromaticity information, and maturity information corresponding to the classified object.

The photographed image server 41 may receive the maturity analysis information and store it in the analysis database 42 (S308).

<Strawberry maturity labeling> division label Explanation Step 1 immature very immature Step 2 30_immature 30% mature Step 3 50_immature 50% maturity Step 4 60_immature 60% mature Step 5 80_immature 80% mature Step 6 mature very mature

Referring to [Table 2], the 1st stage of maturity, 'immature', refers to the unripe strawberry, and the labels corresponding to the 2nd to 5th stages of maturity are '30_immature', '50_immature', '60_immature', '80_immature ' denotes strawberries at 30% maturity, strawberries at 50% maturity, strawberries at 60% maturity, and strawberries at 80% maturity, respectively. Finally, 'mature', which is the 6th stage of maturity, refers to strawberries that are fully ripe, indicating that they are ready-to-harvest strawberries.

The growth analysis system according to the present invention generates a learning model through deep learning on strawberry images pre-classified according to maturity as the need to predict the harvest time according to the maturity of crops arises when building an automated crop harvesting platform, and actual growth By applying the image captured in the environment to the generated learning model, there is an advantage in that crop images can be classified according to maturity.

In addition, when the growth analysis system according to the present invention is mounted on an automatic crop harvesting robot, there is an advantage in that crops can be automatically classified and harvested according to maturity.

4 is a view for explaining the detailed structure of the growth analysis system according to an embodiment of the present invention.

Referring to FIG. 4 , the growth analysis system 400 may largely include an image capturing system 410 , an image analysis system 420 , a measurement information analysis system 450 , and a sugar content measurement system 460 .

Here, the image analysis system 420 and the measurement information analysis system 450 may be implemented by being embedded in one deep learning processor board.

The deep learning processor board is equipped with a memory so that an image file can be stored. Here, the memory may be implemented as a removable memory - for example, an SD card.

In an embodiment, the deep learning processor board may be operated with low power without the need for a separate heat sink, thereby providing a compact and inexpensive growth analysis system 400 .

The image capturing system 410 may include a 3D DEPTH camera 411 and a spectroscopic camera (RGN camera, 412 ).

The image analysis system 420 may include a communication unit 430 and an extraction unit 440 .

The communication unit 430 may be configured to include a USB communication module 431 , a Wi-Fi communication module 432 , and an Ethernet communication module 433 .

For example, the 3D DEPTH camera 411 may be connected to the USB communication module 431 through a USB communication cable to exchange information with the image analysis system 420 .

The 3D DEPTH camera 411 is equipped with an RGB sensor and an infrared projector to capture an RGB image and a depth image.

The spectroscopic camera 412 may be connected to the Wi-Fi communication module 432 through Wi-Fi communication to exchange information with the image analysis system 420 .

The spectral camera 412 is provided with a hyperspectral scanner to take an RGN image. Here, the RGN image can be used for calculating the vegetation index and analyzing the sugar content.

The sugar content measurement system 460 may be connected to the USB communication module 431 or the Wi-Fi communication module 432 to exchange information with the image analysis system 420 .

The extraction unit 440 includes an image generation unit 441 , an object detection unit 442 , an object information extraction unit 443 , a vegetation index (NDVI) calculation unit 444 , a sugar content extraction unit 445 and a data transmission unit ( 446) may be included.

The image generator 441 may generate an image file for an image captured by cameras included in the image capturing system 410 . Here, the image file may include an RGB image file, a depth image file, and an RGN image file.

The object detector 442 may identify an object included in the image file based on the YOLO algorithm.

The object information extraction unit 443 may extract size information, shape information, maturity information, and the like corresponding to the identified object.

The vegetation index calculator 444 may extract an NDVI value based on the RGN image.

The sugar content extraction unit 445 may extract the sugar content of the corresponding crop based on the sugar content sensing information received from the sugar content measurement system 460 . The sugar content extraction unit 445 may further extract the sugar content of the corresponding crop based on the RGN image.

The data transmission unit 446 may transmit the generated image file and various types of extraction information to the corresponding server in the cloud through the Ethernet communication module 443 .

The measurement information analysis system 450 includes a harvest time determining unit 451 that determines a harvest time of a corresponding crop based on various pieces of information extracted according to the analysis of the image analysis system 420, and a method of determining a harvesting method of the corresponding crop. It may include a harvest method determination unit 452 , a failure determination unit 453 for determining whether or not the seedlings are defective, and a growth model update unit 454 .

As an example, the harvest time determiner 451 may determine the harvest time of the crop using a random forest method.

For example, when the classified object is a seedling, the defective determining unit 453 may determine whether the seedling is defective based on growth information such as leaf length/leaf width/leaf type index of the crop and location information.

The growth model update unit 454 may obtain a growth profile by accessing an external database through Ethernet communication, and may acquire growth measurement information from various sensors. Here, the sensor may be integrally mounted to the growth analysis system 400, but this is only one embodiment, and the growth analysis system 400 communicates with an external sensor through any one of a USB cable or Wi-Fi communication or Ethernet communication. It is possible to acquire various types of sensing information through communication connection.

The growth model update unit 454 supplements the setting information of the growth model with an optimal value by analyzing the correlation between the growth profile and the growth measurement information, and updates the growth model by storing the supplemented setting information in the external database. have.

Here, the setting information may include at least one of temperature information, humidity information, carbon dioxide concentration information, and electrical conductivity.

5 is a perspective view of a growth analysis system according to an embodiment of the present invention.

6 is an exploded view of a growth analysis system according to an embodiment of the present invention.

Referring to FIG. 6 , the growth analysis system 600 includes a main body 610 , a control board 620 , first to third brackets 631 to 633 , a 3D DEPTH camera 640 , a spectroscopic camera 650 and a front surface. It may be configured to include a cover (660). Here, an RGN camera or an NIR camera may be applied instead of the spectroscopic camera 650 .

An external power terminal and an Ethernet cable connection port may be provided on the rear surface of the main body 610 .

The control board 620 may be coupled to the first bracket 631 on one side inside the main body 610 .

The 3D DEPTH camera 640 and the spectroscopic camera 650 may be coupled to one side of the main body 611 and/or the front cover 660 through the second bracket 632 and the third bracket 633, respectively.

The front cover 660 includes a first opening 661 for exposing the lens of the spectroscopic camera 650 to the outside and a second opening 662 for exposing the lens and sensor of the 3D DEPTH camera 640 to the outside. can be configured.

A vent 611 is formed in the body 610 so that internal heat may be discharged to the outside.

7 is a perspective view showing the structure of the bracket of FIG.

8 shows YOLO test results for strawberry maturity labeling according to an embodiment.

8 , each strawberry is identified through the object box, and a label corresponding to the identified strawberry may be displayed on one side of the object box. In this case, labels of different colors may be displayed according to the maturity level of the strawberry.

9 is a flowchart illustrating a procedure for detecting an object from a crop image according to an embodiment of the present invention.

In detail, FIG. 8 is an image analysis flowchart illustrating a procedure for detecting an object from a strawberry image.

9 is a flowchart illustrating a procedure for detecting an object from a crop image according to an embodiment of the present invention.

In detail, FIG. 8 is an image analysis flowchart illustrating a procedure in which the growth analysis system detects an object from an input image through a neural network in which the YOLO algorithm is mounted.

The growth analysis system may divide the entire coordinates of the input image into S x S grid cells (S910).

The growth analysis system may generate two bounding boxes per one grid cell, and calculate the probability that an object is in the bounding box—that is, confidence— ( S920 ).

Here, the higher the reliability, the thicker the border of the corresponding bounding box may be displayed.

The growth analysis system may classify a class in order to identify which object the image in each grid cell is included in (S930). Here, the grid cells may be displayed in different colors according to the classified class.

The growth analysis system may detect an object based on the calculated reliability and the classified class (S940).

When the reliability within the same class is less than or equal to the reference value - that is, when it is close to 0 - the corresponding bounding box is recognized as a background and all can be removed.

One vector is output for each bounding box. Each vector has as many elements as the number of classes, and each element represents the classification score of the corresponding class. According to the method described above, the growth analysis system may detect an object by classifying an internal object based on vector values of the remaining bounding boxes after deleting unnecessary bounding boxes.

In an embodiment, the growth analysis system may detect an object using a non-maximal suppression algorithm for boundary boxes remaining after removal.

10 shows a growth information registration screen according to an embodiment of the present invention.

In an embodiment, the growth information registration screen may be generated by a growth analysis system and provided to a user terminal, etc., but this is only one embodiment, and another embodiment is to be generated by a photographed image server located in the cloud and provided to the user terminal can

Referring to FIG. 10 , the growth information registration screen may be displayed by comparing information measured by the growth model system and growth profile information, and comparing and displaying current environmental conditions and growth model information.

ⓐ RGB image, DEPTH image, RGN image captured by the image recording system

ⓑ Corresponding values of growth length, stem thickness, flower bed height and growth profile measured in the growth model system

ⓒ Sensor values measured in the environmental monitoring system and growth model values in the growth model system

ⓓ Maturity Analyzed in Growth Analysis System

ⓔ Vegetation index (NDVI value) analyzed in the growth analysis system

When the growth information registration screen window is opened, the most recently photographed image and the photographing date may be displayed on one side of the screen.

In addition, when a specific photo is selected by the user, the shooting date, maturity and vegetation index corresponding to the selected photo may be displayed on one side of the screen.

In addition, measurement information of the measurement date of the closest date to the photographing date of the user-selected picture may be displayed on one side of the screen.

11 is a view for explaining a procedure for generating a growth information registration screen according to an embodiment of the present invention.

11 , the growth analysis system generates image files and image information, the environmental monitoring system generates the latest environmental information, and the growth model system generates growth profiles, growth models, growth headers and growth information details. can do.

For example, information generated by the growth analysis system, the environmental monitoring system, and the growth model system may be transmitted to the photographed image server 41 of the cloud 40 and stored in the analysis database 42 .

Referring to FIG. 11 , the photographed image server 41 may display at least one image and image information corresponding thereto by inquiring from the analysis database 42 according to a user's inquiry request.

When any one of the displayed images is selected by the user, the photographed image server 41 may query and compare current measurement information (image information) and profile information from the analysis database 42 . Here, the profile information may include not only growth profile information but also growth information header information and detailed growth information information.

The photographed image server 41 may compare and display the growth model information and the latest environmental information from the analysis database 42 .

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. A software module may reside in a storage medium (ie, memory and/or storage) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM.

An exemplary storage medium is coupled to the processor, the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral with the processor. The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (18)

In the smart edge device-based growth analysis method,
obtaining and receiving an image photographed through a provided camera;
detecting and classifying an object from the acquired image;
obtaining a pre-trained deep learning model to be used according to the classification processing result from an external database;
detecting an analysis target object through machine learning based on the obtained deep learning model;
analyzing the maturity level of the detected object; and
determining a crop harvest time based on the analyzed maturity level;
Growth analysis method comprising a. According to claim 1,
generating an image file for the acquired image; and
Transmitting the generated image file to a cloud server
Including, wherein the image file is a growth analysis method, characterized in that stored in a database linked with the server. According to claim 1,
The camera is a 3D DEPTH camera capable of capturing RGB images and DEPTH images, a spectroscopic camera capable of capturing RGN images, and a thermal imaging infrared camera. In claim 1,
The method further comprising obtaining sugar content measurement information corresponding to the detected object, and further analyzing the maturity level based on the sugar content measurement information. In claim 3,
obtaining object information corresponding to the detected object based on the image information obtained from the 3D DEPTH camera; and
Calculating a vegetation index value based on the calculated reflectance based on the image information obtained from the spectroscopic camera
further comprising,
The object information includes at least one of size information, shape information, growth information, and chromaticity information,
Growth analysis method, characterized in that further based on the object information and the vegetation index value to determine the harvest time of the crop. In claim 5,
The reflectance includes reflectance Red for the red region of the visible ray band and reflectance NIR for the near-infrared band,
The vegetation index value NDVI is given by the following formula:
NDVI = (NIR-Red)/(NIR+Red)
Growth analysis method, characterized in that calculated by. In claim 1,
when the detected object is a seedling, obtaining crop growth information and location information; and
Determining whether the seedling is defective based on the obtained growth information and location information
Further comprising, the growth information growth analysis method, characterized in that it comprises at least one of leaf length, leaf width, and leaf type index. In claim 1,
The step of detecting the analysis target object based on the obtained deep learning model,
dividing the overall coordinates of the acquired image into a plurality of grid cells;
generating two bounding boxes per one grid cell, and calculating a reliability that is a probability that an object is contained in the bounding box;
classifying a class for identifying which object the image in each grid cell is included in; and
Detecting the analysis target object based on the calculated reliability and the classified class
Growth analysis method comprising a. In claim 1,
obtaining a growth profile from the external database;
obtaining growth measurement information;
analyzing the correlation between the growth profile and the growth measurement information;
supplementing the setting information of the growth model based on the correlation analysis result; and
Storing the supplemented setting information in the external database
Further comprising, the setting information is a growth analysis method, characterized in that it comprises at least one of temperature information, humidity information, carbon dioxide concentration information and electrical conductivity. an imaging system that provides RGB images, DEPTH images and RGN images captured by 3D DEPTH cameras and spectroscopic cameras;
An Ethernet communication module equipped with a pre-trained deep learning model to generate an image file based on the image obtained from the image capturing system, detect and classify an object from the obtained image, and use according to the classification processing result an image analysis system that acquires from an external database through, detects an analysis target object through machine learning based on the acquired deep learning model, and analyzes a maturity level corresponding to the detected object; and
Measurement information analysis system for determining crop harvest time based on the analyzed maturity level
A growth analysis system comprising a. 11. The method of claim 10,
The image capturing system transmits the generated image file to a server located in a cloud environment through the Ethernet communication module, and the image file is stored in a database interworking with the server. 11. The method of claim 10,
The image analysis system,
a USB communication module for communication with the 3D DEPTH camera; and
Wi-Fi communication module for communication with the spectroscopic camera
Growth analysis system comprising a. In claim 10,
Further comprising a sugar content measurement system for measuring a sugar content corresponding to the detected object, wherein the measurement information analysis system further analyzes the maturity level based on the measured sugar content. In claim 12,
The image analysis system,
an object information extraction unit for extracting object information corresponding to the detected object based on the image obtained from the 3D DEPTH camera; and
Vegetation index calculation unit for calculating a vegetation index value based on the reflectance calculated based on the image information obtained from the spectroscopic camera
including,
The object information includes at least one of size information, shape information, growth information, and chromaticity information,
Growth analysis system, characterized in that the harvest time of the crop is determined further based on the object information and the vegetation index value. 15. In claim 14,
The reflectance includes reflectance Red for the red region of the visible ray band and reflectance NIR for the near-infrared band,
The vegetation index value NDVI is given by the following formula:
NDVI = (NIR-Red)/(NIR+Red)
Growth analysis system, characterized in that calculated by. In claim 10,
The measurement information analysis system,
When the detected object is a seedling, a defect determination unit that determines whether the seedling is defective based on crop growth information and location information
It further comprises, wherein the growth information is a growth analysis system, characterized in that it comprises at least one of a leaf length, a leaf width, and a leaf type index. In claim 10,
The image analysis system,
After dividing the entire coordinates of the obtained image into a plurality of grid cells, generating two bounding boxes per one grid cell, calculating the reliability that is the probability that an object is in the bounding box, and the image in each grid cell is Classification of a class for identifying which object is included in the growth analysis system, characterized in that it comprises an object detection unit for detecting the object to be analyzed based on the calculated reliability and the classified class. In claim 10,
The measurement information analysis system,
Further comprising a growth model update unit that supplements the setting information of the growth model by analyzing the correlation between the growth profile and the growth measurement information,
The setting information is a growth analysis system, characterized in that it includes at least one of temperature information, humidity information, carbon dioxide concentration information, and electrical conductivity.

Images