KR20220022961A - Advanced crop diagnosing system - Google Patents

Abstract

The present invention relates to a crop growth analyzing system using an RGB-Depth image and, more specifically, to a crop growth analyzing system, which generates a bag file (.bag file) in which filming environment data including an RGB-Depth camera filming image of a camera unit and a Depth sensor value is compressed to store the bag file in a database unit and supports a server unit to extract and image-process the bag file, thereby improving an image recognition rate to 95% or higher.

Description

Improved crop growth analysis system {Advanced crop diagnosing system}

The present invention is a crop growth analysis system using RGB-Depth images. Specifically, a bag file (.bag file) is created by compressing the photographing environment data including the depth sensor value together with the RGB-Depth camera photographed image of the camera unit. It is a crop growth analysis system that improves the image recognition rate to more than 95% by storing it in the department and supporting the server part to extract it and process the image.

The first research on artificial intelligence began as an academic interest in whether machines could think on their own, and was developed by neurosurgeon Warren McCullick into neural networks and reticulated neuron models, and Frank Rosenblatt's perceptron theory and 'XOR' The problem was presented and progress was made one step further. In particular, the development of multi-layer perceptron theory, big data technology, and high-speed Internet ensured breakthroughs in the field of artificial intelligence. Finally, in 2006, Jeffrey Hinton announced the deep learning deep neural network and the confrontation between Sedol Lee's 9th-dan AlphaGo and artificial intelligence and deep learning were the main differences in our society. It was an incident that informed us that we were deeply in.

However, it was not easy to introduce artificial intelligence in agriculture, a representative traditional industry, and until recently, only basic research was conducted, and only automation technology using the senses of farmers and specialized robots was used.

Meanwhile, Korean Patent Publication No. 10-2131941 proposed as a crop growth measurement technology has improved convenience so that sensors and cameras are driven along rails installed in greenhouses, and Korean Patent Publication No. 10-2114384 is an image-based crop. It was difficult to apply the mobile app and device for measuring growth data to the field, but Korean Patent Publication No. 10-2012-0075559 received environmental information and crop growth information in the house with a sensor, but the height corresponding to the growth information This is a technology that automates the movement by moving and photographing crops at the corresponding height according to the movement, analyzing the images, comparing the normal growth of the crops, analyzing the causes of abnormalities, and controlling the actuators corresponding to them to be executed. It has been evaluated as a rudimentary automation device technology to the extent that it was developed. Along with this, the latest mobile phones with RGB-Depth cameras have started to be released recently.

Korea Patent Publication No. 10-1763835 developed by the present inventor identifies each organ image such as leaf, stem, flower room, fruit, etc. of a crop object from a crop community as a video image to measure the growth of the first and second cameras and greenhouses. The environmental information collection system is combined and the collected information provides a crop growth measurement system using full-scale image images processed by the information processing device. And the image was analyzed based on the edge information, but the amount of growth was measured by modularizing it with an equation considering the temperature. In addition, Korea Patent Publication No. 10-1832724, which has further improved this, transmits and stores color RGB images and IR (depth) images captured by a smart terminal to the server, so that the length of each crop organ including leaves, nodes, flower rooms, and fruits; The width and thickness were measured to diagnose the growth rate and predict the harvest, but the stored color RGB images and IR (depth) images still could not solve the pixel defect problem during the extraction and analysis process, so the environmental data at the time of shooting could not be reflected. Therefore, in the image processing process, when using a normal jpg file, it was difficult to secure a certain quality, and the processing process for precision work became more complicated and the analysis results according to the workers were different.

Meanwhile, R-CNN (regions with convolutional neuron networks) among image recognition technologies recognizes objects by inputting data and labels through images, sets arbitrary BBs (bounding boxes) through selective search of hierarchical structures, and As a technology to recognize images by comparing IOUs, the Fast R-CNN method is an improved learning method in which the cnn is reduced to one and the region proposal is moved to the feature map. The Mask R-CNN method has the same basic structure as the Faster R-CNN method, but by using the ROI ALIGN method instead of the ROI POOLING), the mask and the clash prediction are separated and more accurate It has been difficult to find support for data processing, but it has been difficult to find cases where it has been specifically used in a crop growth analysis system so far.

As such, although all of the crop growth measurement techniques used so far have advanced hardware, strong improvement has been required in terms of software.

Korean Patent Publication No. 10-1763835 Korean Patent Publication No. 10-1822410 Korean Patent Publication No. 10-2114384 Korean Patent Publication No. 10-2012-0075559 Korean Patent Publication No. 10-2020-0029657

「Indoor smoke detection using RGB-Depth camera」 (JKIECS, vol. 9, no. 2, 155-160, 2014)

In order to solve the above problems, the present invention is to support image processing by storing the shooting environment data including the depth sensor value at the time of shooting necessary for the analysis of the captured image together with the RGB and the depth image.

In addition, the present invention intends to present an improved image processing module for precise measurement.

In order to solve the above problems, the present invention provides a crop growth analysis system comprising a camera unit, a server unit and a database unit, wherein the camera unit includes: a camera capable of acquiring an RGB image and a depth image; and a correction module that extracts a depth sensor value by correcting pixel defects of the RGB image and the depth image, wherein the correction module generates metadata including the depth sensor value, and then converts the metadata, the RGB image and the depth image. It can be provided as a feature of generating a bag file (.bag file) by compressing it together.

In addition, the server unit of the crop growth analysis system of the present invention includes an image acquisition unit that acquires images from a camera, an image identification unit that receives compressed files stored from the database unit and separates images and metadata, and analyzes the separated images and metadata. An image analysis unit for measuring crop organs, and a data storage unit for storing the measured data value in a database unit may be provided.

In addition, the present invention crop growth analysis system image identification unit can provide that consisting of an image processing module that sequentially processes faster R-CNN and mask R-CNN methods.

In addition, the database unit of the present invention may provide that including the deep learning big data storing the prior learning data.

The present invention also provides a method of using a crop growth analysis system comprising a camera unit, a server unit, and a database unit, wherein the camera unit includes a correction module, wherein the correction module generates metadata including a depth sensor value, and the metadata , it is possible to provide a crop growth analysis method characterized in that the RGB image and the depth image are compressed together to generate and use a bag file (.bag file).

The present invention stores data including metadata including the depth sensor value corrected for pixel defects along with RGB and depth images and uses it for analysis, so that even if the depth sensor value of the image, which could only be checked in real time, is analyzed later, it is precisely Support to measure crop organs.

The present invention compresses and generates data including metadata including the depth sensor value into a bag file (.bag file), and transmits and stores it on the cloud, so it is possible to achieve cost reduction by quickly and accurately supporting image processing speed. do.

The present invention can support more improved precise object separation and measurement by designing as an image processing module that sequentially processes the first faster R-CNN and the second mask R-CNN method during image processing of a captured image including metadata.

Figure 1 shows the outline of the inventor's Patent Registration No. 10-1832724 introduced in the background, Figure 2 shows the entire specific processing process of the patent.
3 shows the entire time-series processing process of the present invention.
4 illustrates a naming rule of a bag file (.bag file) generated when storing a captured image according to the present invention. By using the shooting location and time information for naming, it is possible to support fast information processing.
5 is an example in which the RGB image (A) and the depth image (B) are separated and displayed from the bag file stored in the database unit of the present invention.
6 and 7 are primary images by detecting the organs of tomato and melon crops from the RGB image separated from the bag file of the present invention in a faster r-cnn method, and selecting tagging. The processing process is illustrated.
8 and 9 are a process of applying the secondary mask r-cnn method to the object subjected to the primary image processing in FIGS. 6 and 7 of the present invention, segmentation and background removal It exemplifies the exact separation of objects.
10 is an illustration showing the organs and items to be analyzed during the analysis of crop growth according to the present invention, exemplifying tomatoes and melons.
11 illustrates the length and width analysis process in units of 1 cm along the bending angle of the crop organ using the depth sensor value extracted from the bag file of the present invention.
12 is an example of displaying the results of analysis for each organ from an image of a specific crop according to the present invention.
13 is an example of diagnosing, predicting, and displaying the growth state based on the amount of change in growth for each organ of a specific crop of the present invention.
14 is a display of melon growth results as an example of the diagnosis and prediction of FIG. 13 of the present invention.
15 illustrates an operation screen of a camera shooting program implemented in a smartphone app manual as an embodiment of the present invention.
16 illustrates an image analysis result screen implemented in a smartphone app manual as an embodiment of the present invention.

Among the crop organs used as the measurement object of the present invention, a leaf or a leaf is a leaf blade or a leaf body, and functionally, it is a flat-shaped organ that photosynthesizes by receiving sunlight, and a leaf tree ( The number of leaves) and leaf area (calculated by measuring leaf length, which is the length of a leaf, and leaf width, which is the width) are analysis variables.

A flower room is a crop organ, also called a flower shop, and the flowers blooming in the flower room become fruits after fruiting. For each crop, the proper number of flower beds and flowers must be thoroughly managed at the stage of harvesting to ensure marketability at harvest, so it is used as a major item for proposing on-site measures after measuring growth.

The fruit is the most important crop organ of the orchard, and the length of the straight line, the exaggeration, the width, and the number of fruits are used as main variables for diagnosis and harvest prediction.

The stem is a crop organ that supports the leaves, is properly arranged to effectively expose the leaves to sunlight, supports many organs such as leaves, flower beds, fruits, etc. It is a major variable related to the overall growth process and harvest, such as generating new tissues through

On the other hand, the petiole (patiole) is a crop organ with a stalk function that connects the stem and the leaf (the leaf body), and changes the leaf to face the sunlight.

Hereinafter, the present invention will be described in detail.

First, FIG. 3 shows the entire process of the present invention, and will be described in comparison with Patent Registration No. 10-1832724 of FIGS. 1 and 2, which is a prior art introduced in the background art.

The crop growth analysis system of the present invention consists of a camera unit 100, a server unit 200, and a database unit 300 as shown in FIG. In connection, remote data transmission can be made in a cloud environment. That is, the present invention provides a crop growth analysis system comprising a camera unit, a server unit, and a database unit, the camera unit comprising: a camera capable of acquiring an RGB image and a depth image; and a correction module that extracts a depth sensor value by correcting pixel defects of the RGB image and the depth image, wherein the correction module generates metadata including the depth sensor value, and then converts the metadata, the RGB image and the depth image. It is characterized in that it is compressed together to create a bag file (.bag file).

In addition, the server unit of the crop growth analysis system of the present invention includes an image acquisition unit that acquires images from a camera, an image identification unit that receives compressed files stored from the database unit and separates images and metadata, and analyzes the separated images and metadata. It is characterized in that it comprises an image analysis unit for measuring the crop organs, and a data storage unit to store the measured data value in the database unit.

In addition, the image identification unit of the crop growth analysis system of the present invention is characterized in that it consists of an image processing module that sequentially processes faster R-CNN and mask R-CNN methods.

In addition, the database unit of the present invention is characterized in that it includes deep learning big data storing the pre-learning data.

The present invention also provides a method of using a crop growth analysis system comprising a camera unit, a server unit, and a database unit, wherein the camera unit includes a correction module, wherein the correction module generates metadata including a depth sensor value, and the metadata , it is possible to provide a crop growth analysis method characterized in that the RGB image and the depth image are compressed together to generate and use a bag file (.bag file).

The overall time flow of the analysis system of the present invention is as follows.

As shown in FIG. 3, the present invention starts with capturing RGB images and depth images of the camera unit. The captured image generates metadata including the shooting environment data such as the corrected depth sensor value after checking for pixel defects in the camera correction module.

In addition, the correction module compresses the generated metadata, RGB image, and depth image in the form of a bag file (.bag file), which is stored in the database unit through the image acquisition unit of the server unit.

In addition, the image identification unit of the server unit extracts the bag file (.bag file) already stored in the database unit, separates the RGB image, the depth image, and the metadata (S5), and the image analysis unit uses the first Faster R-CNN and the second It is processed by putting it into the image processing module composed of the Mask R-CNN module (S6). ID is assigned to the processed image (S7), growth is measured using the depth sensor value (S8), and the separated image and measured value are stored again in the database unit (S9).

After that, if necessary, using the pre-learning data (deep learning big data) stored in the database unit and the measurement values of the accumulated managed crop organizations, the growth change analysis (S11), growth diagnosis and change amount prediction (S12) process are performed, and then the on-site action method is presented (S13). In this process, the camera unit, such as a smartphone including a camera, may notify and display not only the analysis result but also diagnosis, harvest prediction, and action method.

Hereinafter, each processing step will be described in detail.

The present invention takes an RGB image and a depth image through the camera unit 100, and generates and compresses metadata including a depth sensor value corrected by checking pixel defects in the depth image image and shooting environment data. do. In the present invention, it was compressed using a bag file format used for precision robot control because of its high degree of compression and excellent usability and stability.

The function of generating and compressing metadata including the pixel defect check and adjustment depth sensor value and the photographing environment data may be performed through a correction module.

In addition, it was more preferable that the RGB image and the Depth image were photographed at a distance of 0.6 to 1.2 mm from the camera in the embodiment experimented by the inventor in actual field photography.

That is, the present invention uses RGB images and Depth image images together when analyzing crop growth in the known patents of the background art, but these images analyzed in the known patents are stored in the transmission database unit as they are in the form of normal image files such as JPG format. Therefore, in this case, after a predetermined period of time has elapsed, the server unit calls it again and it is impossible to reflect the environmental data at the time of image processing during image processing, so the analysis result is significantly different from the real-time analysis, and the image recognition rate is also very difficult to exceed 90%.

However, the present invention compresses and stores the metadata including the shooting environment data, such as the depth sensor value, in a bag file together with the RGB image and the depth image image by using the correction module, so that it is extracted from the server after a predetermined time has elapsed. Even when analyzing , since the shooting environment data such as the depth sensor value is used for mapping, there is no image distortion no matter who analyzes anytime, anywhere, and the image recognition rate after image processing was able to achieve more than 95%. Through this, it supports the analysis more precisely than before when comparing the growth rate by measuring the crop organs.

In addition, the camera unit may be an attachable camera detachable to a smart terminal capable of photographing RGB images and depth images, or may be a smartphone manufactured as an integrated device. According to the trend of demanding 5G communication technology and VR function, the case of including RGB-Depth camera by default in high-end smartphones is increasing recently.

In addition, the camera unit may control the captured RGB image and the depth image to be stored in the database unit through the server unit. As an example, as shown in FIG. 15, the organs of growing crops are photographed at a predetermined fixed position through a smartphone app, and after checking pixel defects in the depth image image at the time of shooting, metadata including the corrected depth sensor value is converted into RGB image and depth It can be created and transmitted as a bag file together with the video and stored in the database unit.

In addition, the metadata may be generated by further adding photographing time information necessary for information processing in the server unit.

Hereinafter, the server unit 200 will be described.

The server unit 200 of the present invention includes an image acquisition unit that acquires an image from a camera unit, an image identification unit that receives a compressed file stored from the database unit and analyzes the image and metadata, and analyzes the separated image and metadata to produce crops. It may include an image analysis unit for measuring an organ, and a data storage unit for storing the measured data value in a database unit, and may be provided in the form of a smartphone app or a computer program.

The image identification unit of the present invention calls the bag file stored in the database, and then separates the metadata, RGB image, and depth image including the depth sensor value and the shooting environment data (S5) and puts it in the image processing process.

In addition, the image analysis unit of the present invention is responsible for the fine image processing process in the first and second phases. The first is a Faster R-CNN method and the second is a Mask R-CNN method. It can be provided as an image processing module sequentially combined, It can be supported to assign and manage IDs for each object by accumulating with a time difference from the time of shooting to the present.

In addition, in the Faster R-CNN method of the present invention, as in the examples of FIGS. 8 and 9 , an object is detected from the original image, selected by tagging, and then the object is precisely segmented through the Mask R-CNN method. separate In this process, metadata including the corrected depth sensor value supports more precise separation, the separated object is measured for each institution measurement item as shown in FIG. 11, and the measurement result can be supported to be displayed on the display screen as shown in FIG. 12. .

After the image processing and growth rate measurement as described above, when the image and the measurement value are stored in the database, it is possible to provide measurement results for each crop organ such as a leaf, a stem, a flower room, and a fruit, as shown in FIG. 13 . Through this, it supports to easily diagnose and manage the growth status of a specific crop in a specific location given an ID.

In addition, when a harvest prediction algorithm for a specific crop of the server is applied, it is possible to provide a harvest prediction from the measured values stored in the database.

Hereinafter, the database unit 300 will be described in detail.

The database unit 300 may include a pre-learning data unit, a basic data unit, and a measurement data unit, and may further include a harvest prediction algorithm.

The pre-learning data unit is built by recording, storing, and learning the RGB images of crop organs in the analysis target area so that the crop growth analysis system of the present invention can be applied in earnest. You can build and utilize deep learning big data by assigning a specific ID to each crop organization using the image of the crop and managing it cumulatively.

The basic data unit is a data unit that stores the metadata including the depth sensor value received from the camera unit, a bag file in which the RGB image and the depth image are compressed together, as it is, and is called from the camera unit or the server unit at any time. managed as much as possible.

In addition, the measurement data unit is a data unit that stores the processed images and measurement values assigned IDs through image processing to which the crop growth analysis system of the present invention is applied in earnest. The measurement values and images managed in time series are used for diagnosis and harvest prediction. It is used for presenting on-site measures by referring to database data such as pre-learning data.

100: camera unit 200: server unit
300: database unit

Claims (5)

In the crop growth analysis system comprising a camera unit, a server unit and a database unit,
The camera unit,
A camera capable of acquiring RGB images and depth images; and
A correction module for extracting a depth sensor value by correcting pixel defects of the RGB image and the depth image;
The correction module generates metadata including the depth sensor value, and then compresses the metadata, RGB image, and depth image together to generate a bag file (.bag file), crop growth analysis system According to claim 1,
The server unit,
an image acquisition unit for acquiring an image from a camera; an image identification unit that receives the compressed file stored from the database unit and separates the image and metadata; an image analysis unit that analyzes and measures the separated image and metadata; and a data storage unit configured to store the measured data value in the database unit; 3. The method of claim 2,
The image identification unit,
Crop growth analysis system, characterized in that it consists of an image processing module that sequentially processes faster R-CNN and mask R-CNN methods The crop growth analysis system according to claim 1, wherein the database unit includes deep learning big data storing prior learning data.
In the method of using a crop growth analysis system consisting of a camera unit, a server unit and a database unit,
The camera unit includes a correction module, wherein the correction module generates metadata including a depth sensor value and compresses the metadata, RGB image, and depth image together to generate and use a bag file (.bag file) Crop growth analysis method

Images