KR20240071483A - Apparatus and method for monitoring plants using ai machine vision - Google Patents

Abstract

A plant monitoring method using AI machine vision according to the present invention includes the process of identifying plants based on an image of at least one cultivation pot or an image of a plant corresponding to the at least one cultivation pot; And it may include a process of transmitting the image or monitoring information of the identified plant to a mobile terminal.

Description

Plant monitoring method and device using AI machine vision {APPARATUS AND METHOD FOR MONITORING PLANTS USING AI MACHINE VISION}

The present invention relates to a method and device for monitoring plants using AI machine vision, and more specifically, to a method and device for recognizing and monitoring plants inside a plant cultivator using AI machine vision technology.

The content described in this section simply provides background information about the present invention and does not constitute prior art.

Recently, with the rapid development of ICT, smart farm technology that can monitor and control various variables related to the growth of crops, such as temperature, humidity, and amount of sunlight, is attracting attention as a future agricultural technology, and plant cultivation devices are used for indoor agriculture and It is being used in relation to this. A plant cultivator may refer to a device that grows seeds or seedlings into mature plants by artificially controlling the environment of the cultivation space, such as temperature, humidity, and amount of sunlight. For example, a plant cultivator artificially creates a lighting environment through electronic control of the light source module, artificially creates a moisture environment and nutrient environment through electronic control of the culture medium module, and artificially creates airflow through electronic control of the air conditioning module. It is possible to create an environmental and thermal environment.

Meanwhile, in relation to smart farm technology, unlike plant cultivators being used in the agricultural industry, plant cultivators that can be used as home appliances are being used in the recent market to meet the needs of consumers who want a well-being life and high-quality food ingredients. . However, since users of home plant cultivators may not be skilled in plant cultivation, it is necessary to provide information necessary for plant cultivation. In other words, conventional home plant growers provide some simple information about the plants being grown, but in order to provide some information, there is an inconvenience in that the user has to enter identification information such as the name of the plant and information about the location, and the actual plant There is a problem in that intuitive monitoring of the plant growth process, including images, is not provided.

Korean Patent Publication No. 10-2016-0130589 (Plant cultivation apparatus and method, November 14, 2016)

As mentioned above, the plant growth process can be intuitively monitored by recognizing the cultivated plant using AI machine learning and transmitting the plant image to the user's mobile terminal without inputting separate information about the plant being cultivated. The purpose is to provide a monitoring method and device.

In order to achieve the above object, a plant monitoring method using AI machine vision according to the present invention includes the process of identifying plants based on an image of at least one repot or an image of a plant corresponding to the at least one repot; And it may include a process of transmitting the image or monitoring information of the identified plant to a mobile terminal.

Depending on the embodiment, the process of identifying the plant may include: identifying a feature map for an image of the at least one repot or an image of a plant corresponding to the at least one repot; may include.

Depending on the embodiment, the process may include: identifying a feature map for the normal state of the plant based on images of the plant corresponding to the at least one repot taken at intervals of a preset period of time; It may further include.

In addition, a plant monitoring device using AI machine vision according to the present invention includes: a control unit that identifies plants based on images of at least one cultivation pot or images of plants corresponding to the at least one cultivation pot; And a communication unit that transmits the image or monitoring information of the identified plant to a mobile terminal; may include.

Depending on the embodiment, the control unit may identify a feature map for the image of the at least one redistribution or the image of the plant corresponding to the at least one redistribution.

Depending on the embodiment, the control unit may identify a feature map for the normal state of the plant based on images of the plant corresponding to the at least one redistribution photographed every preset period of time.

The plant monitoring method and device using AI machine vision provided as an embodiment of the present invention can eliminate the inconvenience of users inputting additional information about plants grown using AI machine vision.

In addition, the plant monitoring method and device using AI machine vision can provide monitoring to the user by transmitting images of cultivated plants to a mobile terminal.

In addition, by using plant images used for plant identification in AI machine vision for monitoring, waste in terms of data storage capacity can be reduced and data processing for additional image shooting or deletion can be reduced.

Meanwhile, the effects of the present invention are not limited to the effects mentioned above, and various effects may be included within the range apparent to those skilled in the art from the contents described below.

Figure 1 is a diagram for explaining a plant monitoring device using AI machine vision according to an embodiment of the present invention.
Figure 2 is a diagram for explaining a plant cultivation system including a plant monitoring device using AI machine vision according to an embodiment of the present invention.
Figure 3 is a diagram for explaining a plant monitoring method using AI machine vision according to an embodiment of the present invention.
Figure 4 is a diagram for explaining a method of identifying plants through AI machine vision based on images of redistribution according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a plant image provided to a mobile terminal in a plant cultivation system according to an embodiment of the present invention.
Figure 6 is a diagram for explaining the form of redistribution according to an embodiment of the present invention.

Hereinafter, the terms used in this specification will be briefly explained, and the structure and operation of a preferred embodiment of the present invention will be described in detail as specific details for practicing the present invention. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

The word “exemplary” is used herein to mean “used as an example or illustration.” Any embodiment described herein as “exemplary” should not necessarily be construed as preferred or as having an advantage over other embodiments.

Additionally, the term “unit” used in the specification refers to a hardware element such as software, FPGA, or ASIC, and the “unit” performs certain roles. However, “wealth” is not limited to software or hardware. The “copy” may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, “part” refers to elements such as software elements, object-oriented software elements, class elements, and task elements, processes, functions, properties, procedures, subroutines, and programs. Contains segments of code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within elements and “parts” may be combined into smaller numbers of elements and “parts” or may be further separated into additional elements and “parts”.

Additionally, all “units” in this specification may be controlled by at least one processor, and at least one processor may perform the operations performed by the “units” of the present disclosure.

Embodiments of the present specification may be described in terms of a function or a block that performs a function. Blocks that may be referred to as 'units' or 'modules' of the present disclosure include logic gates, integrated circuits, microprocessors, microcontrollers, memories, passive electronic components, active electronic components, optical components, and hardwired circuits. It is physically implemented by analog or digital circuits, etc., and can optionally be driven by firmware and software.

Embodiments herein may be implemented using at least one software program running on at least one hardware device and may perform network management functions to control elements.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

In Figure 1, a plant cultivation system including a plant monitoring device using AI machine vision is described, and in Figures 2 and 3, a plant monitoring method and device using AI machine vision are described. after. In Figures 4 and 5, a plant monitoring method using AI machine vision to identify plants using the shape or color of specific repots is explained.

Figure 1 is a diagram for explaining a plant cultivation system including a plant monitoring device using AI machine vision according to an embodiment of the present invention.

Referring to FIG. 1, the plant cultivation system 100 includes a plant monitoring device 110 that identifies at least one plant or calculates monitoring information for the identified plant, and at least one plant identified or observed in the plant monitoring device 110. A management server 120 that collects identification information or monitoring information about one or more plants through wired or wireless communication from the plant monitoring device 110 and analyzes the collected information, and identification information, monitoring information, and analysis results about the plants. It may include at least one mobile terminal 130 that can request through wireless communication, receive the requested information and reception results, and output them to the user.

Depending on the embodiment, the wireless communication commonly used in the plant cultivation system 100 may apply Internet of Things (IoT) communication technology, but in addition to various types of long-distance wireless communication technologies such as mobile communication and satellite communication, Bluetooth, etc. The same short-range wireless communication, etc. can be applied, and embodiments of the present invention are not limited to the type of communication technology. For example, the wireless communication commonly used in the plant cultivation system 100 may use commercial RF communication technology such as ultra-wideband (UWB) or Wi-Fi (WiFi) communication, and may use multiple transmission and reception. Antennas (Multiple Input Multiple Output, MIMO, multiple antennas) can be used together.

Depending on the embodiment, the plant monitoring device 110 may transmit plant identification or monitoring information to the mobile terminal 130 through the management server 120, but may transmit plant identification or monitoring information to the mobile terminal 130 rather than through the management server 120. ) You can transmit identification information or monitoring information about plants directly.

Depending on the embodiment, the plant monitoring device 110 may be included in a plant grower, and the plant grower may refer to devices that can electronically control the cultivation environment of plants, including lighting control, temperature control, culture medium control, and air conditioning control. , temperature control, etc., can automatically control the appropriate cultivation environment according to the variety of the plant being cultivated.

Depending on the embodiment, the mobile terminal 130 can monitor the cultivation process of plants in real time using information received from the plant monitoring device 110, and at the same time, related information such as cultivation records is converted into a database and stored in the mobile terminal. It can be stored at (130). Furthermore, the user can remotely control the plant monitoring device 110 through the mobile terminal 130 to customize the plant cultivation environment. The mobile terminal 130 can request an image of a plant to be checked by the plant monitoring device 110 or identification information or monitoring information about the plant through an interface such as an application, and the plant monitoring device 110 or management Images of plants stored in the server 120 or identification information or monitoring information about plants may be received.

Figure 2 is a diagram for explaining a plant monitoring device using AI machine vision according to an embodiment of the present invention.

Referring to FIG. 2, the plant monitoring device 200 using AI machine vision is an image of a plant located inside a plant cultivator (not shown) or at least one cultivation pot (or cultivation container or cultivation kit) where individual plants grow. an AI machine vision module 210 that learns the plant that is the target of identification based on the image of the image, and at least one camera included in the sensing unit 220 by using the learning result of the AI machine vision module 210 A sensing unit ( 220), a memory 240 that stores information about the learning results of the AI machine vision module 210 and information about the plant to be identified, and an image about the plant or status information about the plant by an external management server or mobile terminal, or It may include a communication unit 250 capable of transmitting growth information.

Depending on the embodiment, the AI machine vision module 210 may remove noise from image data generated by a camera, distinguish areas of plants to be identified, and extract outlines.

Depending on the embodiment, the AI machine vision module 210 can identify the species of the plant to be identified through AI deep learning and learn plant identification information and images from plant image data using a deep neural network. You can. In order to learn plant image data, preprocessing processes such as image annotation can be performed, and the annotation displays a shooting frame aligned with the edges of the object in the image and a bounding box ( bounding box), a point that marks the object to be searched in the image, a keypoint that identifies the outline of the object to identify the shape of the object to be detected, a keypoint containing polygons and points, and a polyline ( It may include at least one of data formats such as poly line and polygon.

Depending on the embodiment, the AI machine vision module 210 may include a PC card such as a frame grabber, an interface, etc. for receiving images generated from a camera. The AI machine vision module 210 can generate a feature map for a plant through an encoder (may be referred to as an encoder or encoder neural network) that can receive a plant image (RGB image), The feature map for plants can be referred to as the result of a convolution operation from the pixel information of the RGB image to extract the features of the input RGB image. The feature map for the plant can be input to a decoder (may be referred to as a decoder neural network) and restored to the RGB image size. The feature map for plants through this encoding and decoding has information about the surrounding environment and objects other than the image of the plant itself removed, and can predict the shape, location, part information, disease information, identification information, etc. of the plant detected in the image. It can be understood that possible feature points have been extracted. Depending on the embodiment, the AI machine vision module 210 may include a backbone network to extract features for the RGB image, and may utilize additional positional encoding to contain each relative position information in the RGB image. Additionally, pixel information of each RGB image may be classified as sequential data and may include a transformer architecture for sequential data processing, and each data may be composed of a feature map with a specific dimension. Afterwards, through a query, each object can be classified to indicate the class of each object, where the object exists, and the location of the bounding box.

Depending on the embodiment, the AI machine vision module 210 may include an image processing device using a camera, and may identify plants to be identified only through image processing without using AI deep learning. Specifically, the brightness of the image may be adjusted. After adjustment, image data processing can be performed through image segmentation techniques such as thresholding and segmentation/integration of areas based on image uniformity (edge-based segmentation), and connectivity analysis from the segmented images. The characteristics of plants to be identified can be extracted using extraction techniques such as . Afterwards, plants can be recognized and classified from the extracted images.

Depending on the embodiment, the control unit 230 uses FOV (Field of View), WD (Working Distance), Resolution, DoF (Depth of Field), etc. to generate an image of a plant to be identified from at least one camera. To control at least one camera or lighting (not shown), etc. Additionally, the control unit 230 may pre-process images of plants and apply filters to control brightness, color, etc. The control unit 230 can specify the plant to be identified after image preprocessing and identify the plant based on pattern matching, color, size, shape, etc. The control unit 230 can control information about the identified plant and images generated from the camera to be transmitted to an external management server or mobile terminal through the communication unit 250.

Depending on the embodiment, the communication unit 250 may use commercial RF communication technology such as ultra-wideband (UWB) or Wi-Fi (WiFi) communication, and may use multiple transmit/receive antennas (Multiple Input Multiple Output). , MIMO, and multiple antennas) can be used together.

Figure 3 is a diagram for explaining a plant monitoring method using AI machine vision according to an embodiment of the present invention.

Referring to FIG. 3, a plant monitoring device using AI machine vision can identify plants based on the image of the plant or the image of the plant generated through at least one camera (S310).

Depending on the embodiment, a plant monitoring device using AI machine vision may identify the shape or color of a cultivation pot using AI technology to distinguish plant species or monitor the state of the plant. For example, the shape of the redistribution may be a polygonal shape such as a circle, square, or hexagon, and the color of the redistribution may have at least one color or combination of colors, such as yellow, blue, or red. Depending on the embodiment, a plant monitoring device using AI machine vision identifies and learns the outline (or color) of a plant or cultivation pot and the corresponding plant from a previously learned plant image or image of a cultivation pot through AI deep learning. Through encoding, the feature map can be identified, and through the identified feature map, the plant or repot to be monitored and the corresponding plant can be identified.

Depending on the embodiment, a plant monitoring device using AI machine vision adjusts the image brightness of each plant or redistribution without using AI deep learning, then sets a threshold (Thresholding) and divides/integrates areas according to image uniformity ( Image data processing can be performed through image segmentation techniques such as edge-based segmentation, and plants or redistributions can be identified using extraction techniques such as connectivity analysis from segmented images.

Depending on the embodiment, a plant monitoring device using AI machine vision may be included in a plant cultivator, and at least one plant being cultivated inside the plant cultivator can be re-ported or re-ported to a re-port location using AI deep learning or image processing. Information that can monitor the plant species or plant growth status can be identified, and by providing notifications through a mobile terminal, the user can check the information and supply water or nutrients or harvest it. .

Depending on the embodiment, a plant monitoring device using AI machine vision can distinguish plant species by the shape or color of the top of the cultivation pot through AI deep learning or image processing, and capture images of plants at preset time intervals to store them in memory. It can be saved in .

According to the embodiment, a plant monitoring device using AI machine vision can learn a plant image in a normal state by shooting and learning an image of an identification target plant at a preset time period and identify a feature map in a normal state, and Based on the image of the plant to be identified and the feature map of the normal state at each time period, the analysis result of the quality of the plant to be identified or the deficiency element (at least one of water, nutrients, or amount of light) can be identified.

Depending on the embodiment, the cultivation pot may contain elements necessary for growing plants, such as seeds and media, in an integrated manner, and the cultivation shelf (grow stand) may include a plurality of cultivation pots arranged and divided into cells in a grid pattern. .

Depending on the embodiment, a plant monitoring device using AI machine vision may transmit images and monitoring information of identified plants to a management server or mobile terminal (S320).

Depending on the embodiment, a plant monitoring device using AI machine vision may store plant images used to extract feature maps of plants to be identified, or transmit them to a management server or mobile terminal. By using plant images used for plant identification in AI machine vision for monitoring, waste in terms of data storage capacity can be reduced and data processing for additional image shooting or deletion can be reduced.

Depending on the embodiment, the type and growth state of the plants inside can be checked without opening the door of the plant cultivator, which is manufactured in a sealed manner to maintain a constant cultivation environment such as temperature and humidity, through a plant monitoring device using AI machine vision. You can automatically recognize the location of the plant and monitor its growth status.

Figure 4 is a diagram for explaining a method of identifying plants through AI machine vision based on images of redistribution according to an embodiment of the present invention, and Figure 5 is a diagram showing a portable terminal in a plant cultivation system according to an embodiment of the present invention. This is a drawing to explain the plant image provided. Additionally, Figure 6 is a diagram for explaining the form of redistribution according to an embodiment of the present invention.

Referring to FIGS. 4 to 6 , the cultivation shelf 410 may include at least one redistribution device 420 arranged according to a preset layout. As shown in FIG. 6, the shape of the redistribution port 420 may have a polygon such as square (a), circle (b), or hexagon (c), and the colors of the redistribution port 420 are yellow, blue, It may have at least one color, such as red. A plant monitoring device using AI machine vision can identify plants contained in a cultivation pot by identifying the shape or color of the cultivation pot.

Various embodiments of the present invention include one or more instructions stored in a storage medium (e.g., memory) readable by a machine (e.g., a vehicle generated data recorder or computer). It can be implemented as software. For example, the processor of the device may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term is used when data is semi-permanently stored in a storage medium. There is no distinction between temporary storage and temporary storage. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this specification may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices. It can be distributed (e.g. downloaded or uploaded) directly between devices (e.g. smartphones) or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily. Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

Claims (6)

A process of identifying a plant based on an image of at least one repot or an image of a plant corresponding to the at least one repot; and
A plant monitoring method using AI machine vision, including a process of transmitting the image or monitoring information of the identified plant to a mobile terminal. According to paragraph 1,
The process of identifying the plant is,
Identifying a feature map for an image of the at least one redistribution or an image of a plant corresponding to the at least one redistribution; Plant monitoring method using AI machine vision, including. According to paragraph 1,
A process of identifying a feature map for the normal state of a plant based on images of the plant corresponding to the at least one repot taken every preset period of time; A plant monitoring method using AI machine vision, further comprising: a control unit that identifies a plant based on an image of at least one redistribution or an image of a plant corresponding to the at least one redistribution; and
A communication unit that transmits the image or monitoring information of the identified plant to a mobile terminal; Plant monitoring device using AI machine vision, including. According to clause 4,
The control unit,
A plant monitoring device using AI machine vision that identifies a feature map for an image of the at least one redistribution or an image of a plant corresponding to the at least one redistribution. According to clause 4,
The control unit,
A plant monitoring device using AI machine vision that identifies a feature map for the normal state of the plant based on images of the plant corresponding to the at least one redistribution taken every preset period of time.

Images