KR20210119843A - Monitoring device and monitoring method - Google Patents

Abstract

A monitoring device is provided. The monitoring device includes: an acquisition unit for acquiring a lichen image obtained by photographing lichen that grows on the outside of a building; and an analysis unit for analyzing air pollution around the building through the analysis of the lichen image.

Description

MONITORING DEVICE AND MONITORING METHOD

The present invention relates to an apparatus and method for observing moss and monitoring air pollution.

In the case of Korea, we are struggling to establish air quality management measures from an integrated perspective that considers the linkage between fine dust, climate change, and energy policies. are promoting together.

In the case of overseas, Indonesia is proposing 14 specific action plans in the transportation and urban planning sectors to achieve the Jakarta Clean Air 2030 roadmap.

According to these air-related trends, it is necessary to prepare a plan to efficiently analyze air pollution around buildings.

Korean Patent Publication No. 1051142 describes the generation of RGB composite images using raw data received from weather satellites, and the classification of yellow dust and anthropogenic air pollution using the composite images.

Korean Patent Publication No. 1051142

An object of the present invention is to provide a monitoring device and method for analyzing air pollution around a building using moss.

The monitoring device of the present invention includes: an acquisition unit for acquiring a moss image in which a moss (lichen) growing naturally on the exterior of a building is photographed; It may include; an analysis unit for analyzing the air pollution around the building through the analysis of the moss image.

The monitoring method of the present invention applies a moss image in which moss (lichen) that grows on the exterior of a building is photographed to a convolutional neural network (CNN), and uses the color characteristics of the moss image analyzed through the CNN. Air pollution can be monitored.

The monitoring method of the present invention learns the color of lichen that changes according to air pollution for each air pollution degree, and based on the learning, when a moss image containing moss is input, the color of the moss contained in the moss image is A moss classification model to be analyzed is generated, and the moss classification model may classify the color analyzed through the analysis of the moss image according to the pre-learned color for each air pollution degree.

The monitoring apparatus and method of the present invention can determine the air pollution around the building through the analysis of the moss existing on the exterior of the building.

The present invention can analyze the image of moss that grows naturally on the exterior of a building, and can determine the level of air pollution around a building, such as a building.

The present invention can provide a moss classification model that analyzes the air pollution state around the building by using an image of the moss existing on the exterior of the building.

1 is a schematic diagram showing a monitoring device of the present invention.
2 is a schematic diagram illustrating operations of an acquisition unit, a search unit, and an editing unit.
3 is a schematic diagram showing the VGG-16 architecture.
4 is a schematic diagram showing the operation procedure of the VGG-16 architecture.
5 is a schematic diagram showing a data set of moss.
6 is a visual representation of the color of moss in contact with different levels of contamination.
7 is a schematic diagram showing the operational steps of the VGG-16 architecture for image classification.
8 is a schematic diagram illustrating a pollution detection model using VGG-16.
9 is a graph showing classification accuracy.
10 is a graph showing classification accuracy.
11 is a graph illustrating recall.
12 is a graph showing F-Measure.
13 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when it is mentioned that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but another element in the middle. It should be understood that there may be On the other hand, in this specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also, in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Also in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It should be understood that the existence or addition of other features, numbers, steps, operations, components, parts or combinations thereof is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of a plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a schematic diagram showing a monitoring device of the present invention.

The monitoring device shown in FIG. 1 may include an acquisition unit 100 and an analysis unit 200 .

The acquisition unit 100 may acquire the moss image i0 in which the moss (lichen) 90 that grows on the outer surface of the building 10 is photographed.

The analysis unit 200 may analyze air pollution around the building 10 through the analysis of the moss image i0.

The monitoring device of the present invention may include an acquisition unit 110 , a search unit 130 , an editing unit 150 , an extraction unit 210 , a flattening unit 230 , and a classification unit 250 . The acquisition unit 110 , the search unit 130 , and the editing unit 150 may be provided in the acquisition unit 100 . The extraction unit 210 , the planarization unit 230 , and the classification unit 250 may be provided in the analysis unit 200 .

2 is a schematic diagram illustrating operations of the acquiring unit 110 , the search unit 130 , and the editing unit 150 .

The acquisition unit 110 may acquire a plurality of photographed images i1, i2, i3, ... from the photographing unit 300 photographing the exterior of the building 10 . For example, the acquisition unit 110 may be provided with a wireless communication module for wireless communication with the photographing unit 300 in real time. The acquisition unit 110 may acquire the photographed images i1, i2, i3, etc. captured by the photographing unit 300 in real time through a wireless communication module that communicates with the photographing unit 300 in real time.

The search unit 130 may search for a specific photographed image i3 including the moss 90 from among a plurality of photographed images. In order to promote control convenience and speed, the photographing unit 300 may photograph the exterior of the building according to a set rule or randomly, regardless of the presence or absence of moss. Accordingly, there may be an image in which moss is not included in the photographed image photographed by the photographing unit 300 . The search unit 130 searches for a specific captured image i3 containing moss by applying an image analysis technique, a deep learning technique, etc. to the plurality of captured images i1, i2, i3, etc. obtained by the obtaining unit 110. can

The editing unit 150 may create a moss image i0 by editing a specific captured image i3 including the moss 90 . For example, the editing unit 150 may generate an enlarged image k including only the moss area j by enlarging the area (moss area j) of the moss 90 included in the specific captured image i3. In this case, the moss image i0 may include an enlarged image k. For example, the moss image i0 may be the same as the enlarged image k. The entire area of the moss image i0 including the enlarged image k is filled with moss, and even if any pixel forming the moss image i0 is selected, the color of the moss can be recognized.

The analysis unit 200 may extract color information from the enlarged image k and analyze air pollution using the extracted color information.

According to the editing unit 150, the analysis unit 200 does not need to perform an additional operation for distinguishing the moss 90 from the specific photographed image i3. Accordingly, according to the editing unit 150 , the processing load of the analysis unit 200 may be reduced.

On the other hand, the extraction unit 210 provided in the analysis unit 200 may directly distinguish the moss area j from the specific photographed image i3 without the help of the editing unit 150 . According to the present embodiment, the editing unit 150 may be excluded. When the editing unit 150 is excluded, the specific photographed image i3 including the moss may be provided to the extraction unit 210 as the moss image i0 as it is. In other words, the specific photographed image i3 searched for by the search unit 130 may soon become the moss image i0.

In this case, the extraction unit 210 may directly classify or extract the moss area j in which the moss 90 exists from the specific photographed image i3 . Also, the extractor 210 may extract the color feature of the entire area or a partial area of the moss area j. Color information or color values of a plurality of image pixels forming the moss region j may be different from each other. The extractor 210 may output one representative value representing the color of the entire area or a partial area forming the moss area 90 by applying techniques such as standard deviation, dispersion, and planarization. In this case, one representative value output from the extractor 210 may correspond to the color feature.

The analysis unit 200 may analyze air pollution by using the color features extracted by the extraction unit 210 .

The present invention can monitor air pollution by using the property of moss that changes color according to air pollution. To monitor air pollution using the color of moss, the native location of moss must first be identified. In addition, the appearance of the moss should be photographed in a state where the colors can be clearly distinguished. The building 10 such as a building may satisfy all the corresponding conditions. Therefore, the exterior of the building 10 may be suitable as a target for monitoring air pollution using the color of moss.

In addition, when the moss 90 that grows naturally on the outer surface of the high building 10 such as a building is monitored, air pollution can be identified for each altitude.

The photographing unit 300 may be provided to photograph the moss that grows on the outer surface of the building 10 .

The photographing unit 300 may photograph the outer surface of the building 10 while flying in the air. As an example, the photographing unit 300 includes a flight module 310 that flies in the air using rotation of a propeller, etc., a camera 390 installed in the flight module 310 and optically photographing the outer surface of the building 10 , flight It may include an altimeter 330 for sensing the altitude.

The altimeter 330 of the photographing unit 300 may detect the flight altitude at a specific point in time when the exterior of the building is photographed.

The photographing unit 300 may provide a photographed image photographed at a specific time point and a flight altitude sensed at a specific time point to the acquisition unit 100 .

The acquisition unit 100 may acquire a moss image using the photographed image, and provide the moss image to the analysis unit 200 along with the flight altitude.

The analysis unit 200 may generate air pollution information for each altitude by matching the flight altitude to the air pollution analyzed for the moss image. The photographing unit 300 may photograph the outer surface of the building 10 using a camera 390 having an optical axis L extending in a direction perpendicular to the direction of gravity. The height of the outer surface of the building included in the photographed image photographed by the camera 390 having the optical axis L may directly correspond to the flight altitude of the photographing unit 300 . In other words, the building part included in the photographed image may be located at the same height as the photographing unit 300 . Accordingly, the air pollution state determined using the corresponding photographed image may indicate air pollution at the corresponding height point determined using the altimeter 330 or may indicate the high level air pollution determined using the altimeter 330 .

The extraction unit 210 provided in the analysis unit 200 may extract color features from the moss image.

The planarization unit 230 may planarize the color features extracted by the extraction unit 210 .

The classification unit 250 may classify the data output from the planarization unit 230 according to the contamination level.

The analysis unit 200 may analyze air pollution using a moss classification model using VGG-16 based on a convolutional neural network (CNN).

VGG-16 may have 16 stacks in which convolution and maxpool layers are combined. VGG-16 may have a high density layer.

The moss classification model extracts color features from a moss image, passes the extracted color features through a convolution and maxpool layer stack to output a flattened result, and converts the flattened result to a dense layer. layer) can be passed through.

The analysis unit 200 may classify the moss image using the output data of the high-density layer, and analyze air pollution according to the classification result.

The monitoring method of the present invention may be performed by the monitoring device shown in FIG. 1 . The monitoring method of the present invention applies a moss image in which a moss (lichen) 90 that grows on the outside of the building 10 is photographed to a convolutional neural network (CNN), and uses the color characteristics of the moss image analyzed through the CNN. Air pollution around the building 10 can be monitored.

A moss classification model that analyzes air pollution using the color of moss is required. Other monitoring methods for obtaining a moss classification model may be provided.

For example, according to the monitoring method of the present invention, the monitoring device may learn the color of lichen that changes according to air pollution for each air pollution degree. Based on the learning, when a moss image including moss is input, the monitoring device may generate a moss classification model that analyzes the color of moss included in the moss image.

The moss classification model can classify the color analyzed through the analysis of the moss image according to the pre-learned color for each air pollution level. The moss classification model can be used to classify the color of moss in other monitoring devices that analyze the captured images. When moss is classified as a specific color, it may be estimated that the air pollution state previously matched to the specific color represents pollution around the exterior of the building including the moss.

The relationship between the moss and air pollution described above, and the specific content of the moss classification model will be described in detail below.

Air pollution can have various effects on living organisms. Moss is a biomarker that is very sensitive to air quality and absorbs elements present in the air. Moss contamination levels can reflect the same impact on living organisms. Through the extraction of color features for moss, the impact of pollution rates on cities can be analyzed and how many pollutants are deposited in the human body can be explained. The color of moss can change differently depending on air pollution.

3 is a schematic diagram showing the VGG-16 architecture. 4 is a schematic diagram showing the operation procedure of the VGG-16 architecture.

The VGG-16 model (moss classification model) proposed in the present invention can receive a moss image including moss as an input and extract color features from the moss image. The extracted color features can pass through convolution and max full-layer stack in VGG-16 architecture. Finally, the input image can be flattened (flattened) and classified according to the level of contamination. The accuracy can be compared with existing methods that support vector machines and particle swarm optimization.

Earth's atmosphere protects living organisms from many particles, such as X-rays, UV rays, and cosmic rays colliding around planets. It also helps to maintain a normal temperature by maintaining a favorable balance for the sustainability of life. When toxic substances are mixed in the air, it creates air pollution and affects living things. Air pollution may be due to gases emitted from vehicles or factories, and the like.

From smog across cities to smoke in homes, air pollution can be a huge threat to health and climate.

Lichens may contain complex organisms formed as a result of the association of one or two species of fungi and algae. Moss can grow in a variety of places, such as bark, walls, rocks, tombstones, roofs, and soil.

According to the present invention, the level of contamination around a building can be monitored or analyzed based on image analysis of moss. A collection of 9718 images targeting moss with different structures and compositions can be used as a data set as shown in FIG. 5 in the present invention, which is proposed to analyze air pollution around buildings.

5 is a schematic diagram showing a data set of moss. 6 is a visual representation of the color of moss in contact with different levels of contamination. 7 is a schematic diagram showing the operational steps of the VGG-16 architecture for image classification.

Moss analysis for contamination identification may correspond to a multi-class image classification problem.

As an example, the monitoring device of the present invention uses the color of moss that varies according to air pollution, in the order of air pollution from a clean state to a serious pollution state, 'Not Polluted', 'Moderately Polluted', 'Heavily Polluted', ' It can be classified as 'Extremely Polluted'. 'Not Polluted' may indicate a state in which there is no air pollution. 'Extremely Polluted' may indicate a condition with severe air pollution.

VGG-16 may be a proposed CNN-based approach for image classification. Moss classification may include a two-step process. Step 1 ('Step 1' in FIG. 7 ) may focus on learning a feature set from an input image. VGG-16 can be stacked as a combination of convolution and max full layers that can learn feature sets from input images. Step 2 ('Step 2' in FIG. 7 ) may receive and flatten the output of step 1.

The flattened output may be provided as an input to the high-density layer as shown in FIG. 4 . High-density layer 2 can be activated with the drop-out function. Softmax can be used as an activation function for multi-class classification. The input image is classified according to the output of the high-density layer, and the classification may indicate the level of contamination. The VGG-16 model is constructed using training data from all available images in the data set. The model is limited to an epoch value of 35, which can reduce the number of forward and backward passes.

The overall contamination detection around the building 10 , such as a building, may be performed by the VGG-16 architecture. Moss can grow on rocks, walls and roofs, and the present invention can use moss as an indicator of contamination around buildings. There are about 2300 species of lichens with varying physical and chemical compositions used to indicate the level of pollution in their surroundings. Moss can grow wild around buildings. The moss may be monitored through a photographing unit 300 such as a drone. The image captured through the photographing unit 300 may be used as an input to the VGG-16 moss classification model.

The proposed VGG-16 model can receive input and extract color features from images. Extracted features can pass through convolution and max full-layer stack in VGG-16 architecture. Finally, the input image can be flattened and classified according to the level of contamination.

8 is a schematic diagram illustrating a pollution detection model using VGG-16.

The performance of the VGG-16 model can be verified through test data and other existing machine learning models shown in FIG. 8 . In this work, we implemented the most familiar machine learning models, SVM and PSO. Compared to other machine learning models, VGG-16 can provide better performance in all performance measures.

9 is a graph showing classification accuracy. 10 is a graph showing classification accuracy. 11 is a graph illustrating recall. 12 is a graph showing F-Measure.

Compared with the machine learning algorithms SVM and PSO, the VGG-16 based deep learning algorithm correctly classified the maximum number of images. The highest accuracy of the VGG-16 is 93.6%, and the accuracies of PSO and SVM are 84.8% and 81.2%. SVM is one of the best multi-class classifiers, but its performance degrades when the number of features in each data point exceeds the number of training data samples. The VGG-16 performs better in terms of precision, recall, and other parameter measurements such as F-Measure measurements and can produce maxima of 94.4%, 92.9% and 93.6%, respectively. VGG-16 can perform improved multi-class classification over other machine learning models.

The VGG-16 is a deep architecture with a 16-layer stack of convolutions and up to full layers. Moss images can be grouped separately for training and model validation. The overall performance of VGG-16 can be measured as shown in Table 1 in terms of accuracy and loss function. In the proposed task, we use the optimized number of epochs (EPOCH) to control the total number of forward and backward passes. A visual representation of the VGG-16 performance is shown in FIGS. 9 to 12 .

EPOCH Training Accuracy Validation Accuracy Training Loss Validation Loss 3 0.58 0.69 0.81 0.73 7 0.826 0.841 0.56 0.45 10 0.845 0.86 0.26 0.19 13 0.886 0.89 0.13 0.09 17 0.926 0.932 0.09 0.08 20 0.928 0.94 0.08 0.03 23 0.931 0.935 0.04 0.02 27 0.935 0.939 0.02 0.01 30 0.936 0.941 0.02 0.01 35 0.936 0.941 0.02 0.01

The VGG-16 algorithm can produce a maximum training accuracy of 93.6 and a validation accuracy of 94.1. This model can produce stable accuracy at epoch value 17 and loss value stable at epoch 20. This clearly shows the stability of the model. In addition, the VGG-16 model can produce maximum accuracy at the 30th epoch and minimum loss values at the 27th epoch.

13 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 13 may be a device (eg, a monitoring device, etc.) described herein.

In the embodiment of FIG. 13 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . In addition, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, methods, and the like described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100 .

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110 . Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the embodiment of the present invention is not implemented only through the apparatus and/or method described so far, and a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded may be implemented. And, such an implementation can be easily implemented by those skilled in the art from the description of the above-described embodiments.

Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also presented. It belongs to the scope of the invention.

10...Building 90...Moss
100...Acquisition unit 110...Acquisition Department
130...Search Department 150...Edition Department
200...analysis unit 210...extraction unit
230...flattening unit 250...classifying unit
300...Record 310...Flight Module
330...altimeter 390...camera

Claims (8)

an acquisition unit that acquires a moss image in which lichens growing on the exterior of a building are photographed;
an analysis unit for analyzing air pollution around the building through the analysis of the moss image;
A monitoring device comprising a.
According to claim 1,
A monitoring device provided with an extraction unit for extracting color features from the moss image, a flattening unit for flattening the extracted color features, and a classification unit for classifying data output from the flattening unit according to contamination levels.
According to claim 1,
The analysis unit analyzes the air pollution using a moss classification model using VGG-16 based on CNN (Convolutional Neural Network),
The VGG-16 has 16 stacks in which convolution and maxpool layers are combined, and the VGG-16 has a high-density layer,
The moss classification model extracts color features from the moss image, passes the extracted color features through convolution and a maxpool layer stack, and outputs a flattened result, and the flattened result is pass through the dense layer,
The analysis unit classifies the moss image by using the output data of the high-density layer, and analyzes the air pollution according to the classification result.
According to claim 1,
An acquisition unit for acquiring a plurality of photographed images from a photographing unit for photographing the exterior of the building, a search unit for searching for a specific photographed image including the moss among a plurality of the photographed images, and editing the specific photographed image to obtain the moss image An editorial unit to create is provided;
The editing unit generates an enlarged image including only the moss by magnifying the moss included in the specific shot image,
The moss image includes the enlarged image,
The analysis unit extracts color information from the enlarged image, and analyzes the air pollution by using the extracted color information.
According to claim 1,
An acquisition unit for acquiring a plurality of photographed images from a photographing unit for photographing the exterior of the building, a search unit for searching for the moss image including the moss from among a plurality of the photographed images, a moss area in which the moss exists in the moss image An extraction unit for extracting and extracting color features of the moss area is provided,
The analysis unit is a monitoring device for analyzing the air pollution by using the color feature extracted by the extraction unit.
According to claim 1,
A photographing unit for photographing the exterior of the building while flying in the air and sensing the flight altitude is provided,
The photographing unit detects the flight altitude at a specific point in time when the exterior of the building is photographed,
The photographing unit provides the photographed image photographed at the specific time point and the flight altitude sensed at the specific time point to the acquisition unit,
the acquisition unit acquires the moss image by using the photographed image, and provides the moss image along with the flight altitude to the analysis unit;
The analysis unit matches the flight altitude to the air pollution analyzed for the moss image, and generates air pollution information for each altitude,
The photographing unit is a monitoring device for photographing the exterior of the building using a camera having an optical axis extending in a direction perpendicular to the direction of gravity.
A monitoring method performed by a monitoring device, comprising:
Monitoring to monitor air pollution around the building by applying a moss image from which lichens growing on the exterior of a building are photographed to a convolutional neural network (CNN), and using the color characteristics of the moss image analyzed through the CNN Way.
A monitoring method performed by a monitoring device, comprising:
Learning the color of lichen that changes according to air pollution by air pollution level,
Based on the learning, when a moss image containing moss is input, a moss classification model is generated that analyzes the color of the moss contained in the moss image,
The moss classification model is a monitoring method for classifying the color analyzed through the analysis of the moss image according to the pre-learned color for each air pollution degree.

Images