KR102694202B1 - System and method for analyzing slope condition based on internet on things technique and image analysis - Google Patents

Abstract

According to one technical aspect of the present invention, a slope condition analysis method based on IoT and image analysis is performed on an edge server that operates in conjunction with a plurality of sensor devices provided on a slope and providing sensing data and a camera device providing image data of the slope, the method including: a step of determining a first collapse determination factor based on sensing displacement measurement based on displacement data included in the plurality of sensing data; a step of determining a second collapse determination factor based on image analysis based on the image data; a step of determining a collapse pattern occurring on the slope based on the first collapse determination factor and the second collapse determination factor; and a step of identifying at least one object indicated in the image data of the slope using a first deep learning artificial intelligence model, and setting an object risk level based on a positional correlation between the collapse pattern and a damaged object.

Description

Method and system for analyzing slope condition based on internet on things technique and image analysis {System and method for analyzing slope condition based on internet on things technique and image analysis}

The present invention relates to a method and system for analyzing slope conditions based on IoT and image analysis.

As construction technology advances, slopes are increasingly being formed adjacent to roads and residential environments. In particular, the risk of steep slope collapse is increasing due to recent abnormal climate conditions such as global warming.

In particular, in the process of constructing social overhead capital (SOC) such as roads, steep slopes are constructed as part of the facility, but unlike other facilities, they are made of natural materials such as soil, making systematic management difficult.

The maintenance of these slopes in the past has been mostly managed through inspections by personnel based on management guidelines. This management by personnel is not possible in real time, and the reliability of the inspection results is limited by the level of technical knowledge of the inspector.

Accordingly, various technologies for slope management and monitoring are being developed, and the Road Slope Maintenance Management System (CSMS) used by the Ministry of Land, Infrastructure and Transport is an example of such conventional technologies.

However, in the case of these conventional technologies, there are limitations in measurement reliability, and in the case of the road slope maintenance management system (CSMS), there is a limitation in that it is difficult to expand application due to the wire-based system.

Registered Patent No. 10-2241254

One technical aspect of the present invention is to solve the problems of the above-mentioned conventional technology by constructing a slope condition analysis system using IoT sensors and a local edge server, thereby providing a slope condition analysis technology based on IoT and image analysis that can accurately predict the collapse risk of steep slopes in advance and quickly and accurately transmit an alarm accordingly.

In addition, one technical aspect of the present invention is to provide a technology for analyzing the state of a slope based on IoT and image analysis, which can more accurately determine the risk of collapse by combining a first collapse determination element based on displacement measurement of a measurement sensor installed on a slope and a second collapse determination element based on artificial intelligence-based image analysis.

In addition, one technical aspect of the present invention is to provide a slope condition analysis technology based on IoT and image analysis, which can more accurately verify measurement errors occurring in ground displacement sensors by setting a valid verification range that changes according to the size of the displacement value when determining the first collapse judgment element by displacement measurement.

In addition, one technical aspect of the present invention is to provide an IoT and image analysis-based slope condition analysis technology that provides a safer slope monitoring technology by comprehensively utilizing deep learning-based artificial intelligence models to increase the analysis accuracy of image data and implement object-centered risk analysis.

The above objects and various advantages of the present invention will become more apparent to those skilled in the art from the preferred embodiments of the present invention.

One technical aspect of the present invention proposes a slope condition analysis method based on IoT and image analysis. The slope condition analysis method based on IoT and image analysis is a slope condition analysis method performed on an edge server that operates in conjunction with a plurality of sensor devices provided on a slope and providing sensing data and a camera device providing image data of the slope, the method including: a step of determining a first collapse determination factor based on sensing displacement measurement based on displacement data included in the plurality of sensing data; a step of determining a second collapse determination factor based on image analysis based on the image data; a step of determining a collapse pattern occurring on the slope based on the first collapse determination factor and the second collapse determination factor; and a step of identifying at least one object displayed in the image data of the slope using a first deep learning artificial intelligence model, and setting an object risk level based on a positional correlation between the collapse pattern and the damaged object.

Another technical aspect of the present invention proposes a slope condition analysis system based on IoT and image analysis. The IoT and image analysis-based slope condition analysis system may include a plurality of sensor devices disposed on a slope to provide sensing data, a camera device providing image data of the slope, and an edge server operating in conjunction with the plurality of sensor devices and the camera device, wherein the edge server determines a first collapse determination factor based on a change in displacement data included in the sensing data, determines a second collapse determination factor based on image analysis based on the image data, and determines a collapse pattern occurring on the slope based on the first collapse determination factor and the second collapse determination factor, while identifying a damaged object in the image data and setting an object risk level based on a positional correlation between the collapse pattern and the damaged object.

Another technical aspect of the present invention proposes a storage medium. The storage medium is a storage medium storing computer-readable instructions, which, when executed by a computing device, can cause the computing device to perform the following operations: determining a first collapse determination factor based on a sensing displacement measurement based on displacement data included in a plurality of sensing data; determining a second collapse determination factor based on an image analysis based on image data; determining a collapse pattern occurring in the slope based on the first collapse determination factor and the second collapse determination factor; identifying at least one object displayed in the image data for the slope using a first deep learning artificial intelligence model; and setting an object risk level based on a positional correlation between the collapse pattern and the damaged object.

The above-mentioned means for solving the problem do not enumerate all the features of the present invention. Various means for solving the problem of the present invention can be understood in more detail by referring to the specific embodiments of the detailed description below.

According to one embodiment of the present invention, by constructing a slope condition analysis system using IoT sensors and a local edge server, it is possible to accurately predict the risk of collapse of a steep slope in advance and quickly and accurately transmit an alarm accordingly.

In addition, according to one embodiment of the present invention, there is an effect of more accurately determining the risk of collapse by combining the first collapse determination factor by displacement measurement of a measurement sensor installed on a slope and the second collapse determination factor by image analysis based on artificial intelligence.

In addition, according to one embodiment of the present invention, when determining a second collapse determination element by displacement measurement, there is an effect of more accurately verifying a measurement error occurring in a ground displacement sensor by setting a valid verification range that changes according to the size of the displacement value.

In addition, according to one embodiment of the present invention, there is an effect of being able to provide a safer slope monitoring technology by improving the analysis accuracy of image data and implementing object-centered risk analysis by comprehensively utilizing deep learning-based artificial intelligence models.

FIG. 1 is a drawing illustrating an example of an IoT and image analysis-based slope condition analysis system according to one embodiment of the present invention.
FIG. 2 is a drawing illustrating an example of the sensor device illustrated in FIG. 1.
FIG. 3 is a drawing explaining one application example of the sensor device illustrated in FIG. 1.
FIG. 4 is a drawing illustrating an example of the camera device illustrated in FIG. 1.
Figure 5 is a drawing illustrating an example of the edge server illustrated in Figure 1.
FIG. 6 is a diagram illustrating an example of a slope condition analysis method based on IoT and image analysis performed in the slope condition analysis system illustrated in FIG. 1.
FIG. 7 is a block diagram illustrating one embodiment of a control unit of the edge server illustrated in FIG. 5.
FIGS. 8 to 15 are flowcharts illustrating embodiments of control methods performed by the control unit of the edge server illustrated in FIG. 7.

Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

However, the embodiments of the present invention can be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. In addition, the embodiments of the present invention are provided in order to more completely explain the present invention to a person having average knowledge in the relevant technical field.

That is, the above-mentioned purpose, features and advantages are described in detail below with reference to the attached drawings, so that a person having ordinary knowledge in the technical field to which the present invention belongs can easily practice the technical idea of the present invention. In describing the present invention, if it is judged that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, a detailed description will be omitted. Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

In addition, the singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, the terms "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the specification, and should be construed as not including some of the components or some of the steps, or may include additional components or steps.

In addition, various components and sub-components thereof are described below in order to explain the system according to the present invention. These components and sub-components thereof may be implemented in various forms, such as hardware, software, or a combination thereof. For example, each component may be implemented as an electronic configuration for performing a corresponding function, or may be implemented as software itself that can be operated in an electronic system, or may be implemented as a functional element of such software. Or, it may be implemented as an electronic configuration and corresponding operating software.

The various techniques described in this specification may be implemented with hardware or software, or a combination of both, where appropriate. The terms "unit," "server," and "system," as used herein, may likewise be treated as equivalent to a computer-related entity, i.e., hardware, a combination of hardware and software, software, or software at runtime. In addition, each function executed in the system of the present invention may be configured as a module unit, and may be recorded in a single physical memory, or may be recorded while being distributed between two or more memories and recording media.

Various embodiments of the present application may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium that can be read by a machine (e.g., a user terminal (100) or a computing device (300). For example, the processor (301) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is temporarily stored in the storage medium.

Although various flowcharts are disclosed to explain embodiments of the present invention, these are for the convenience of explaining each step, and each step is not necessarily performed in the order of the flowchart. That is, each step in the flowchart may be performed simultaneously with each other, may be performed in the order according to the flowchart, or may be performed in the reverse order of the order in the flowchart.

FIG. 1 is a drawing illustrating an example of an IoT and image analysis-based slope condition analysis system according to one embodiment of the present invention.

The IoT and image analysis-based slope condition analysis system illustrated in Fig. 1 may include a plurality of sensor devices (100), camera devices (200), and edge servers (300).

The sensor device (100) is placed on a slope and generates sensing data to provide it to an edge server (300). The sensor device (100) is equipped with a power generation module such as a solar module that can generate its own power, and generates sensing data including displacement data to provide it to the edge server (300). The sensor device (100) can form a communication network with the edge server (300) using a short-range wireless communication network, and thus, separate installation work such as wired installation is omitted, enabling efficient installation.

The description in this specification assumes that there are multiple sensor devices (100), but it is not necessarily limited thereto and can be implemented with a single sensor device. This is because, as described below, the sensor device (100) has a two-axis detection range, so accurate measurement is possible with a single sensor device.

The camera device (200) captures an inclined surface on which the sensor device (100) is placed to generate image data and provides the data to the edge server (300). Depending on the embodiment, the camera device (200) may be directly connected to the edge server (300) by wire or may be implemented as a component, or may be implemented as an independent device separate from the edge server (300).

In one embodiment, the camera device (200) can perform image analysis functions on its own, and such an embodiment is described below with reference to FIG. 13.

The edge server (300) can detect collapse of a slope and perform status analysis in conjunction with a sensor device (100) and a camera device (200). This edge server (300) can be located near the slope and can provide notification data on the collapse situation to a central server (not shown) or an adjacent edge server.

The edge server (300) determines whether a slope has collapsed by combining analysis of both physical displacement data measured by the sensor device (100) and image data captured by the camera device (200), thereby enabling faster and more precise analysis.

In addition, the edge server (300) is linked with the sensor device (100) and the camera device (200) based on a wireless communication method such as a short-range wireless communication method, WiFi, or a mobile wireless communication method, so that the system is easy to install and a variety of data transmissions are also possible.

In addition, edge servers (300) can be placed on each slope, so that a distributed edge computing environment is built for each slope, thereby enabling distributed processing and rapid peripheral propagation.

Hereinafter, with reference to FIGS. 2 to 13, various embodiments of the IoT and image analysis-based slope condition analysis system and the IoT and image analysis-based slope condition analysis method performed through the system will be described.

FIG. 2 is a drawing illustrating an example of the sensor device illustrated in FIG. 1, and FIG. 3 is a drawing illustrating application examples of the sensor device illustrated in FIG. 1.

Referring to FIGS. 2 and 3, the sensor device (100) may include a storage panel (110) and a battery (120) for storing its own power. For example, the storage panel (110) may be a solar panel or a solar thermal panel, and electric energy stored by the storage panel (110) is stored in the battery (120).

The sensor device (100) may include a display unit (130), and the display unit (130) may include an indicator element (e.g., a specific mark or a display module, etc.) that indicates that the sensor device is installed. The camera device (200) may identify the sensor device (100) based on this indicator element.

The sensor device (100) may include a displacement sensor unit (140) and a temperature and humidity sensor unit (150). The displacement sensor unit (140) may include a ground displacement sensor (141) located underground to detect ground displacement and a ground displacement sensor (142) located above ground to detect ground displacement. In this way, since displacement measurement is possible for each of the ground and the ground, errors can be minimized by comparing and analyzing the measurement data of the ground and the ground.

In one embodiment, a buffer module (143) may be provided between the ground displacement sensor (141) and the ground displacement sensor (142). For example, the sensor device (100) may be installed on the ground surface so that the ground displacement sensor (141) is located in the part that is embedded in the ground in the form of a support, and the ground displacement sensor (142) is located in the part above ground, and this buffer module (143) may be located below the ground surface close to the ground in the ground. The buffer module (143) is intended to limit the influence of ground vibration on the ground, and a structural or material buffer such as a spring may be used.

In one embodiment, the sensor of the displacement sensor unit (140) may be a two-axis displacement sensor. For example, it may be a two-axis displacement sensor that detects vertical displacement and horizontal displacement, respectively. By using such a two-axis displacement sensor, the accuracy is improved because the collapse pattern can be analyzed by distinguishing the direction of each sensor.

The temperature and humidity sensor unit (150) may include a temperature sensor (151) and a humidity sensor (152).

The wireless communication unit (160) can form a wireless communication network with the edge server (300), and various wireless communication methods can be applied, such as short-range wireless communication such as Zigbee, long-range wireless communication such as a mobile communication network, or wireless communication such as WiFi.

The light-emitting display unit (170) includes a display element and can blink to indicate the position of the sensor device (100).

For example, the light-emitting display unit (170) can display the identification number assigned to the sensor device (100) as a visual flashing signal, through which the edge server (300) can easily individually identify each sensor device (100) in the image data.

The control unit (180) can control the components of the sensor device (100) to transmit sensing data to the edge server (300). The sensing data can include ground displacement data, ground displacement data, and temperature and humidity data.

FIG. 4 is a drawing illustrating an example of the camera device illustrated in FIG. 1.

Referring to FIG. 4, the camera device (200) may include a storage panel (210), a battery (220), a camera unit (230), a wireless communication unit (240), and a control unit (250). The storage panel (210), the battery (220), and the wireless communication unit (240) can be easily understood with reference to the description above with reference to FIG. 2, and therefore, a duplicate description is omitted.

The camera unit (230) captures a slope on which the sensor device (100) is installed to generate image data. According to an embodiment, the camera unit (230) includes a driving module to support tilting and panning, and may support a zoom function that allows zooming in and out.

The control unit (250) controls the camera device (200) to provide image data on the slope to the edge server (300).

In one embodiment, the control unit (250) may perform some of the functions performed by the edge server (300) on its own. For example, the control unit (250) may also be implemented in a manner that performs AI-based image analysis on image data and provides the results thereof to the edge server (300). Such an embodiment will be described later with reference to FIG. 13.

FIG. 5 is a drawing illustrating an example of the edge server illustrated in FIG. 1, and is a drawing illustrating an exemplary computing operating environment of the edge server.

FIG. 5 is intended to provide a general and simplified description of a suitable computing environment in which embodiments of edge servers (300) may be implemented, and with reference to FIG. 5, a computing device is illustrated as an example of an edge server (300).

The computing device may include a control unit (303) and a system memory (301).

The control unit (303) may include a plurality of processing units that cooperate in executing the program. Depending on the exact configuration and type of the computing device, the system memory (301) may be volatile (e.g., RAM), nonvolatile (e.g., ROM, flash memory, etc.), or a combination thereof. The system memory (301) includes a suitable operating system (302) for controlling the operation of the platform, such as the WINDOWS operating system from Microsoft Corporation or Linux. The system memory (301) may also include one or more software applications, such as program modules, applications, etc.

The computing device may include additional data storage (304), such as a magnetic disk, an optical disk, or tape. Such additional storage may be removable storage and/or non-removable storage. Computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. System memory (301), storage (304) are both examples of computer-readable storage media. Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory or other memory technology, CD-ROMs, DVDs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which stores desired information and which can be accessed by the computing device (300).

As an input unit (305) of a computing device, for example, a keyboard, a mouse, a pen, a voice input device, a touch input device, and comparable input devices may be included. An output device (306) may include, for example, a display, a speaker, and other types of output devices. Since these devices are widely known in the art, a detailed description thereof will be omitted.

The computing device may also include a communication component (307) that allows the device to communicate with other devices over a network, such as a wired or wireless network, a satellite link, a cellular link, a local area network, and comparable mechanisms, for example in a distributed computing environment. The communication component (307) is one example of a communication medium, which may include computer-readable instructions, data structures, program modules, or other data therein. By way of example and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media.

The edge server (300) can be described as a functional configuration implemented in such a computing environment. This will be described later with reference to FIG. 7.

FIG. 6 is a diagram illustrating an example of a slope condition analysis method based on IoT and image analysis performed in the slope condition analysis system illustrated in FIG. 1.

The sensor device (100) is placed on a slope and provides sensing data (S601). The sensing data may include ground displacement data, ground displacement data, and temperature and humidity data. The ground displacement data and ground displacement data may be two-axis sensing data.

The camera device (200) acquires an image of a slope, i.e., the area around the sensor device, and provides it to the edge server (300) as image data (S602).

The edge server (300) can determine a first collapse judgment factor based on displacement measurement based on displacement data included in a plurality of sensing data (S603).

The edge server (300) can determine a second collapse judgment factor based on image analysis based on image data for the slope (S604).

The edge server (300) can determine a collapse pattern occurring on a slope based on the first collapse determination element and the second collapse determination element (S605).

Here, the edge server (300) can identify key objects such as people in the image and perform more advanced responses.

For example, the edge server (300) can identify key objects such as 'people, cargo' in the image. Key objects that are at risk of collapse damage are referred to as 'damaged objects' hereinafter.

For example, the edge server (300) can identify and determine which collapse risk scenario the damaged object corresponds to based on the damaged object, and perform corresponding responses and alarms. For example, the edge server (300) can identify at least one object displayed in the image data for a slope using a first deep learning artificial intelligence model, and set the object risk level based on the positional relationship between the collapse pattern and the damaged object.

For example, the edge server (300) can dynamically change and set a detection area that becomes a collapse detection target according to the movement of the damaged object.

Additionally, the edge server (300) can check a propagation target suitable for the collapse pattern - for example, an adjacent edge server or control server - and transmit collapse situation data (S606).

Hereinafter, various embodiments of such edge servers will be described in more detail with reference to FIGS. 7 to 12.

FIG. 7 is a block diagram illustrating one embodiment of the control unit illustrated in FIG. 5, and FIG. 8 is a flowchart illustrating a slope status analysis method based on IoT and image analysis performed by the control unit (303).

Referring to FIG. 7, the control unit (303) may include a data collection module (310), a displacement-based analysis module (320), an image-based analysis module (330), a collapse judgment module (340), and a situation propagation module (350).

Referring further to Fig. 8, the data collection module (310) can control the communication unit (307) to collect multiple sensing data from multiple sensor devices (S810). In addition, the data collection module (310) can collect image data on a slope from a camera device.

The data collection module (310) can provide the collected sensing data to a displacement-based analysis module, and can provide image data to an image-based analysis module (330).

The displacement-based analysis module (320) can determine a first collapse judgment factor based on displacement measurement based on displacement data included in a plurality of sensing data (S820). The first collapse judgment factor is information on collapse derived based on displacement measurement.

In one embodiment, the displacement-based analysis module (320) can generate a ground displacement map and an underground displacement map to determine the first collapse judgment factor. FIG. 9 illustrates such an embodiment, and will be further described with reference to FIG. 9. The displacement-based analysis module (320) can generate a ground displacement map and an underground displacement map by reflecting the positions of the sensor devices associated therewith for each of a plurality of displacement data sets received from a plurality of sensor devices (S910). The displacement-based analysis module (320) can correct an error in displacement data between the ground displacement map and the underground displacement map by using a displacement critical range between the ground displacement and the underground displacement (S920).

For example, the displacement-based analysis module (320) may be equipped with ground displacement fluctuation critical range data corresponding to each fluctuation range of ground displacement or ground displacement fluctuation critical range data corresponding to each fluctuation range of ground displacement, which may be data derived from the criticality between ground displacement and ground displacement collected through the system.

The displacement-based analysis module (320) can search for displacements exceeding this critical range and error-correct ground displacement-ground displacement data having a correlation exceeding this critical range.

With this error correction, if this threshold range is exceeded only in one sensor device, it is treated as an error value and is excluded.

Meanwhile, if the data measured from multiple adjacent sensor devices exceeds the critical range, the number of multiple adjacent sensor devices and the area covered thereby are checked, and if the exceedance of the critical range falls within a certain continuous change range with the data within the critical range of other sensor devices, the data exceeding the critical range can be used as is.

On the other hand, if the exceedance of the critical range corresponds to a discontinuous feature because it exceeds the continuous change range and the data within the critical range of other sensor devices, the error can be corrected by normalizing the data exceeding the critical range based on the data within the critical range of other sensor devices in the vicinity.

Various methods can be applied for normalization. The displacement-based analysis module (320) can compare the corrected ground displacement map and the corrected ground displacement map with the collapse pattern data to determine the first collapse judgment factor.

The failure pattern may be indicative of the type of failure of the slope, for example, the type of failure may be determined to be at least one of plane failure, wedge failure, plane wedge composite failure, circular failure, arc failure, tipping, and rockfall.

The displacement-based analysis module (320) has associated data of ground displacement maps and underground displacement maps for each collapse pattern as collapse pattern data, and thus, by comparing the ground displacement maps and underground displacement maps calculated based on displacement data respectively calculated from a plurality of sensor devices with the collapse pattern data, the type of collapse can be distinguished and the first collapse judgment factor can be determined.

In one embodiment, the displacement-based analysis module (320) can correct displacement data by using the relationship between horizontal displacement and vertical displacement for displacement data error correction.

FIG. 10 illustrates such an embodiment, which is further described with reference to FIG. 10.

The displacement-based analysis module (320) compares the ground displacement map and the underground displacement map by relating them to each sensor device, calculates the displacement difference between the ground displacement and the underground displacement, i.e., the vertical displacement, and selects the first displacement data set for the first sensor device having the maximum vertical displacement among them (S921). The displacement-based analysis module (320) can confirm the ground-ground displacement threshold for the corresponding vertical displacement based on the maximum vertical displacement (S922). This ground-ground displacement threshold is a value determined in advance from the established data.

The displacement-based analysis module (320) does not perform error correction if the maximum vertical axis displacement of the first displacement data set is within the ground-ground displacement threshold. On the other hand, the displacement-based analysis module (320) performs horizontal-vertical error correction if the maximum vertical axis displacement exceeds the ground-ground displacement threshold. To this end, the displacement-based analysis module (320) can check the maximum horizontal axis displacement difference for the first displacement data set that exceeds the threshold (S923).

The displacement-based analysis module (320) can calculate a horizontal threshold corresponding to the maximum vertical axis displacement, which can be calculated based on the related data between the vertical axis displacement and the horizontal axis displacement that have been established. If the maximum horizontal axis displacement difference in the first displacement data set exceeds the horizontal threshold, the displacement-based analysis module (320) can correct the horizontal axis displacement value and the vertical axis displacement value of the first sensor device with the average horizontal axis displacement value and the average vertical axis displacement value of a plurality of sensor devices adjacent to the corresponding sensor device (S924).

Since this horizontal-vertical correction is based on the correlation of horizontal-vertical data constructed using a two-axis displacement sensor, it has the effect of being able to check and correct errors in one axis sensor in advance.

Referring again to FIG. 8, the image-based analysis module (330) can determine a second collapse judgment factor based on image analysis based on image data for the slope (S830). The second collapse judgment factor is information on collapse derived based on image analysis. For example, the second collapse judgment factor may be about a collapse pattern based on image analysis.

For example, the collapse pattern may distinguish the type of failure of the slope, and for example, the type of failure may be determined as at least one of plane failure, wedge failure, plane wedge composite failure, circular failure, arc failure, tipping, and rockfall. For example, the image-based analysis module (330) may determine which type of failure the pattern corresponds to based on the image.

In one embodiment, the image-based analysis module (330) can determine collapse determination factors centered on the area surrounding the sensor device.

Fig. 11 illustrates such an embodiment, and is further described with reference to Fig. 11. The image-based analysis module (330) can identify the location of a sensor device in image data (S1110).

For example, the image-based analysis module (330) can identify each sensor by identifying the light-emitting display of the light-emitting display unit of the sensor device (100) in the image data. For example, by making the light-emitting display different by at least one of the light-emitting color, signal, and frequency, each sensor can display its own identification information as a light-emitting display, and the image-based analysis module (330) can distinguish and identify each sensor device by identifying the light-emitting displays of each sensor device in the image data.

The image-based analysis module (330) can set the surrounding area of the sensor device in the image data as a risk detection area for consecutive frames of the image data (S1120).

Collapse judgment factors can be determined based on image changes in the risk detection area (S1130).

Referring again to FIG. 8, the collapse judgment module (340) can determine a collapse pattern occurring on a slope based on the first collapse judgment element and the second collapse judgment element (S840).

The situation propagation module (350) can identify a propagation target in response to a collapse pattern determined by the collapse judgment module and transmit collapse situation data regarding the collapse situation to the identified propagation target (S850).

In one embodiment, the image-based analysis module (330) can perform object identification as well as the above-described collapse pattern identification to support the setting of the damaged object, and the collapse judgment module (340) can perform active collapse judgment and monitoring control according to the movement of the damaged object. This will be described below with reference to FIGS. 12 to 15.

FIG. 12 illustrates an embodiment of an image-based analysis module (330), and the image-based analysis module (330) may include an object identification module (331), a collapse-inducing factor determination module (332), a collapse damage object determination module (333), and an image collapse pattern determination module (324).

In addition, FIG. 13 illustrates an embodiment of a collapse judgment module (340), and the collapse judgment module (340) may include a collapse pattern judgment module (341), a detection area setting module (342), and a situation response setting module (343).

Referring further to Fig. 14, the operation of the image-based analysis module (330) and the collapse judgment module (340) is described.

The object identification module (331) can identify objects in image data. For example, the object identification module (331) can identify objects in image data using a first deep learning AI model learned for object identification (S1410).

Here, the identified objects include not only damaged objects such as people and cargo, but also all objects related to the collapse such as rocks, trees, accumulated snow, and flowing soil, and for this purpose, the first deep learning AI model is pre-learned about these objects and can identify objects if they exist in the image data. For example, the first deep learning AI model can set the type (class) information for the identified object and the bounding box for the object.

The collapse-inducing factor judgment module (332) and the collapse damage object judgment module (333) receive object identification information from the object identification module (331) and can determine and set the collapse-inducing factor object and the collapse damage object, respectively (S1420).

For example, the collapse damage object judgment module (333) can set objects such as people, cargo, and facilities as damage objects.

For example, if the collapse inducing factor determination module (332) identifies an object type as accumulated snow, flowing soil, rolling stones, etc., it can determine this as a collapse inducing factor. The collapse inducing factor determined by the collapse inducing factor determination module (332) can be provided to the collapse pattern determination module (341) and used as an element for determining a collapse pattern.

The image decay pattern determination module (324) can determine the image decay pattern, i.e., the second decay determination element, based on the temporal change of image data.

In one embodiment, the image collapse pattern judgment module (324) can distinguish patterns using a deep learning-based artificial intelligence model. The image collapse pattern judgment module (324) can learn a collapse pattern deep learning model based on a plurality of slope learning images classified by each destruction type of the collapse pattern - the collapse pattern including at least one of plane destruction, wedge destruction, compound destruction, circular destruction, and arc destruction. The image collapse pattern judgment module (324) can identify collapse patterns for risk detection areas of consecutive frames using the collapse pattern deep learning model.

In another embodiment, the image collapse pattern judgment module (324) can judge the collapse pattern based on the change of the object. That is, the collapse pattern includes at least one of plane collapse, wedge collapse, compound collapse, circular collapse, and arc collapse, and the movement of the fixed object fixed to the corresponding position is set differently according to each collapse pattern. To this end, the image collapse pattern judgment module (324) can distinguish fixed objects, such as stones and trees, which serve as a criterion for identifying the collapse pattern among the identified objects, and distinguish the collapse pattern based on the movement of the fixed object. To this end, the image collapse pattern judgment module (324) includes movement characteristic data for each type of fixed object that is set differently for each collapse pattern, and can determine which collapse pattern it corresponds to based on the type of the fixed object and its movement.

As described above, the collapse pattern judgment module (341) can judge a collapse pattern based on the first collapse judgment element based on sensor data and the second collapse judgment element based on image data (i.e., image collapse pattern). The collapse pattern judgment will be described below with reference to FIG. 15.

The collapse pattern judgment module (341) can set an expected risk area according to a collapse pattern using a pre-learned second deep learning artificial intelligence model (S1450). The second deep learning artificial intelligence model can set an expected risk area for each collapse pattern, and this second deep learning artificial intelligence model is a model learned to visually express the expected risk area in an image. That is, a risk area is set differently for each collapse pattern for a slope, and the second deep learning artificial intelligence model can display this in the image data.

The detection area setting module (342) can set the object risk level based on the positional relationship between the collapse pattern and the damaged object.

The detection area setting module (342) can set the predicted movement path of the damage object based on the previous movement history of each damage object. For example, the detection area setting module (342) can set the predicted movement path for each damage object using the previously learned third deep learning artificial intelligence model (S1460). This third deep learning artificial intelligence model is an artificial intelligence model that performs movement prediction according to the temporal flow of various damage objects, and can set different predicted movement paths for each object type and movement status.

Thereafter, the detection area setting module (342) can set the object risk level for each damaged object based on the relationship between the expected risk area for each collapse pattern set by the collapse pattern judgment module (341) and the predicted movement path of the damaged object (S1470).

Thereafter, the detection area setting module (342) can change and set the risk detection area targeting the damaged object by using the predicted movement path of the damaged object (S1480). That is, if the damaged object is moving within the image data, the risk detection area where the collapse is judged to be dynamically changed and set in response to the movement of the damaged object. This is because, if there is a moving object, collapse prediction and notification must be performed with the highest priority, and therefore, unlike when there is no object, the risk detection area is changed and set in response to the movement of the object.

The situation response setting module (343) can set a situation response in response to the above-described collapse pattern, presence or absence of a damaged object, etc. The situation response set by the situation response setting module (343) can be propagated through the situation propagation module (350), and for example, a danger alert broadcast or notification to another adjacent edge server can be appropriately performed according to the situation.

Figure 15 is a flowchart explaining a collapse pattern determination method by a collapse pattern judgment module (341).

Referring to FIG. 15, the collapse pattern judgment module (341) can compare a first collapse pattern by a first collapse judgment element and a second collapse pattern by a second collapse judgment element (S1510).

If the first collapse pattern and the second collapse pattern are identical, the collapse pattern judgment module (341) determines the collapse pattern as the matching pattern (S1520, No).

If the first collapse pattern and the second collapse pattern are different (S1520, yes), the collapse pattern judgment module (341) can determine the collapse pattern by associating it with the risk of the damaged object (S1530 to S1550).

The collapse pattern judgment module (341) can check the first risk prediction area of the first collapse pattern and the second risk prediction area of the second collapse pattern (S1530). The setting of these risk prediction areas is as described above in Fig. 14.

The collapse pattern judgment module (341) selects one risk prediction area among the first risk prediction area of the first collapse pattern and the second risk prediction area of the second collapse pattern based on the correlation with the damaged object. That is, the collapse pattern judgment module (341) can select one risk prediction area with a higher risk among the two risk prediction areas based on the predicted movement path of the damaged object (S1540).

The collapse pattern judgment module (341) can determine a collapse pattern related to a selected risk prediction area as a final collapse pattern (S1550).

These embodiments are configured to avoid collapse patterns that are more risky and dangerous to the victim object, thereby ensuring greater safety for the victim object. Even if there is a possibility of error in judging the collapse pattern, by making a decision based on such relevance to the victim object, the stability of the victim object can be further secured.

The present invention described above is not limited by the above-described embodiments and the attached drawings, but is limited by the scope of the patent claims described below, and those skilled in the art will readily appreciate that the configuration of the present invention can be variously changed and modified within a scope that does not depart from the technical spirit of the present invention.

100 : Sensor device 200 : Camera device
300 : Edge Server
110 : Battery panel 120 : Battery
130: Display section 140: Sensor section
141: Ground displacement sensor 142: Ground displacement sensor
151: Temperature sensor 152: Humidity sensor
160: Wireless communication unit 170: Light-emitting display unit
180 : Control Unit
190 : Storage
210 : Battery panel 220 : Battery
230: Camera section 240: Wireless communication section
250 : Control Unit
301: System memory 302: Operating system
303: Control unit 304: Storage unit
305: Input section 306: Output section
307 : Communication Department
310: Data collection module 320: Displacement-based analysis module
330: Image-based analysis module 340: Collapse judgment module
350: Situation propagation module

Claims (15)

A method for analyzing the condition of a slope performed on an edge server that operates in conjunction with a plurality of sensor devices that are placed on a slope and provide sensing data and a camera device that provides image data for the slope,
A step of determining a first collapse judgment factor based on a sensing displacement measurement based on displacement data included in the plurality of sensing data;
A step of determining a second collapse judgment factor based on image analysis based on image data;
A step of determining a collapse pattern occurring on the slope based on the first collapse judgment element and the second collapse judgment element; and
A step of identifying at least one object displayed in the image data for the above-mentioned slope using a first deep learning artificial intelligence model, and setting the object risk level based on the positional relationship between the collapse pattern and the damaged object; including;
The step of determining the collapse pattern occurring on the above slope is:
A step of setting an expected risk area according to a collapse pattern using a pre-trained second deep learning artificial intelligence model;
A step of setting a predicted movement path of the damaged object based on the existing movement history of each damaged object, and changing and setting a risk detection area targeting the damaged object by using the predicted movement path of the damaged object;
A step of identifying a collapse pattern for the above-mentioned risk detection area of consecutive frames using a collapse pattern deep learning model; and
A step of setting the object risk level for each damaged object based on the relationship between the expected risk area for each collapse pattern and the predicted movement path of the damaged object; including;
Slope condition analysis method based on IoT and image analysis.
In the first paragraph,
Based on the above image data, the step of determining the second collapse judgment factor based on image analysis is:
A step of identifying the location of a sensor device in the above image data;
A step of setting a risk detection area based on the location of the sensor device in the image data; and
A step of determining a second collapse judgment element based on an image change in the risk detection area for successive frames of the image data; comprising;
Slope condition analysis method based on IoT and image analysis.
In the second paragraph,
The step of determining the second collapse judgment factor based on the image change in the risk detection area for the consecutive frames of the image data is as follows.
A step of learning a collapse pattern deep learning model based on multiple slope learning images classified by each type of collapse pattern - the collapse pattern includes plane collapse, wedge collapse, compound collapse, circular collapse and arc collapse; and
A step of identifying a collapse pattern for the risk detection area of the continuous frames using the collapse pattern deep learning model; comprising;
Slope condition analysis method based on IoT and image analysis.
delete delete In the first paragraph,
The above sensor device,
A ground displacement sensor located underground to detect ground displacement and generate ground displacement data; and
A ground displacement sensor positioned on the ground to detect ground displacement and generate ground displacement data;
The above sensing data is,
Including the above-mentioned ground displacement data and the above-mentioned ground displacement data,
Slope condition analysis method based on IoT and image analysis.
In Article 6,
The step of determining the first collapse judgment factor based on the above displacement measurement is:
A step of generating a ground displacement map and an underground displacement map by reflecting the positions of the sensor devices associated with each of the plurality of displacement data sets received from the plurality of sensor devices;
A step of correcting the error of displacement data between the ground displacement map and the ground displacement map by using the displacement critical range between the ground displacement and the ground displacement; and
A step of comparing the corrected ground displacement map and the corrected ground displacement map with the collapse pattern data to determine a first collapse judgment factor; including;
Slope condition analysis method based on IoT and image analysis.
A plurality of sensor devices arranged on a slope and providing sensing data;
A camera device providing image data for the above slope; and
An edge server that operates in conjunction with the plurality of sensor devices and the camera device, the edge server determining a first collapse determination factor based on a change in displacement data included in the sensing data, determining a second collapse determination factor based on image analysis based on the image data, and determining a collapse pattern occurring in the slope based on the first collapse determination factor and the second collapse determination factor, while identifying a damaged object in the image data and setting an object risk level based on a positional relationship between the collapse pattern and the damaged object; including:
The above edge server,
A communication unit forming a short-range wireless communication channel with the plurality of sensor devices and the camera device; and
A control unit that determines a collapse pattern for the slope by combining changes in displacement data included in the sensing data received through the communication unit and changes in image data;
The above control unit,
A data collection module that collects the plurality of sensing data from the plurality of sensor devices and collects image data for the slope from the camera device;
A displacement-based analysis module that determines a first collapse judgment factor based on displacement measurement based on displacement data included in the plurality of sensing data;
An image-based analysis module that determines a second collapse judgment factor based on image analysis based on image data for the above-mentioned slope and identifies at least one object displayed in the image data; and
A collapse judgment module that determines a collapse pattern occurring on the slope based on the first collapse judgment element and the second collapse judgment element, and sets the object risk level based on the positional relationship between the collapse pattern and the damaged object;
The above collapse judgment module is,
A collapse pattern judgment module that sets an expected risk area according to a collapse pattern using a pre-learned second deep learning artificial intelligence model, and identifies a collapse pattern for a risk detection area of consecutive frames using a collapse pattern deep learning model;
A detection area setting module that sets a predicted movement path of each damaged object based on the existing movement history of each damaged object, changes and sets a risk detection area targeting the damaged object using the predicted movement path of the damaged object, and sets an object risk level for each damaged object based on the correlation between the predicted risk area for each collapse pattern and the predicted movement path of the damaged object; including;
Slope condition analysis system based on IoT and image analysis.
delete In Article 8,
The above control unit,
A situation propagation module that confirms a transmission target in response to a collapse pattern determined by the collapse judgment module and transmits data on a collapse situation to the confirmed transmission target; further comprising;
Slope condition analysis system based on IoT and image analysis.
In Article 8,
The above sensor device,
A ground displacement sensor located underground to detect ground displacement and generate ground displacement data; and
A ground displacement sensor positioned on the ground to detect ground displacement and generate ground displacement data;
The above sensing data is,
Including the above-mentioned ground displacement data and the above-mentioned ground displacement data,
Slope condition analysis system based on IoT and image analysis.
In Article 11,
The above displacement-based analysis module,
For each of the plurality of displacement data sets received from the plurality of sensor devices, a ground displacement map and an underground displacement map are generated by reflecting the positions of the sensor devices associated therewith, and an error in displacement data between the ground displacement map and the underground displacement map is corrected by using a displacement critical range between the ground displacement and the underground displacement, and then the corrected ground displacement map and the corrected underground displacement map are compared with the collapse pattern data to determine a first collapse judgment factor.
Slope condition analysis system based on IoT and image analysis. In Article 8,
The above image-based analysis module,
A collapse pattern deep learning model is trained based on multiple slope learning images classified by each type of collapse pattern - the collapse pattern includes plane collapse, wedge collapse, compound collapse, circular collapse, and arc collapse - and the collapse pattern deep learning model is used to identify the collapse pattern for the risk detection area of consecutive frames.
Slope condition analysis system based on IoT and image analysis.
In Article 8,
The above collapse judgment module is,
Setting the expected risk area according to the above collapse pattern, setting the expected movement path of the damaged object, and then setting the object risk level based on the correlation between the expected risk area and the expected movement path of the damaged object.
Slope condition analysis system based on IoT and image analysis.
In a storage medium storing computer-readable instructions,
The above instructions, when executed by a computing device, cause the computing device to:
An operation of determining a first collapse judgment factor based on a sensing displacement measurement based on displacement data included in a plurality of sensing data;
A step of determining a second collapse judgment factor based on image analysis based on image data;
An operation for determining a collapse pattern occurring on a slope based on the first collapse determination element and the second collapse determination element;
An operation of identifying at least one object displayed in the image data for the above slope using a first deep learning artificial intelligence model; and
An operation to set the object risk based on the positional relationship between the above collapse pattern and the damaged object;
The action that determines the collapse pattern occurring on the above slope is:
Actions to set expected risk areas based on collapse patterns using a pre-trained second deep learning artificial intelligence model;
An action of setting a predicted movement path of a damaged object based on the existing movement history of each damaged object, and changing and setting a risk detection area targeting the damaged object using the predicted movement path of the damaged object;
An operation of identifying a collapse pattern for the above-mentioned risk detection area of consecutive frames using a collapse pattern deep learning model; and
An operation for setting the object risk level for each damaged object based on the relationship between the expected risk area for each collapse pattern and the predicted movement path of the damaged object; including;
Storage medium.