KR102058452B1 - IoT Convergence Intelligent Video Analysis Platform System - Google Patents

Abstract

The present invention relates to an IoT convergence intelligence type image analysis platform system and, more specifically, to an IoT convergence intelligence type image analysis platform system, capable of improving determination reliability on a warning situation and preventing degradation of work efficiency due to a false alarm of a partial intelligence image system by providing finally verified information to a client terminal after checking whether generation of an alarm is true or false in accordance with a comparison condition between CCTV image data and IoT measurement sensor data. To achieve the purpose of the present invention, the IoT convergence intelligence type image analysis platform system includes: multiple IoT measurement sensors collecting sensing data on a subject by sensing an abnormality of the subject; a CCTV camera collecting image data on the subject to sense the abnormality of the subject; a network video recorder (NVR) server receiving and storing the image data by being connected to the CCTV camera to communicate with the CCTV using data; a data integrating and linking unit connected to the IoT measurement sensors and the NVR server to communicate with the IoT measurement sensors and the NVR server using the data and processing the sensing data and the image data as integrated data by individually receiving the sensing data and the image data; a platform server connected to the data integrating and linking unit to communicate with the data integrating and linking unit by using the data and receiving integrated data, wherein the platform server also analyses the sensing data and the image data in the integrated data and generates situation data and warning data about the abnormality of the subject; and the client terminal connected to the platform server to communicate with the platform server using the data and displaying the situation data and the warning data by receiving the situation data and the warning data from the platform server.

Description

IoT Convergence Intelligent Video Analysis Platform System

The present invention relates to an IoT converged intelligent video analysis platform system, and more specifically, after confirming the authenticity of an alarm according to a mutual comparison condition between CCTV image data and IoT measurement sensor data, a final verified alarm is provided to a client terminal. The present invention relates to an IoT converged intelligent video analytics integrated surveillance platform system that can prevent the degradation of work efficiency due to false alarms of an intelligent video system and can improve the reliability of judgment on an alarm situation.

In general, the intelligent video analysis system identifies objects and actions only through the acquired images and images, so that a number of screens have been analyzed in the post-cause analysis black box type rather than the precautionary purpose that was managed only through the experience and continuous operation of a comprehensive situation room controller. Image analysis system.

Since then, as the new technology has become more popular, it has been introduced as an image operation system that automatically detects abnormal behaviors by automating intelligent image analysis and controls a large number of screens with a limited number of people.

However, the system reliability is deteriorated due to the failure to continuously manage and learn the rules set by the user in advance or to detect or error due to external environmental factors such as image quality, illumination, weather, vibration, etc. to be.

1 is a block diagram showing the internal configuration of a conventional facility monitoring system and intelligent image analysis system.

Referring to FIG. 1, the conventional measurement sensor monitoring system includes an IoT measurement sensor 10, a communication connection unit 20, a DB server (Data Base Server) 30, and a client terminal 40. The configuration of the system is composed of a CCTV camera 50, NVR server (Network Video Recorder Server, 60).

The IoT measurement sensor 10 is a device integrated in a distributed control system DCS (Programmable Logic Controller) or a distributed control system (DCS) that collects operation control and state value data of each facility into one place. The apparatus includes a sensing data transmission unit 11 configured to take data from another system using a standard communication protocol for communication linkage. A representative equipment protocol is Object Linking and Embedding for Process Control (OPC).

The communication connection unit 20 sends the acquired data to the server in accordance with the data base structure and the data processing unit 21 for filtering according to the quality and quality of the sensing data and dead band conditions for each device through standard communication protocol communication. The state is composed of a data transmission section 22.

DB server 30 is a data storage unit 31 for storing the collected sensing data in chronological order, a sensing data processing unit 32 for real-time updating the data defined in the DB field name, and is obtained in real time from each sensor The sensing data analyzing unit 33 analyzes the data value and the set alarm value in real time, and the sensing data alarm processing unit 33 stores the data reaching the alarm value and processes the alarm.

In addition, there is a client terminal 40 composed of data and alarm display unit 41 for displaying the end user can determine the operational status of the facility through the inquiry and comparison of the data and alarm data collected from the measurement sensor in real time. This can be composed of various environments such as Dashbord, Web, Mobile, C / S (Client & Server).

The conventional video surveillance system includes a CCTV camera 50 installed in each major facility zone includes a video data transmission unit 51 for converting the photographed analog video data to digital to be transmitted to a real-time video data storage server, NVR server ( 60 is an image data storage unit 61 for compressing and storing the collected image data in a digital format, an image control unit 62 for performing CCTV control and editing of captured image frames, and displaying or recording and reproducing a real-time image screen. It consists of the video screen display part 63 to display.

In the conventional system as described above, since the operation state monitoring system and the image monitoring system monitored through the measurement sensor are designed in separate structures, data interworking between the two systems and the structured DB structure for each facility have not been established. Even if an alarm occurs in the video analysis system, it is difficult to manually check the alarm of the measuring sensor of the facility.

In order to prevent facility abnormalities and safety accidents, it is necessary to check prompt and accurate information and respond quickly, but it is almost impossible to monitor hundreds of thousands of measurement sensor data and hundreds of image data in real time with limited personnel in real time and cope with problems. It's close.

Intelligent video analysis system using algorithm-embedded CCTV applied in some areas was introduced, but reliability cannot be guaranteed due to frequent errors and unnecessary alarms of intelligent video analysis system, and it is essential to replace algorithm built-in CCTV H / W. And constraints on the installation period.

Korean Patent Registration No. 10-1513215

The present invention has been made to solve the above problems, and after confirming the authenticity of the alarm according to the mutual comparison condition between CCTV image data and IoT measurement sensor data, provides a verified alarm to the administrator to generate in the existing video analysis system Inaccurate alarms and unnecessary object recognition can be minimized, and even if abnormal situation of IoT measurement sensor data is provided to the user, the user can check the authenticity of the alarm by providing the video data of the corresponding facility. By processing and storing the data in one server, it provides the user with the past image of the equipment and the corresponding data at the same time when analyzing the cause of the future equipment failure, leading to a drastic reduction in the time required for the analysis and increased analysis reliability. IoT convergence intelligent video analysis platform system The purpose is to provide.

The present invention has the following features to achieve the above object.

The present invention provides a plurality of IoT measurement sensors for collecting the sensing data of the object to detect the presence or absence of a specific object; A CCTV camera for collecting image data of the object to detect an abnormality of the specific object; A network video recorder (NVR) server connected to the CCTV camera to perform data communication and receiving and storing the image data; A data integration connection unit connected to the IoT measurement sensors and the NVR server to perform data communication and receiving the sensing data and the image data and processing the integrated data into integrated data; A platform server connected to the data integration linker so as to communicate data, and receiving integrated data, and analyzing sensing data and image data in the integrated data to generate situation data and alarm data on an abnormality of a corresponding object; And a client terminal connected to the platform server to perform data communication and receiving and displaying the situation data and the alert data from the platform server.

The IoT measurement sensor includes a sensing data transmission unit for transmitting the generated sensing data to the data integration unit, the CCTV camera includes an image data transmission unit for transmitting the generated image data to the NVR server.

The NVR server receives image data from the CCTV camera, compresses and stores the image data into digital data, and performs monitoring, event management and control of the CCTV camera, and stores the image data. Image control unit for receiving video from the video frame editing unit, the image screen display unit for outputting the image data received from the CCTV camera in real time, and the image data received in real time output from the image screen display unit receives it and provides it to the data integration connection unit And an image data providing unit.

The data integrated connection unit may include a sensing data receiving unit receiving the sensing data from sensing data transmitting units of the IoT measurement sensors, an image data receiving unit receiving the image data from the image data providing unit of the NVR server, and the sensing data. The integrated data processor for classifying the sensing data and the image data received from the receiver and the image data receiver by object device, and separating and processing the data into frame types for each channel, and the integrated data from the integrated data processor. Integrated data transmission unit for processing according to the transmission to the platform server.

The platform server may include a data storage unit for receiving the integrated data from the integrated data transmitter and storing the integrated data; a sensing data processor for extracting sensing data by requesting real-time or stored integrated data from the data storage unit; The driving alarm analysis unit which receives the sensing data extracted from the data processing unit and compares and analyzes the state value of the operation information data of the object with the preset alarm data to determine whether it is an alarm situation and generates sensing alarm data in case of an alarm situation. By extracting the image data from the data storage unit in real time or stored integrated data, the scene is separated from the image data, the representative shot, the object, the action verb, the key frame and the visual knowledge are generated, and compared with the preset alarm data. Analyze to determine if an alarm situation If it is yellow, the image analysis unit generates image alarm data, and receives driving information data or sensing alarm data from the driving alarm analysis unit, receives visual knowledge from the image analyzing unit, and integrates the sensing data and image data. It includes an integrated alarm processing unit for classifying the alarm data according to the weight of the alarm situation.

In addition, the image analyzer extracts the image data from the integrated data received from the data storage unit to separate the scene (Scene), extracting a representative shot (shot) in each of the separated scene, and the representative shot Extracting an object from a constituent frame; extracting an action verb based on the mutual relationship between the extracted objects; and a frame that is determined to best represent the relationship with the object on which the action verb is extracted. Determining the keyframe as a keyframe, and generating a visual knowledge based on the selected keyframe is performed.

In addition, the integrated alarm processor receives and stores visual knowledge from the image analyzer, and visualizes and transmits the visual knowledge to the client terminal according to the interface of the client terminal.

According to the present invention, when an abnormality of the measurement sensor data is detected, the image of the corresponding facility is provided to the user in advance to remind the user that it is the central management target, and to prevent the problem, and to recognize the object and behavior generated in the intelligent image analysis system. By comparing the authenticity of the alarm against the sensor data to determine the authenticity and comparing and analyzing the conditions provided to the user has an effect that can provide accurate image analysis results.

1 is a block diagram showing the internal configuration of a conventional facility monitoring system and intelligent image analysis system.
2 is a block diagram showing an internal configuration of an IoT converged intelligent video analysis platform system according to an embodiment of the present invention.
3 is a flowchart illustrating an analysis process of an image analyzer according to an exemplary embodiment of the present invention.
4 is a conceptual diagram illustrating the structure of a scene.
5 is a diagram illustrating an alarm data generation process and a data and alarm display process according to an embodiment of the present invention.
6 is a view showing the recognition time of the client in the process of fire of the facility according to an embodiment of the present invention.

In order to explain the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, the following describes exemplary embodiments of the present invention and looks at it with reference.

First, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention, and singular forms may include plural forms unless the context clearly indicates otherwise. Also in this application, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, one or more other It is to be understood that the present invention does not exclude the possibility of the presence or the addition of features, numbers, steps, operations, components, parts, or a combination thereof.

In describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

2 is a block diagram showing an internal configuration of an IoT converged intelligent video analysis platform system according to an embodiment of the present invention.

Referring to the drawings, the IoT convergence intelligent video analysis platform system 100 according to an embodiment of the present invention includes a plurality of IoT measurement sensors 110 for collecting sensing data of an object to largely detect the presence or absence of a specific object; CCTV camera 120 for collecting the video data of the object to detect the abnormality of a specific object, and NVR (Network Video Recorder) server connected to the CCTV camera 120 so as to communicate data to receive and store the video data And a data integration linker 140 connected to the IoT measurement sensors 110 and the NVR server 130 to receive data and processing the received data and the image data as integrated data. Is connected to the data integration connection unit 140, the platform server 150 and the platform for generating the processing data by analyzing the sensing data and image data connected to the platform Member connected 150 and to enable data communication is configured to include a client device 160 for display by receiving the said processed data analysis from the platform server 150.

Here, the IoT measurement sensor 110 is a device facility integrated in a Distributed Control System (DCS) or a Programmable Logic Controller (PLC). The data will be described later using a standard communication protocol for communication linkage. It includes a sensing data transmission unit 111 to transmit to the data communication connection unit 140.

The IoT measurement sensor 110 is a IoT-based sensor unit is installed in a plurality of facilities or facilities in the industrial plant to be monitored for monitoring and to prevent safety accidents to measure the abnormality of the facility or facilities.

For example, the IoT measurement sensor 110 is configured to include in detail a measurement sensor unit, a power supply and a sensing data transmission unit 111, each IoT measurement sensor 110 is given a unique identification number. When the sensing data is transmitted through the sensing data transmitter 111, the corresponding unique identification number is transmitted together.

In addition, the IoT measurement sensor 110 may be configured in a variety of forms, such as a strain meter, a displacement meter, an accelerometer, a temperature sensor, a humidity sensor, a corrosion measurement system independently according to the measurement purpose, and be provided as a combination of a plurality of measurement sensors It may be.

In addition, the IoT measurement sensor 110 is installed in a form that is directly attached to or coupled to an object, such as a target facility or equipment, or installed in a spaced apart from the object is responsible for the target sensing of the object, the plurality of IoT measurement is provided The sensors 110 minimize the number of measurement sensors continuously measured by being constantly powered so that the sensors 110 can be operated with only minimum power, and the remaining sensors are in the minimum standby power state, or the IoT measurement sensor 110 Each solar panel and a storage battery (not shown) provided in the can be configured to produce and store the required power by itself.

The IoT measurement sensor 110 as described above continuously detects the presence or absence of an object according to a cycle according to a setting in real time, generates sensing data for this, and integrates the data integration unit 140 through the sensing data transmission unit 111. Will be sent to.

On the other hand, the CCTV camera 120 is installed on one side of a specific facility or equipment, similar to the IoT measurement sensor 110 described above, the real-time or a predetermined period or when a setting event occurs to shoot the corresponding facilities or facilities and generated images accordingly The data is converted into a digital form and transmitted to the NVR server 130.

The transmission of the image data is made through the image data transmission unit 121 included in the CCTV camera 120, the NVR server 130 receives the image data transmitted in real time, compresses and stores the image frame of the image data Edit and perform real time video display and recording.

The NVR (Network Video Recorder) server 130 receives the image data from the CCTV camera 120, the image data storage unit 131 for compressing and storing the digital data, and the CCTV camera 120 The image control unit 132 performs the monitoring, event management and control of the image data received from the image data storage unit 131 and performs image frame editing, and the image data received from the CCTV camera 120. The image screen display unit 133 outputs in real time and the image data providing unit 134 which receives the image data output in real time from the image screen display unit 133 and provides it to the data integration linker 140.

Here, the image control unit 132 monitors the presence or absence of driving abnormality of the CCTV camera 120, sets and controls event management according to occurrence in various setting situations, and sets the storage environment of the image data storage unit 131 and the setting period. If there is a request for data deletion or image data stored from the outside, it may be searched to perform extraction and transmission control of the corresponding data.

On the other hand, the image data output in real time through the image data providing unit 134 is transmitted to the data integration unit 140, wherein the data integration unit 140 and the sensing data of the above-described IoT measurement sensor 110 and The video data of the CCTV camera 120 is received together with each time and by object to generate the integrated data.

The data integration linker 140 includes a sensing data receiver 141 which receives the sensing data from the sensing data transmitter 111 of the IoT measurement sensors 110, and the image data of the NVR server 130. The image data receiving unit 142 receiving the image data from the study 134, the sensing data and the image data received from the sensing data receiving unit 141 and the image data receiving unit 142 channel by object device and time Integrated data processing unit 143 for separating and processing in a separate frame form, and receives the integrated data from the integrated data processing unit 143 and processed according to the database structure of the platform server 150 to the platform server 150 An integrated data transmission unit 144 is included.

Here, the sensing data receiver 141 performs filtering according to the quality and dead band condition of the sensing data transmitted through the data integration linker 140, and the image data receiver 142 is a real-time video standard protocol for each requested channel. Filter the streamed data according to the preset filtering conditions.

In addition, the integrated data processing unit 143 classifies the sensing data and the image data received from the sensing data receiving unit 141 and the image data receiving unit 142 by object device and time and divides the integrated data separated into frames for each channel. Create

The integrated data generated according to this is delivered to the platform server 150 by the integrated data transmission unit 144.

The platform server 150 is connected to the data integration linker 140 so as to be capable of data communication, and receives integrated data. The platform server 150 analyzes sensing data and video data to generate situation data and alarm data. The platform server 150 receives the integrated data from the integrated data transmission unit 144 and stores the data in the data storage unit 151 and the integrated data request from the data storage unit 151 for real-time or stored integrated data. The sensing data processor 152 extracts the sensing data and the sensing data extracted from the sensing data processor 152, and compares and analyzes the state value of the operation information data of the object with the preset alarm data to determine whether or not it is an alarm situation. The driving alarm analysis unit 153 and the data storage unit 151 for generating sensing alarm data when it is determined and the alarm situation. It extracts video data by requesting real-time or stored integrated data, separates the scene from the video data, generates representative shots, objects, action verbs, keyframes and visual knowledge, compares and analyzes the preset alarm data to determine whether it is an alarm situation. The image analyzing unit 154 determines whether the alarm is generated and generates image alarm data, and receives driving information data or sensing alarm data from the driving alarm analyzing unit 153 and receives visual knowledge from the image analyzing unit 154. The integrated alarm processing unit 155 is configured to integrate the sensing data and the image data to classify and generate the alarm data according to the weight of whether the current situation is an alarm situation.

The data storage unit 151 stores the sensing data and the image data received through the integrated data transmission unit 144 according to a real time predefined Real Time Data Base (RTDB) structure, and the sensing data processing unit 152 or If there is a request from the image analyzer 154 is configured to deliver the stored data.

In addition, the driving alarm analysis unit 153 compares the real-time or pre-stored sensing data received from the sensing data processing unit 152 with preset alarm data to generate alarm data and situation data when setting thresholds are exceeded and set thresholds. If not exceeded, only the situation data which is sensing data according to the current situation is generated.

In addition, the image analysis unit 154 extracts image data by requesting real-time or stored integrated data from the data storage unit 151, but separates a scene from the image data, representative shots, objects, behavior verbs, key frames and the like. It is provided to generate the visual knowledge, wherein the image analysis unit 154 extracts the image data from the integrated data received from the data storage unit 151 to generate the visual knowledge to separate the scene (Scene) (S100) And extracting a representative shot from each of the separated scenes (S110), extracting an object from a frame constituting the representative shot (S120), and based on a correlation between the extracted objects. Extracting an action verb (S130), selecting a frame that is determined to be best represented by the object on which the action verb is extracted, as a key frame (S140), and the line A visual knowledge is generated based on the determined key frame (S150).

3 is a flowchart illustrating an analysis process of an image analyzer according to an exemplary embodiment of the present invention, and FIG. 4 is a conceptual diagram illustrating a structure of a scene.

First, the terms scene, shot, representative shot, key frame, and primitive visual knowledge will be briefly described with reference to FIG. 4.

The input of the visual knowledge providing apparatus for constructing the visual knowledge targets a digital image sequence (for example, a moving image, a digital motion picture, etc.). Digital image sequences are those in which digital images exist continuously per unit time. Typically, the density of a digital image sequence is expressed in terms of a few frames per second (eg, 10 frames per second, 20 frames per second).

For convenience of understanding and explanation, an example is described as a movie, which is a typical digital image sequence. A scene is the smallest scene unit in a movie scenario, where a scene in a movie refers to a series of events that occur at the same time zone in one place, and if any of the places or times change, it becomes a different scene. A scene is composed of one or more shots, and one shot may be a scene. Shot is the lowest unit when dividing a scene into cuts, and generally, shots are uninterrupted shooting or recording that begins when the director says "action" and ends when he says "cut".

These shots combine to form a scene, and a series of scenes combine to create a sequence. In other words, a typical digital image sequence is a scene-specific event, and a particular event consists of one or more shots. When a shot including an event representing a scene is defined as a representative shot in one or more shots constituting the scene, the representative shot includes a frame including an event representing a scene. More specifically, when one frame including an event representing the scene is selected in this representative shot, the frame may be a key frame.

A special example of the digital image sequence is a CCTV image as in the present invention. CCTV images are continuously stored, so it is difficult to distinguish between scenes and shots. Therefore, in case of digital image sequence recorded continuously such as CCTV image, it is classified into normal situation (the situation where no special event occurs) or abnormal condition (the situation where special event is expected) based on specific threshold information set by the administrator. Each abnormal situation is treated as one scene. In the case of a CCTV image, a separate scene becomes one shot (representative shot), and one frame representing an abnormal event is extracted from the shot and used as a key frame.

Visual knowledge expresses the information inherent in the scene in the form of a standardized metadata. It extracts representative shots from scenes that divide digital image sequences, such as CCTV images, based on events, and extracts objects from representative shots using traditional vision techniques. It extracts (syntactic information of the scene), infers event information (semantic information of the scene) based on the correlation between the extracted objects, and expresses these two information in the form of standardized metadata.

Accordingly, the image analyzer 154 according to the present invention extracts the image data from the integrated data received from the data storage unit 151 and separates the scene (Scene) (S100).

At this time, the image analyzer 154 monitors the amount of change in the image data and tags the point of time when the amount of change exceeds the threshold as the start of the scene, and the point of time when the amount of change falls below the threshold as the end point of the scene. The interval image is separated and the separated image is saved as an analysis scene.

Such a threshold may be set by an administrator, and it is preferable to set the threshold high when the amount of change is large and set the threshold low when the amount of change is small according to the object.

Then, the image analyzer 154 extracts a representative shot from each of the separated scenes (S110), for example, by moving the interval region along the separated scene using a sliding window technique, entropy in the interval region. , And a section having the highest entropy can be extracted as a representative shot.

When the representative shot is extracted, the image analyzer 154 extracts the object from the frame constituting the representative shot (S120), for example, an object using a conventional vision technique for the image frame constituting the extracted representative shot. Extract and recognize and calculate the coordinates on the image of the object.

Accordingly, when object extraction is completed, an action verb is extracted based on the relationship between the extracted objects (S130).

For example, the behavior for the event of interest is defined in a formal format such as ontology, and a composition rule of unit actions constituting the defined behavior may be built in advance. The image analyzer 154 satisfies the pre-composite composition rule based on the temporal and / or spatial change relations of the extracted objects (for example, moving speed, direction of movement, and combinations of objects forming mutual relations). Can infer multiple behavioral verbs Typically, a plurality of behavior verbs may be inferred from various relationship information between objects, and ranking may be assigned to the inferred behavior verbs. Behavior definition and composition rules model behavior and examine the characteristics of the movements of body parts, various actions of one person, behaviors occurring between two objects, and group behaviors occurring in groups of multiple objects. It can be built as a formal ontology based on that.

Accordingly, the image analyzing unit 154 selects a frame that is determined to be the best representation of the object and the correlation on which the behavior verb is extracted as a key frame (S140), for example, a plurality of constituent representative shots. An image frame that has the highest ranking behavior verb in the image frame may be selected as a keyframe.

Through this, visual knowledge is generated based on the selected key frame (S150). For example, an image of a selected keyframe, a visual word for the area of the minimum bounding rectangle (MBR) of the objects contained in the keyframe, an object property and an on-screen position of the objects, an inferred behavior verb, and an action verb. By synthesizing the syntactic rules used in the extraction of the syntactic and semantic elements, visual knowledge can be generated in a formal format such as XML. In addition, the visual knowledge may include basic information of the image source used in the keyframe selection process, for example, the type of camera, the position of the camera, the management ID of the camera, the path where the raw image is stored, the name of the raw image, and the scene. Start and end frame numbers, start and end frame numbers of representative shots, frame numbers of key frames, and the like.

Accordingly, the generated visual knowledge is transmitted to the integrated alarm processor 155, and the integrated alarm processor 155 is weighted to determine whether the current situation is an alarm situation along with driving information data or sensing alarm data received from the driving alarm analyzer 153. Alarm data is classified and generated accordingly. In addition, the visual knowledge is stored in the database of the integrated alarm processor 155, which is provided separately, or stored in the alarm data storage 156 (S160). In this case, for example, for convenience of retrieval of the generated visual knowledge, indexing is performed by time unit, space unit, object unit, event unit, visual word unit, etc. and stored as keyframe information representing the scene. Can be.

In addition, the integrated alarm processing unit 155 or the alarm data storage unit 156 visualizes the stored visual knowledge to provide the client (S170). That is, the visual knowledge stored in the database of the integrated alarm processing unit 155 or the alarm data storage unit 156 is loaded into the visual knowledge visualization interface according to the input constraints of the data and the alarm display unit 161 of the client terminal 160, Loaded visual knowledge can be displayed and navigated through the visual knowledge visualization interface.

5 is a diagram illustrating an alarm data generation process and a data and alarm display process according to an embodiment of the present invention.

Referring to the drawing, the integrated alarm processor 155 receives the driving information data and the sensing alarm data according to the driving data trend and the threshold analysis from the driving alarm analyzer 153 and receives the image from the visual knowledge generated from the image analyzer 154. Receives alarm images and analyzes the preceding relationships among them to recognize the situation, and compares the received alarm data with AND and OR conditions, analyzes, and prioritizes the classification process based on weight. The alert is sent to the client terminal 160.

In addition, the alarm data storage unit 156 receives and stores the image or the sensing data from the integrated alarm processing unit 155, and is configured to search for and transmit the request according to the request condition when the request of the client terminal 160 is required.

In FIG. 5, the alarm data generation condition of the integrated alarm processing unit 155 is generated by additionally checking whether or not the video alarm data is generated after the sensing alarm data is generated. The generation of the sensing alarm data may be a post condition.

In addition, even if the sensing alarm data is generated, but the video alarm data does not occur, the manager to send the current sensing alarm data and the video data of the object from the current to the previous predetermined time to prevent the safety accident to the client terminal 160 Can be configured to determine whether or not the alarm situation or to determine the situation that the risk situation is likely to occur even if not the current alarm situation.

In addition, since the sensing alarm data may not occur but only the image alarm data according to the failure of the IoT measurement sensor 110, the alarm data is classified with weights and priorities according to various situations, and then transmitted to the client terminal 160. It would be desirable to be configured such that a response most appropriate to the situation may be performed.

On the other hand, the client terminal 160 is connected to the platform server 150 to enable data communication is provided to receive and display the status data and alarm data from the platform server 150.

To this end, the client terminal 160 includes a data and alarm display unit 161 to display the integrated alarm received from the integrated alarm processing unit 155 according to weight and priority.

The selected integrated alarm is preferably displayed so that real-time data, alarm data, trends, statistical tables, and historical data can be compared, analyzed, and judged at the same time as the measurement sensor real-time data and abnormally detected extracted video screen of the same facility. .

6 is a view showing the recognition time of the client in the process of fire of the facility according to an embodiment of the present invention.

As illustrated in FIG. 6, the IoT converged intelligent video analysis platform system 100 according to the present invention uses a fire generation process as an example. However, when the existing video analysis system immediately before a fire occurs, an administrator recognizes a fire situation. In this case, by comparing the video analysis of the IoT measurement sensor 110 and CCTV camera 120 at the same time to provide a fire occurrence situation to the administrator, the recognition time can be much faster and accuracy can be significantly increased.

Although the fire does not appear in the image, the fire has occurred, or the current facilities or facilities often overheat the threshold, so in such a case, the effectiveness of precautionary prevention by the existing image analysis system will be extremely insignificant.

As described above, the present invention has been described with reference to one embodiment shown in the drawings, but this is merely an example, and those skilled in the art may realize various modifications and other equivalent embodiments therefrom. Will understand.

Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

110: a plurality of IoT measurement sensors 120: CCTV camera
130: NVR server 140: data integration link
150: platform server 160: client terminal

Claims (7)

A plurality of IoT measurement sensors 110 for collecting sensing data of an object to detect an abnormality of a specific object;
CCTV camera 120 for collecting the image data of the object to detect the presence or absence of a specific object;
A network video recorder (NVR) server 130 connected to the CCTV camera 120 to perform data communication and receiving and storing the image data;
A data integration linker 140 connected to the IoT measurement sensors 110 and the NVR server 130 to receive data and processing the received data and the image data into integrated data;
The platform server 150 is connected to the data integration connection unit 140 so as to communicate data, and receives the integrated data, and analyzes the sensing data and the image data in the integrated data to generate the situation data and the alarm data regarding the abnormality of the object. ); And
And a client terminal 160 connected to the platform server 150 so as to communicate data, and receiving and displaying situation data and alarm data from the platform server 150.
The platform server 150 is
A data storage unit 151 which receives the integrated data from the data integration linker 140 and stores the integrated data;
A sensing data processor 152 for extracting sensing data by requesting real-time or stored integrated data from the data storage unit 151;
Receiving the sensing data extracted from the sensing data processing unit 152 to compare and analyze the state value of the operation information data of the object and the preset alarm data to determine whether or not the alarm situation, and to generate the sensing alarm data if the alarm situation Driving alarm analysis unit 153,
The image analyzer extracts image data by requesting real-time or stored integrated data from the data storage unit 151, separates a scene from the image data, and generates a representative shot, an object, an action verb, a key frame, and a visual knowledge ( 154),
Receives driving information data or sensing alarm data from the driving alarm analyzing unit 153 and receives visual knowledge from the image analyzing unit 154 to integrate sensing data and image data to alert the current situation according to the weight of the alarm situation. IoT convergence intelligent image analysis platform system, characterized in that it comprises an integrated alarm processing unit 155 for classifying and generating data.
The method of claim 1,
The IoT measurement sensor 110 includes a sensing data transmission unit 111 for transmitting the generated sensing data to the data integration connection unit 140, the CCTV camera 120 is the NVR to the generated image data IoT converged intelligent image analysis platform system, characterized in that it comprises a video data transmission unit 121 for transmitting to the server (130).
The method of claim 2,
The NVR server 130 is
An image data storage unit 131 which receives the image data from the CCTV camera 120 and compresses and stores the image data into digital data;
An image control unit 132 for performing monitoring, event management and control of the CCTV camera 120 and receiving image data from the image data storage unit 131 to perform image frame editing;
An image screen display unit 133 for outputting the image data received from the CCTV camera 120 in real time;
IoT image intelligent platform analysis system characterized in that it comprises a video data providing unit 134 for receiving the real-time output image data from the video screen display unit 133 to provide it to the data integration connection unit 140.
The method of claim 3,
The data integration connection unit 140
A sensing data receiver 141 which receives the sensing data from the sensing data transmitter 111 of the IoT measurement sensors 110;
An image data receiving unit 142 receiving the image data from the image data providing unit 134 of the NVR server 130;
An integrated data processor 143 which classifies the sensing data and the image data received from the sensing data receiver 141 and the image data receiver 142 by object device, separates them into frames for each channel, and integrates them;
IoT convergence comprising an integrated data transmission unit 144 for receiving the integrated data from the integrated data processing unit 143 and processing it according to the database structure of the platform server 150 to transmit to the platform server 150 Intelligent Video Analysis Platform System.
delete The method of claim 1,
The image analyzer 154 is
Separating the scene (Scene) by extracting the image data from the integrated data received from the data storage unit 151 (S100),
Extracting a representative shot from each of the separated scenes (S110);
Extracting an object from a frame constituting the representative shot (S120);
Extracting an action verb based on the mutual relationship between the extracted objects (S130);
Selecting as a key frame a frame that is determined to best express the relationship between the object on which the behavior verb is extracted (S140);
IoT convergence intelligent video analysis platform system, characterized in that the analysis is made, including the step of generating visual knowledge based on the selected key frame (S150).
The method of claim 6,
The integrated alarm processor 155 receives and stores visual knowledge from the image analyzer 154 and visualizes and transmits visual knowledge to the client terminal 160 according to the interface of the client terminal 160. IoT convergence intelligent video analysis platform system.

Images