KR20230066891A - Smart video management device and method with situation awareness using correlation analysis between objects - Google Patents

Abstract

The present invention discloses a situational awareness smart video control device using correlation analysis between objects and a method thereof. According to the present invention, a smart video control device includes: a communication unit which receives a plurality of image information; and a control unit which collects received image information, recognizes a plurality of objects included in each collected image information, sets a bounding box for each recognized object to extract coordinates, uses the extracted coordinates to determine the size and location of each object, performs situation awareness through correlation analysis between each object based on the calculated information, and generates a warning about situation awareness.

Description

Smart video management device and method with situation awareness using correlation analysis between objects}

The present invention relates to video control technology, and more particularly, to a context-aware smart video control device using correlation analysis between objects that performs context-awareness by analyzing correlations between objects included in image information without a separate action recognition sensor. and methods.

Recent physical security systems use digital video recorders (DVRs) and network video recorders (NVRs) like a black box to post-analyze physical threats or breaches through CCTVs (closed circuit televisions) or IP cameras. Through , the era of storing only 24-hour video is rapidly changing to the era of real-time video threat control using a video management system (VMS) for more active response.

However, VMS simply monitors CCTV and IP cameras in real time or detects only simple video detection, resulting in many unrefined false positives. For example, even when only human behavior is to be detected, the fatigue of control personnel is increased by detecting the movement of animals or objects.

In addition, there is a problem in that prevention cannot be performed in a timely manner in the case of a violent crime such as violence or theft that actually requires the help of a control agent during a false positive occurrence time zone when situational awareness is lacking.

Korea Patent Registration No. 10-2244878 (2021.04.21.)

A technical problem to be achieved by the present invention is a context-aware smart video control apparatus and method using correlation analysis between objects that recognizes multiple objects using deep learning technology and analyzes the correlation between recognized objects to obtain context awareness. It aims to provide

Another technical problem to be achieved by the present invention is to provide a context-aware smart video control device and method using correlation analysis between objects that output context-aware analysis results in text form rather than video form.

In order to achieve the above object, a context-aware smart video control device using the correlation analysis between objects according to the present invention collects a communication unit that receives a plurality of image information and the received image information, and includes them in each of the collected image information. Recognize multiple objects, set a bounding box for each recognized object to extract coordinates, calculate information on the size and position of each object using the extracted coordinates, and calculate each object based on the calculated information. and a control unit that performs situational awareness through correlation analysis between objects and generates an alarm for the situational awareness.

In addition, the control unit is characterized in that it recognizes the multiple objects by using a deep learning technology for recognizing multiple objects at the same time.

In addition, the control unit may calculate information on the size and position of each object using vertical and horizontal values of each object based on two vertices located diagonally from each other among the bounding boxes.

In addition, the control unit is characterized in that each of the mutual correlation between the objects that are adjacent to each other among the multiple objects are analyzed, and the situation recognition is derived by combining the analyzed results.

In addition, the control unit may generate the forecast/alarm by converting an analysis result of context awareness in the form of an image into a text form.

A context-aware smart video control method using correlation analysis between objects according to the present invention comprises the steps of receiving a plurality of image information, collecting the received image information, and recognizing multiple objects included in each of the collected image information. and extracting coordinates by setting a bounding box for each recognized object, calculating information on the size and position of each object using the extracted coordinates, and mutual interaction between each object based on the calculated information. It includes performing situational awareness through association analysis and generating a forecast/alarm for the situational awareness.

The context-aware smart video control apparatus and method using the correlation analysis between objects of the present invention recognizes multiple objects using deep learning technology and analyzes the correlation between the recognized objects to accurately and quickly recognize the context. there is.

In addition, by summarizing the context-aware analysis results and outputting them in the form of text, the user can intuitively grasp the current situation.

1 is a configuration diagram for explaining a video control system according to an embodiment of the present invention.
2 is a block diagram for explaining a smart video control device according to an embodiment of the present invention.
3 is a block diagram for explaining a control unit according to an embodiment of the present invention.
FIG. 4 is a diagram for explaining the object recognizing unit of FIG. 3 .
FIG. 5 is a diagram for explaining the object analysis unit of FIG. 3 .
6 and 7 are diagrams for explaining the correlation analysis unit of FIG. 3 .
8 to 12 are diagrams for explaining the warning generator of FIG. 3 .
13 is a flowchart for explaining a smart video control method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function is obvious to those skilled in the art or may obscure the gist of the present invention, the detailed description will be omitted.

1 is a configuration diagram for explaining a video control system according to an embodiment of the present invention.

Referring to FIG. 1 , the video control system 400 recognizes multiple objects using deep learning technology and analyzes correlations between the recognized objects to determine context. The video control system 400 includes the smart video control device 100 and the image capture device 200, and may further include a user terminal 300.

The smart video control device 100 receives a plurality of pieces of image information from the image capture device 200 and collects the received image information. The smart video control device 100 recognizes multiple objects included in each of the collected image information and performs a correlation analysis between the recognized objects to determine context. The smart video control device 100 generates an alert for situation awareness in the form of text and then outputs it. The smart video control device 100 may be a server computing system including a desktop computer, a laptop computer, a server computer, and a cluster computer.

The image capture device 200 captures images of a plurality of preset areas, respectively, and generates a plurality of image information in real time. To this end, the imaging device 200 includes a plurality of cameras. For example, the imaging device 200 may include a first camera 210, a second camera 220 to an N-th camera 230, and the corresponding cameras may be CCTVs, but are not limited thereto. The image capture device 200 transmits the generated image information to the smart image control device 100 .

The user terminal 300 is a terminal device used by a user or a manager, and receives an alarm from the smart video control device 100 and outputs the received alarm. Through this, the user terminal 300 helps to intuitively monitor when a specific event occurs among a plurality of areas captured by the image capture device 200 . In addition, the user terminal 300 may receive a user input, request video information for a region/date/time selected from the user input to the smart video control device 100, receive the video information, and output the received image information. The user terminal 300 may be a personal computing system including a smart phone, desktop, laptop, tablet PC, handheld PC, and the like.

Meanwhile, the video control system 400 establishes a communication network 450 between the smart video control device 100, the image capture device 200, and the user terminal 300 to allow communication between them. The communication network 450 may be composed of a backbone network and a subscriber network. The backbone network may be composed of one or a plurality of integrated networks among an X.25 network, a Frame Relay network, an ATM network, a Multi Protocol Label Switching (MPLS) network, and a Generalized Multi Protocol Label Switching (GMPLS) network. Subscriber networks include FTTH (Fiber To The Home), ADSL (Asymmetric Digital Subscriber Line), cable network, zigbee, Bluetooth, and wireless LAN (IEEE 802.11b, IEEE 802.11a, IEEE 802.11g, IEEE 802.11n ), Wireless Hart (ISO/IEC62591-1), ISA100.11a (ISO/IEC 62734), COAP (Constrained Application Protocol), MQTT (Multi-Client Publish/Subscribe Messaging), WIBro (Wireless Broadband), Wimax, 3G, It may be High Speed Downlink Packet Access (HSDPA), 4G and 5G. In some embodiments, the communication network 450 may be an internet network or a mobile communication network. In addition, the communication network 450 may include all other well-known wireless communication or wired communication methods to be developed in the future.

2 is a block diagram for explaining a smart video control device according to an embodiment of the present invention.

Referring to FIGS. 1 and 2 , the smart video control device 100 includes a communication unit 10 and a control unit 30, and may further include an output unit 50 and a storage unit 70.

The communication unit 10 communicates with the image capture device 200 and the user terminal 300 . The communication unit 10 receives a plurality of pieces of image information from the image capture device 200 . The communication unit 10 transmits at least one of the warning and image information to the user terminal 300 .

The controller 30 performs overall control of the smart video control device 100 . The control unit 30 collects a plurality of pieces of image information received from the communication unit 10 . At this time, the controller 30 performs a pre-processing process on the collected image information. The control unit 30 multi-recognizes objects included in each of the plurality of pre-processed image information. To this end, the control unit 30 may use a deep learning technique for simultaneously recognizing multiple objects, and may preferably use a Yolo algorithm, but is not limited thereto. Here, since the Yolo algorithm is a well-known technology, additional description thereof will be omitted. The controller 30 sets an axis-aligned bounding box for each recognized object, and extracts coordinates for the set bounding box. Here, the bounding box is a box formed in a rectangular shape by aligning objects along an axis and forming a boundary. The boundary control unit 30 calculates information on the size and position of each object using the extracted coordinates. The control unit 30 performs context awareness through correlation analysis between each object based on the calculated information. The control unit 30 may have a structure in which correlations between neighboring objects are first analyzed, and then the analyzed results are combined to determine overall situation. Here, the control unit 30 performs correlation analysis by comparing whether or not it matches a predefined detection pattern (policy) stored in the storage unit 70 . The control unit 30 converts and generates an example/alarm for situation awareness from a video form to a text form, and outputs the generated example/alarm. At this time, the control unit 30 may transmit the forecast/alarm message to the user terminal 300 through the communication unit 10 .

The output unit 50 individually outputs a plurality of pieces of image information received from the communication unit 10 . For example, the output unit 50 divides the image information taken from the first camera 210, the second camera 220 to the Nth camera 230 through one display and outputs each screen, or a plurality of displays. Image information photographed by the first camera 210, the second camera 220 to the Nth camera 230 may be individually output through. In addition, the output unit 50 outputs the forewarning generated by the control unit 30 . The output unit 50 includes at least one display, and the display includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (LCD), and an organic light-emitting diode (LCD). It may include at least one of an emitting diode (OLED), a flexible display, and a 3D display.

The storage unit 70 stores a program or algorithm for driving the smart video control device 100 . The storage unit 70 stores a plurality of pieces of image information received from the communication unit 10 . The storage unit 70 stores warnings generated by the control unit 30 and predefined detection patterns for correlation analysis. Here, the detection pattern is a reference value defined in the form of a pattern in which the situation of the object is. The storage unit 70 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, It may include at least one storage medium of a magnetic disk and an optical disk.

Figure 3 is a block diagram for explaining a control unit according to an embodiment of the present invention, Figure 4 is a view for explaining the object recognition unit of Figure 3, Figure 5 is a diagram for explaining the object analysis unit of Figure 3, 6 and 7 are diagrams for explaining the correlation analysis unit of FIG. 3 , and FIGS. 8 to 12 are diagrams for explaining the forecast/alarm generation unit of FIG. 3 .

1 to 12, the control unit 30 includes an image information collection unit 31, an object recognition unit 32, an object analysis unit 33, a correlation analysis unit 34, and a forecast/alarm generation unit 35 ).

The image information collecting unit 31 collects a plurality of pieces of image information received from the communication unit 10 . The image information collection unit 31 pre-processes each of the collected image information. Here, the preprocessing process refers to a process of compressing, decompressing, filtering, and the like, image information to be suitable for image processing to be performed later.

The object recognition unit 32 recognizes multiple objects included in each preprocessed image information. The object recognizing unit 32 may recognize multiple objects using deep learning technology that recognizes multiple objects at the same time. Here, the deep learning technology may be Yolo algorithm, but is not limited thereto. The object recognizing unit 32 sets a bounding box for each recognized object and extracts coordinates for the set bounding box. Here, the coordinates are the coordinates of the vertices of the bounding box. The object recognition unit 32 may extract the coordinates of all vertices of the bounding box, but may extract the coordinates of two vertices located diagonally from each other among the bounding boxes as shown in FIG. 4 in consideration of the operation speed and system load. there is. Preferably, the object recognition unit 32 may extract the coordinates of the upper left vertex A and the lower right vertex B of the bounding box.

The object analyzer 33 calculates information about the size and position of each object using the extracted coordinates. The object analyzer 33 calculates the horizontal and vertical positions of the upper left vertex (A) and the lower right vertex (B) of the rectangle constituting each bounding box, and calculates the corresponding bounding box based on the calculated results. Calculate information about the size (area) and position of the box.

That is, the object recognizing unit 32 and the object analyzing unit 33 recognize the person 41, the hand 42, and the knife 43 as objects in the image information, respectively, as shown in FIG. information can be calculated.

The correlation analysis unit 34 performs context awareness through correlation analysis between each object based on the calculated information. The correlation analyzer 34 analyzes correlations between objects that are adjacent to each other among multiple objects, and derives a single context by combining the analyzed results. At this time, the correlation analysis unit 34 may perform correlation analysis by defining the number of detections per unit time by setting thresholds according to normal detection and importance.

For example, the correlation analysis unit 34 may perform correlation analysis based on a predefined detection pattern as shown in Table 1.

detection name man with knife in hand detection object person / hand / knife threshold 15 times / 1 minute object pattern person distance hand distance knife ≤0 ≤0

Here, the detection name is a character type detection event predefined by the user, and is defined as a contextual event name that the user can easily recognize, and after selecting an object (class) to be actually recognized for the corresponding pattern, the conditions between the objects are defined. Conditional clauses that can be used at this time are defined through arithmetic, comparison, assignment, bit, logic, and identification operators such as distance, overlap, height, ratio, and size. In addition, the number of detections is defined per unit time by placing a threshold according to true positives and importance. That is, as shown in FIG. Conduct association analysis to obtain situational awareness. In detail, the correlation analysis unit 34 determines that the distance between the recognized person 41 and the hand 42 is 0 or overlapping, meaning the same object (FIG. 6A), and the hand 42 and the knife ( 43) is judged as a situation where the knife 43 is held in the hand 42 (FIG. 6B) with a distance of 0 or overlapping, and the two situations are correlated to analyze 'a person holding a knife 43 in the hand 42 ( 41)' is derived (Fig. 6c). As in the above-described embodiment, when the context recognition result matches the detection pattern, the text image detection name and the analysis image information are grouped and stored in the same information unit, and the stored image can be output when a detailed search of the text image detection event is performed. control (Fig. 7).

In addition, the correlation analysis unit 34 registers and learns various objects as respective detection patterns, and predefines modeled result values. At this time, when the correlation analysis unit 34 creates a condition for a new detection event including one or more objects, the association between objects is determined by using regular expressions or arithmetic, comparison, assignment, bit, logic, and identification operators. can be defined

The forecast/alarm generation unit 35 generates a forecast/alarm for situation awareness. The forecast/alert generating unit 35 converts the situation-aware analysis result in the form of an image into a text format to generate a forecast/alarm. Here, the forecast/alarm is an alarm message that summarizes context-recognized content so that a user who checks the content can intuitively grasp the situation, and supports selectively outputting corresponding image information. The forecast/alert generator 35 outputs the generated forecast/alarm through the output unit 50 . In addition, as shown in FIG. 8 , the forecast/alarm generation unit 35 transmits the text image detection event to the user terminal 300 in the form of a transmission file, and controls the user terminal 300 to output the forecast/alarm.

Here, the detection screen and real-time video are inquired through a link included in media such as Telegram and SNS (FIG. 9), the detection screen and real-time video are inquired through a link included in SMS (FIG. 10), and a pop-up window The detection screen and real-time video can be inquired through the checkbox included in (FIG. 11), and the detection screen and real-time video can be inquired through the menu included in the table (FIG. 12).

13 is a flowchart for explaining a smart video control method according to an embodiment of the present invention.

Referring to FIGS. 1 and 13 , the smart video control method recognizes multiple objects using deep learning technology and analyzes the correlation between the recognized objects to accurately and quickly recognize the situation. The smart video control method summarizes the context-aware analysis results and outputs them in the form of text, so that the user can intuitively grasp the current situation.

In step S110, the smart video control device 100 collects a plurality of image information. The smart video control device 100 pre-processes each of a plurality of collected video information. Here, the preprocessing process refers to a process of compressing, decompressing, filtering, and the like, image information to be suitable for image processing to be performed later.

In step S120, the smart video control device 100 recognizes multiple objects included in each collected image information, sets a bounding box for each recognized object, and extracts coordinates. The smart video control device 100 recognizes multiple objects using deep learning technology that simultaneously recognizes multiple objects. Here, the deep learning technology may be Yolo algorithm, but is not limited thereto. The smart video control device 100 sets a bounding box for each recognized object and extracts coordinates for the set bounding box. Here, the coordinates are the coordinates of the vertices of the bounding box.

In step S130, the smart video control device 100 calculates information about the size and location of each object using the extracted coordinates. The smart video control device 100 calculates the horizontal and vertical position of the upper left vertex and the horizontal and vertical position of the lower right vertex of the rectangle constituting each bounding box, and calculates the size (area) of the corresponding bounding box based on the calculated results. ) and information about the location.

In step S140, the smart video control device 100 performs context awareness through correlation analysis between each object based on the calculated information. The smart video control device 100 analyzes each of the correlations between neighboring objects among multiple objects, and combines the analyzed results to derive whether it is a single situation.

In step S150, the smart video control device 100 determines whether the situation recognition result matches a predefined detection pattern. The smart video control device 100 terminates the system if the situation recognition results do not match, and performs step S160 if they match. The smart video control device 100 stores the text image detection name and analysis image information in the same information unit before performing step S160, and outputs the stored image when the text image detection event is detailed.

In step S160, the smart video control device 100 generates an alert for situational awareness. The smart video control device 100 converts the context-aware analysis result in the form of a video into a text form to generate a warning. Here, the forecast/alarm is an alarm message that summarizes context-recognized content so that a user who checks the content can intuitively grasp the situation, and supports selectively outputting corresponding image information. The smart video control device 100 may output the generated example/alert or transmit the image detection event to the user terminal 300 in the form of a transmission file so that the user terminal 300 outputs the generated example/alarm.

The method according to an embodiment of the present invention may be provided in the form of a computer readable medium suitable for storing computer program instructions and data. Such a computer-readable recording medium may include program commands, data files, data structures, etc. alone or in combination, and includes all types of recording devices storing data that can be read by a computer system. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks). Optical media), magneto-optical media such as floptical disks, and program instructions such as ROM (Read Only Memory), RAM (RAM, Random Access Memory), flash memory, etc. and a hardware device specially configured to do so. In addition, the computer-readable recording medium is distributed in computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention belongs.

Although the above has been described and illustrated in relation to preferred embodiments for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described in this way, without departing from the scope of the technical idea. It will be readily apparent to those skilled in the art that many changes and modifications can be made to the present invention. Accordingly, all such appropriate changes and modifications and equivalents should be regarded as falling within the scope of the present invention.

10: Ministry of Communications
30: control unit
31: image information collection unit
32: object recognition unit
33: object analysis unit
34: correlation analysis unit
35: warning generating unit
50: output unit
70: storage unit
100: smart video control device
200: imaging device
210: first camera
220: second camera
230: Nth camera
300: user terminal
400: video control system
450: communication network

Claims (6)

a communication unit for receiving a plurality of pieces of image information; and
Collects the received image information, recognizes multiple objects included in each of the collected image information, sets a bounding box for each recognized object to extract coordinates, and uses the extracted coordinates to extract the size of each object. and a control unit that calculates information about a location, recognizes a situation through correlation analysis between each object based on the calculated information, and generates an example/alarm for the awareness of the situation;
Context-aware smart video control device using correlation analysis between objects including. According to claim 1,
The control unit,
Context-aware smart video control device using correlation analysis between objects, characterized in that for recognizing the multiple objects using deep learning technology for recognizing multiple objects at the same time. According to claim 1,
The control unit,
Based on two vertexes located diagonally from each other among the bounding boxes, it is a situation using correlation analysis between objects, characterized in that the information on the size and position of each object is calculated using the vertical and horizontal values of each object. Smart video control device. According to claim 1,
The control unit,
A context-aware smart video control device using correlation analysis between objects, characterized in that each of the multiple objects analyzes the correlation between neighboring objects, and derives the context-awareness by combining the analyzed results. According to claim 1,
The control unit,
A context-aware smart video control device using correlation analysis between objects, characterized in that for generating the forecast and alarm by converting the context-aware analysis result in the form of a video into a text form. Receiving a plurality of pieces of image information;
collecting the received image information;
Recognizing multiple objects included in each of the collected image information and extracting coordinates by setting a bounding box for each of the recognized objects;
Calculating information on the size and position of each object using the extracted coordinates;
performing context awareness through correlation analysis between objects based on the calculated information; and
Generating a warning about the situation awareness;
Context-aware smart video control method using correlation analysis between objects including.

Images