KR102615378B1 - Behavioral recognition-based risk situation detection system and method - Google Patents

Abstract

The present invention relates to a system and method for detecting dangerous situations based on behavior recognition. According to the present invention, image data is acquired from a photographing unit installed in a residential space and performs photography, and images captured through the photographing unit, and the obtained image It includes a risk detection device that performs a judgment on a risk situation by recognizing the behavior of an object corresponding to a person after blurring the data, wherein the risk detection device receives the captured image from the photographing unit and generates a plurality of frames. an image extraction unit that acquires the image data and performs a blur process on the image data; An object detection unit that detects the object from blurred image data, identifies people according to the detected object, and generates object image data through preprocessing, and recognizes behavior from the object image data according to the identified people to determine whether there is a dangerous situation. A behavior recognition-based risk situation detection system including a behavior recognition unit that determines can be provided.
In addition, an image extraction step in which the risk detection device receives the captured image from the photographing unit and acquires image data including a plurality of frames; A blur processing step of blurring the acquired image data; An object detection step of detecting objects corresponding to people in blurred image data to identify people and generating object image data through preprocessing; It is possible to provide a behavior recognition-based risk situation detection method that includes a behavior recognition step of recognizing behavior from the object image data according to the identified number of people and a situation determination step of determining whether or not there is a risk situation according to the recognized behavior.

Description

Behavioral recognition-based risk situation detection system and method}

The present invention relates to a behavior recognition-based risk situation detection system and method, and more specifically, to detect and respond to risk situations of lonely death or home invasion that may occur in a single-person household using video, but through blur processing. This relates to a behavior recognition-based risk situation detection system and method that can guarantee individual privacy.

As the proportion of single-person households increases, many problems are occurring. The rate of lonely deaths is gradually increasing among elderly people over 65 years old who live alone because they have no one to take care of them. As the number of single-person households also increases among young people, the number of people living alone is increasing due to home invasions. Many incidents and accidents are occurring.

Accordingly, in order to prevent dangerous situations such as lonely deaths among elderly people living alone, a monitoring system is being developed that can monitor the condition of the elderly in real time. The elderly living alone in the house are visually monitored and if unusual signs are detected, it is reported to the relevant person. Research is being conducted in a variety of fields, ranging from relatively simple passive monitoring systems that provide notification to systems that analyze the movements of elderly people in detail to detect abnormal signs early and efficiently determine abnormal signs.

Recently, deep learning technology has been used in monitoring systems, and technology is being developed that can detect dangerous situations by automatically determining the movement of the elderly through video information analysis based on video.

However, the above-mentioned monitoring system has the limitation that it can only recognize the actions of one person and cannot recognize the actions of two or more people, making it impossible to judge not only lonely deaths but also dangerous situations due to home invasions. there is a problem.

In addition, the monitoring system has the characteristic of requiring continuous filming of residential spaces, and as the age group of single-person households diversifies, there is a problem that individuals may feel uncomfortable about their privacy being exposed.

Therefore, there is a need to develop a system that can determine risk situations by recognizing the actions of more than one person while ensuring individual privacy.

Korea Patent Publication No. 10-2021-0090771 (2020.01.10), remote monitoring IoT modular house system exclusively for seniors living alone

In order to solve the above problems, the present invention utilizes video to detect and respond to risky situations of lonely death or home invasion that may occur in a single-person household, while ensuring individual privacy through blur processing. The goal is to provide a behavior recognition-based risk situation detection system and method.

In order to solve the above problems, the behavior recognition-based dangerous situation detection system according to an embodiment of the present invention acquires image data from a photographing unit that is installed in a residential space to perform photography and images captured through the photographing unit, It includes a risk detection device that performs a judgment on a risk situation by recognizing the behavior of an object corresponding to a person after blurring the acquired image data, wherein the risk detection device receives the image captured from the photographing unit and detects a plurality of images. an image extraction unit that acquires the image data including the frame and performs blur processing on the image data; An object detection unit that detects the object from blurred image data, identifies people according to the detected object, and generates object image data through preprocessing, and recognizes behavior from the object image data according to the identified people to determine whether there is a dangerous situation. A behavior recognition-based risk situation detection system including a behavior recognition unit that determines can be provided.

In addition, the image extractor is characterized in that the image data is blurred by applying a Gaussian blur filter.

In addition, the object detection unit extracts a bounding box for the object for each frame of blurred image data and detects the number of people according to the person identification unit and the person identification unit. It may include an image pre-processing unit that crops each frame to obtain a plurality of object frames and generates the object image data.

In addition, when a plurality of bounding boxes are extracted and the number of people is determined to be two or more, the person confirmation unit is characterized in that it checks whether there is an intersection area between the bounding boxes.

In addition, when the person is identified as one person, the image pre-processing unit converts the bounding box extracted for each frame into upper left vertex coordinates and lower right vertex coordinates to obtain box vertex coordinate information, and based on a plurality of frames By comparing the box vertex coordinate information, the minimum and maximum coordinates, which are the upper coordinates of the minimum value and the lower coordinates of the maximum value, are extracted based on the x coordinate, and each frame is cropped by setting a crop area based on the minimum and maximum coordinates. Do it as

In addition, if the image pre-processing unit determines that the number of people is two or more people and an intersection area between the bounding boxes exists, the bounding boxes in which the intersection area exists for each frame are converted into coordinates of the upper left vertex and lower right vertex, respectively. Obtain box vertex coordinate information, and extract box area coordinate information, which is the minimum upper coordinate and maximum value lower coordinates based on the x coordinate of the box area, based on the box vertex coordinate information of the bounding box in which the intersection area exists for each frame, Compare the box area coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinates of the minimum value and the lower coordinates of the maximum value based on the x coordinate, and crop each frame by setting the crop area based on the minimum and maximum coordinates. It is characterized by (crop).

In addition, the behavior recognition unit determines whether a dangerous situation exists by recognizing behavior from the object image data according to the number of people identified through the behavior recognition model, and the behavior recognition model selects some of the plurality of object frames of the object image data. Slow model that receives input and extracts spatial and color features; The Fast model receives more object frames than the Slow model and extracts features according to time changes, and recognizes behavior by combining the features extracted from the Slow model and Fast model based on the number of people, and risks depending on the recognized behavior. It may include a prediction unit that determines whether the situation is present or not.

In addition, the behavior recognition-based risk situation detection method according to an embodiment of the present invention includes an image extraction step in which a risk detection device receives a captured image from a photographing unit and acquires image data including a plurality of frames; A blur processing step of blurring the acquired image data; An object detection step of detecting objects corresponding to people in blurred image data to identify people and generating object image data through preprocessing; It is possible to provide a behavior recognition-based risk situation detection method that includes a behavior recognition step of recognizing behavior from the object image data according to the identified number of people and a situation determination step of determining whether or not there is a risk situation according to the recognized behavior.

Here, the object detection step may include an object extraction step of extracting a bounding box for the object for each frame of blurred image data and a person confirmation step of identifying people according to the extracted bounding box. .

In addition, in the object detection step, when the person information identified in the person confirmation step is one person, the bounding box is converted for each frame to obtain box vertex coordinate information, and the box vertex coordinate information is used to determine each frame. It may further include a first pre-processing step of obtaining a plurality of object frames of the object image data by cropping.

In addition, the first preprocessing step includes: a first coordinate conversion step of obtaining box vertex coordinate information by converting the bounding box into upper left vertex coordinates and lower right vertex coordinates for each frame; A first crop area that compares the box vertex coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate, and sets the crop area based on the minimum and maximum coordinates It may include a setting step and a first cropping step of obtaining a plurality of object frames by cropping each frame according to the cropping area.

In addition, the object detection step includes, when the person information identified in the person confirmation step is two or more people, an intersection area determination step of checking whether there is an intersection area between the bounding boxes of the frame, and an intersection area determination step of checking whether an intersection area between the bounding boxes of the frame exists, Box vertex coordinate information is obtained by converting the bounding boxes in which the intersection area exists for each frame, and each frame is cropped using the box vertex coordinate information of the bounding boxes in which the intersection area exists to obtain the object image data. It may further include a second preprocessing step of obtaining the object frame.

In addition, the second preprocessing step includes a second coordinate conversion step of obtaining box vertex coordinate information by converting the bounding boxes with intersection areas for each frame into upper left vertex coordinates and lower right vertex coordinates, respectively; A box area extraction step of extracting box area coordinate information, which is the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate of the box area, based on the box vertex coordinate information of the bounding box where the intersection area exists for each frame; A second crop area that compares the box area coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate, and sets the crop area based on the minimum and maximum coordinates. It may include a setting step and a second cropping step of obtaining a plurality of object frames by cropping each frame according to the cropping area.

In addition, if the situation is determined to be a dangerous situation in the situation determination step, an alarm and emergency contact step may be further included to activate an alarm device or make emergency contact to a registered terminal.

The behavior recognition-based dangerous situation detection system and method according to the embodiment of the present invention as described above uses video to detect a risky situation of a lonely death or home invasion that may occur in a single-person household, preventing or quickly detecting a risky situation. Be able to respond appropriately.

Additionally, by blurring images obtained from video, it is possible to detect dangerous situations while protecting individual privacy.

In addition, by recognizing the actions of multiple people rather than just one person, it is possible to detect dangerous situations caused by others, such as home invasions, as well as lonely deaths, making it possible to respond to a variety of dangerous situations.

1 is a block diagram showing a behavior recognition-based dangerous situation detection system according to an embodiment of the present invention.
Figures 2 (a) and (b) are exemplary diagrams showing frames obtained by the image extraction unit of Figure 1 and frames subjected to blur processing.
Figure 3 is a block diagram showing the configuration of the object detection unit of Figure 1.
Figure 4 is an example diagram showing a bounding box extracted from a frame through the person identification unit of Figure 3.
Figures 5 (a) to (d) are examples showing crop areas set based on a plurality of frames through the image pre-processing unit of Figure 3 when there is only one person.
Figure 6 is an example diagram showing an object frame created by cropping according to the crop area of Figure 5.
Figure 7 is an example diagram showing a crop area set based on a plurality of frames through the image pre-processing unit of Figure 3 when there are two people.
Figure 8 is an example diagram showing an object frame created by cropping according to the crop area of Figure 7.
Figure 9 is a conceptual diagram showing an action recognition model of the action recognition unit of Figure 1.
Figure 10 is a flowchart schematically showing a method for detecting a dangerous situation based on behavior recognition according to an embodiment of the present invention.
Figure 11 is a flow chart sequentially showing step S300 of Figure 10.
Figure 12 is a flow chart sequentially showing step S330 of Figure 11.
Figure 13 is a flow chart sequentially showing step S350 of Figure 11.

Hereinafter, the description of the present invention with reference to the drawings is not limited to specific embodiments, and various changes may be made and various embodiments may be possible. In addition, the content described below should be understood to include all conversions, equivalents, and substitutes included in the spirit and technical scope of the present invention.

In the following description, the terms first, second, etc. are terms used to describe various components, and their meaning is not limited, and is used only for the purpose of distinguishing one component from other components.

Like reference numerals used throughout this specification refer to like elements.

As used herein, singular expressions include plural expressions, unless the context clearly dictates otherwise. In addition, terms such as “comprise,” “provide,” or “have” used below are intended to designate the presence of features, numbers, steps, operations, components, parts, or a combination thereof described in the specification. It should be construed and understood as not excluding in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

In addition, terms such as “…unit”, “…unit”, and “…module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented through hardware or software or a combination of hardware and software. You can.

Hereinafter, a behavior recognition-based dangerous situation detection system and method according to an embodiment of the present invention will be examined in detail with reference to the attached drawings.

Figure 1 is a block diagram showing a behavior recognition-based dangerous situation detection system according to an embodiment of the present invention, and Figures 2 (a) and (b) are blurred frames obtained by the image extraction unit of Figure 1. It is an example showing a frame, Figure 3 is a block diagram showing the configuration of the object detection unit of Figure 1, Figure 4 is an example showing a bounding box extracted from a frame through the person identification unit of Figure 3, and (in Figure 5) a) to (d) are examples showing how the crop area is set based on a plurality of frames through the image preprocessor of Figure 3 when there is one person, and Figure 6 is generated by cropping according to the crop area of Figure 5. This is an example diagram showing an object frame, and Figure 7 is an example diagram showing how a crop area is set based on a plurality of frames through the image pre-processing unit of Figure 3 when there are two or more people, and Figure 8 is a crop area of Figure 7. This is an example diagram showing an object frame created by cropping according to , and FIG. 9 is a conceptual diagram showing the action recognition model of the action recognition unit of FIG. 1.

The behavior recognition-based dangerous situation detection system (hereinafter referred to as the 'dangerous situation detection system') according to an embodiment of the present invention receives images, extracts image data, and blurs it to protect individual privacy, and detects objects in the image data. By recognizing (human) behavior, it is possible to detect dangerous situations.

When a dangerous situation is detected, the dangerous situation detection system activates the alarm device (W) or makes emergency contact to the terminal (T) pre-registered as a contact information, as shown in FIG. 1, and informs the surroundings of the dangerous situation. A response can be made.

Here, the terminal (T) can be a terminal owned by an unspecified number of people, such as guardians and emergency agencies, and can be applied to various devices such as mobile terminals, PCs, and tablets.

Referring to FIG. 1, the dangerous situation detection system may include a photographing unit 1 and a danger detection device 2.

One or more photographing units 1 are installed in a residential space to perform photographing, and can photograph people present in the residential space.

Additionally, the photographing unit 1 can transmit the captured image in conjunction with the hazard detection device 2 to the hazard detection device 2.

Here, the capturing unit 1 may be a video camera, but is not limited to this, and may be a real-time camera, and may also be comprised of a lidar, a thermal imaging camera, etc. to enable privacy protection without filter processing.

The danger detection device (2) acquires image data from the image captured through the photographing unit (1), performs blur processing on the acquired image data, and then recognizes the behavior of objects corresponding to people to determine a dangerous situation. can do.

To this end, the risk detection device 2 may include an image extraction unit 20, an object detection unit 21, a behavior recognition unit 22, a notification unit 23, and a database 24.

The image extraction unit 20 may receive the captured image from the photographing unit 1, obtain image data, and perform blur processing on the acquired image data.

Here, the image data may include a plurality of frames extracted from the image at regular time intervals, but is not limited to this and may be extracted at irregular time intervals rather than regular time intervals. Additionally, the image data is preferably 16 frames, but is not limited thereto.

Specifically, the image extraction unit 20 can extract a plurality of frames from an image through capture using the Python OpenCV library, but is not limited to this.

In addition, the image extraction unit 20 applies a Gaussian blur filter to each frame of the image data obtained as above to perform blur processing, thereby providing a mosaic effect as shown in FIG. 2. This is to ensure the privacy of individual users.

A Gaussian blur filter applies a mathematical function to a frame to create a blur effect. It is a filter that creates a blur effect by cutting out pixel values that fall outside a certain range and adjusting the pixel values of the frame uniformly.

In addition, the image extraction unit 20 can adjust the degree of blur by adjusting the radius value when applying a Gaussian blur filter to a frame. By determining the frame resolution and automatically adjusting the radius value according to the frame, a Gaussian block filter is applied. can do. However, it is not limited to this, and the radius value may be preset.

The object detection unit 21 can apply an artificial intelligence model to detect objects corresponding to people in the blurred image data from the image extraction unit 20 and generate object image data through preprocessing according to the number of people. This is to ensure that the object's behavior is recognized more efficiently and accurately by extracting only the part corresponding to the object area as data and using it in the behavior recognition unit 22.

Referring to FIG. 3, the object detection unit 21 may include a person identification unit 210 and an image pre-processing unit 211.

The person identification unit 210 may identify people by extracting a bounding box (BB) for an object corresponding to a person for each frame of blurred image data.

This person confirmation unit 210 can extract a bounding box (BB) for an object corresponding to a person from a frame by applying the YOLO model among artificial intelligence models.

The YOLO model is a model that detects all objects in an image and displays location information as a bounding box. By applying a single neural network to the entire image through the grid method, it can even process surrounding information, replacing existing classifier-based object detection techniques. It has superior object detection accuracy compared to R-CNN, Fast R-CNN, etc., and can detect objects very efficiently and quickly, enabling real-time object detection.

More specifically, the YOLO model may be composed of a convolutional neural network (CNN), which consists of a convolution layer (Conv) and a fully connected layer (FC). ) can be composed of.

One or more convolutional layers (Conv) are formed to extract the features of the frame of image data. The frame is divided into an S x S grid, weights are applied to the frame, and a feature map is created through convolution association. (feature map) can be created. This single convolutional layer (Conv) can be applied repeatedly several times while changing the position of the pixel or grid cell of the frame to extract features for the frame.

The group of weights used here can be referred to as a weight kernel, and the weight kernel can be composed of a three-dimensional matrix of n x m x d, and a feature map can be generated through a convolution operation by traversing the frame at specified intervals. there is.

At this time, if the frame is an image with multiple channels (e.g., 3 channels of HSV), the weight kernel may traverse each channel of the frame, perform convolution calculations, and then generate a feature map for each channel.

Here, n may represent a row of a specific size in the frame, m may represent a column of a specific size in the frame, and d may represent a channel of the frame.

The pre-combined layer (FC) can predict one or more bounding boxes and class probabilities for an object using the generated feature map.

Here, the bounding box can be composed of (x,y,w,h,c) coordinates, where x,y is the center coordinate value of the bounding box, w,h are the width and height values of the bounding box, and c is the confidence score.

Additionally, the class probability is the conditional probability of what class the object is under the condition that there is an object in the grid cell.

In addition, the fully combined layer (FC) can select the bounding box for the real object using the coordinates and class probability of the bounding box, and can extract only the bounding box (BB) for the object corresponding to a person. At this time, only the bounding box (BB) for the object corresponding to the person can be extracted using the class specific confidence score and IOU (Intersection over Union).

The class specific confidence score can be obtained by multiplying the confidence score of the bounding box and the class probability, and IOU (Intersection over Union) can be obtained by the intersection area area/union area area.

In this way, the person identification unit 210 can extract the bounding box BB for an object corresponding to a person through the YOLO model (see FIG. 4).

Accordingly, the person verification unit 210 can determine the number of people based on the number of extracted bounding boxes (BB) and generate person information.

Meanwhile, when a plurality of bounding boxes (BB) are extracted and it is determined that there are two or more people, the person confirmation unit 210 may check whether there is an intersection area between the bounding boxes (BB).

Accordingly, if the personnel verification unit 210 determines that an intersection area exists, the corresponding image data can be preprocessed by the image preprocessing unit 211, and if it determines that it does not exist, processing of the corresponding image data is terminated. As much as possible, it can be determined as a 'normal situation' rather than a dangerous situation.

The image pre-processing unit 211 may generate object image data by cropping each frame of image data according to the personnel information identified from the personnel verification unit 210 to obtain a plurality of object frames.

The image pre-processing unit 211 can use a bounding box to crop each frame of image data according to person information. For this, refer to FIGS. 5 to 8 for cases where there is one person and when there are two or more people. Let's break it down and explain it in detail. Meanwhile, in the drawings, for convenience of explanation, 4 frames are shown as a standard, but it is not limited thereto.

For example, when the person is identified as one person as shown in FIG. 5, the image preprocessor 211 first combines the coordinates of the bounding box BB extracted for each frame with the upper left vertex coordinates P _l (x _l , y _l ) You can obtain box vertex coordinate information by converting the bottom right vertex coordinates to P _r (x _r , y _r ).

Next, the image preprocessor 211 compares the box vertex coordinate information (P _l , P _r ) based on a plurality of frames to determine the minimum and maximum coordinates, which are the minimum value upper coordinate P _min and the maximum value lower coordinate P _max based on the x coordinate. You can extract and set a box-shaped crop area (CR) based on the minimum and maximum coordinates.

Accordingly, the image pre-processing unit 211 can obtain a plurality of object frames by cropping each frame according to the set crop area (CR), as shown in FIG. 6.

As another example, the image pre-processing unit 211 can acquire object image data through another pre-processing process even when the number of people is determined to be two or more. Using FIGS. 7 and 8 corresponding to the case of two people, Let me explain.

Referring to FIG. 7, the image preprocessor 211 sets the coordinates of the bounding boxes (BB) in which the intersection area exists for each frame into the upper left vertex coordinates P _l (x _l , y _l ) and the lower right vertex coordinates P _r ( By converting to x _r , y _r ), you can obtain box vertex coordinate information of bounding boxes (BB) where the intersection area exists.

Next, the image preprocessor 211 may extract box area coordinate information based on the box vertex coordinate information of the bounding box BB where the intersection area exists for each frame.

Here, the box area coordinate information may be the minimum upper coordinate and the maximum lower coordinate based on the x coordinate of the box area with the minimum size that can include the area of the bounding boxes (BB).

Next, the image preprocessor 211 compares the box area coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the minimum value upper coordinate P _min and the maximum value lower coordinate P _max based on the x coordinate, and calculate the minimum and maximum coordinates. You can set a box-shaped crop area (CR) as a standard.

Accordingly, the image pre-processing unit 211 can obtain a plurality of object frames by cropping each frame according to the set crop area (CR), as shown in FIG. 8.

Cases processed through this process may correspond to cases where there are two or more people and an intersection area between bounding boxes exists.

Meanwhile, when the image pre-processing unit 211 sets the box-shaped crop area (CR) based on the minimum and maximum coordinates, it performs an adjustment process by expanding/contracting the crop area (CR) according to the radius value of the Gaussian blur filter. You may proceed further.

In this way, the degree of privacy and detection accuracy can be adjusted by expanding or reducing the crop area (CR) according to the degree of blur by applying a Gaussian block filter.

For example, the image preprocessor 211 can expand the crop area (CR) to a certain size when the radius value is larger than the preset reference radius value, which means that as the degree of privacy increases, the boundary between the object and the surrounding environment becomes smaller. Since bounding box detection accuracy may decrease due to ambiguity, object detection accuracy can be increased by expanding the crop area (CR) to prevent objects from being cut off as much as possible.

The behavior recognition unit 22 can determine whether a dangerous situation exists by recognizing behavior from object image data according to the number of people identified by applying an artificial intelligence model.

More specifically, the action recognition unit 22 can recognize the behavior of an object from object image data through a behavior recognition model and detect a dangerous situation according to the recognized behavior.

The SlowFast model can be applied as the action recognition model 220. The SlowFast model is a model structure proposed based on research on object information recognition according to cells in the human eye, and can be learned in an end-to-end manner. . The human eye is composed of P-cells and M-cells. M-cells receive information about fast temporal cycles and respond to rapid visual changes, but rarely respond to spatial details, while P-cells Based on the fact that it responds to detailed spatial characteristics but does not respond to temporal information, the SlowFast model is a model that trains two models by setting different sampling rates for images and then combines the features to make predictions. It is proposed.

Referring to FIG. 9, the action recognition model 220 may include a slow model 221, a fast model 222, and a prediction unit 223. In Figure 9, C means the number of channels, T means the number of frames (time axis), and H and W mean the size of the frame.

The Slow model 221 inputs object frames at a low frame rate, and can extract spatial and color features by receiving some of the plurality of object frames of object image data.

The Fast model (222) inputs object frames at a high frame rate, that is, it receives more object frames than the Slow model (221) and can extract features according to time changes.

At this time, the Fast model 222 may have α times more object frames input than the Slow model 221, and considering performance, it may be desirable for α to be set to 8.

For example, if 2 object frames are input to the Slow model (221), 16 object frames can be input to the Fast model (222).

Additionally, the Fast model (222) may have C and T that are β times smaller than the Slow model (221), and β is preferably 1/8, but is not limited to this.

The prediction unit 223 can recognize behavior by combining features extracted from the slow model 221 and the fast model 222 based on personnel information.

In the case of a single person, the prediction unit 223 may recognize the behavior as one of the normal situations such as sitting, standing, walking, lying down, etc., or a dangerous situation such as falling, based on the characteristics.

In addition, in the case of two or more people, the prediction unit 223 recognizes the behavior through the characteristics as one of the general situations such as massage or touching, or the dangerous situation such as kicking, pushing, hitting with an object, stabbing with a knife, or multiple assaults. can do.

Accordingly, the prediction unit 223 can determine whether a dangerous situation exists based on the recognized behavior, and if it is determined to be a 'dangerous situation,' the notification unit 23 can enable response.

If the prediction unit 223 determines that it is a 'dangerous situation', the notification unit 23 can activate the alarm device (W) or make emergency contact to the registered terminal (T).

At this time, the notification unit 23 can also deliver a message when making emergency contact with the terminal T. The message may include recognized behavior information, corresponding object image data, etc.

In addition, a link that can be linked to the photographing unit 1 is attached to the message so that the image can be viewed in real time through the terminal T. At this time, the image may be blurred.

The database 24 includes images received from the photographing unit 1, image data extracted from the image extraction unit 20, object image data generated from the object detection unit 21, and behavior recognized from the behavior recognition unit 22. Information, risk situation judgment information, etc. can be stored.

Meanwhile, the warning device W may be composed of various devices such as speakers, warning lights, and loudspeakers.

The method of detecting a dangerous situation performed by this behavior recognition-based dangerous situation detection system will be described in detail below.

Figure 10 is a flowchart schematically showing a behavior recognition-based risk situation detection method according to an embodiment of the present invention, Figure 11 is a flowchart sequentially showing step S300 of Figure 10, and Figure 12 is a flowchart sequentially showing step S330 of Figure 11. This is a flowchart, and FIG. 13 is a flowchart sequentially showing step S341 of FIG. 11.

Referring to Figure 10, the behavior recognition-based risk situation detection method according to an embodiment of the present invention includes an image extraction step (S100), a blur processing step (S200), an object detection step (S300), a behavior recognition step (S400), and a situation. It may include a determination step (S500) and an alarm and emergency contact step (S600).

In the image extraction step (S100), the risk detection device 2 may receive the captured image from the photographing unit 1 and obtain image data including a plurality of frames.

The blur processing step (S200) is a step in which the risk detection device 2 blurs the image data acquired in step S100. Blur processing can be performed by applying a Gaussian blur filter to each frame of the image data.

In the object detection step (S300), the risk detection device 2 detects an object corresponding to a person in the image data blurred in step S200, identifies the person, and generates object image data through preprocessing.

Referring to FIG. 11, step S300 includes an object extraction step (S310) and a person confirmation step (S320), and a first preprocessing step (S330) performed according to the judgment in the person confirmation step (S320) and intersection area determination. It may further include a step (S340) and a second preprocessing step (S350).

In the object extraction step (S310), a bounding box (BB) for an object corresponding to a person can be extracted for each frame of the image data blurred in step S200. Since the method for extracting bounding boxes for objects corresponding to people has been explained in detail in the above system, it will be omitted.

The person confirmation step (S320) can generate person information by identifying the number of people according to the number of bounding boxes (BB) extracted in step S310. Accordingly, depending on the number of people, it is possible to check whether there are one person or two people.

Step S300 may further include a first preprocessing step (S330) when the personnel information is for one person.

The first preprocessing step (S330) transforms the bounding box (BB) for each frame to obtain box vertex coordinate information (P _l , P _r ), and uses the box vertex coordinate information (P _l , P _r ) to calculate each frame. A plurality of object frames of object image data can be obtained by cropping.

Referring to FIG. 12, step S330 may include a first coordinate conversion step (S331), a first crop area setting step (S332), and a first cropping step (S333).

The first coordinate conversion step (S331) converts the coordinates of the bounding box (BB) into the upper left vertex coordinates P _l (x _l , y _l ) and the lower right vertex coordinates P _r (x _r , y _r ) for each frame to transform the box. Vertex coordinate information (P _l , P _r ) can be obtained.

The first crop area setting step (S332) compares the box vertex coordinate information (P _l , P _r ) based on a plurality of frames, and the minimum and maximum coordinates are the minimum value upper coordinate P _min and the maximum value lower coordinate P _max based on the x coordinate. You can extract and set the crop area (CR) based on the minimum and maximum coordinates.

In the first cropping step (S333), a plurality of object frames may be obtained by cropping each frame according to the cropping area (CR) set in step S332.

In addition, step S300 may further include an intersection area determination step (S340) and a second preprocessing step (S350) when the person information is for two people.

In the intersection area determination step (S340), it is possible to check whether there is an intersection area between the bounding boxes (BB) of the frame. If it is determined in step S340 that an intersection area between the bounding boxes BB exists, the process may proceed to step S350.

On the other hand, if it is determined in step S340 that there is no intersection area between the bounding boxes BB, processing of the corresponding image data may be terminated and proceed again from step S100.

In the second preprocessing step (S350), when it is determined that an intersection area between bounding boxes exists in step S340, the bounding boxes (BB) in which the intersection area exists for each frame are converted to provide box vertex coordinate information (P _l , P _r ). The object frame of the object image data can be obtained by cropping each frame using the box vertex coordinate information (P _l , P _r ) of the bounding boxes (BB) where the intersection area exists.

Referring to FIG. 13, step S350 may include a second coordinate conversion step (S351), a box area extraction step (S352), a second crop area setting step (S353), and a second cropping step (S354).

The second coordinate conversion step (S351) converts the coordinates of the bounding boxes (BB) in which the intersection area exists for each frame into the upper left vertex coordinates P _l (x _l , y _l ) and the lower right vertex coordinates P _r (x _r , y). _r ), you can obtain box vertex coordinate information (P _l , P _r ).

In the box area extraction step (S352), box area coordinate information can be extracted for each frame based on the box vertex coordinate information (P _l , P _r ) converted in step S351.

Here, the box area coordinate information is the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate of the box area.

The second crop area setting step (S353) compares the box area coordinate information of each frame based on a plurality of frames to extract the minimum and maximum coordinates, which are the minimum value upper coordinate P _min and the maximum value lower coordinate P _max based on the x coordinate, You can set a box-shaped crop area (CR) based on the minimum and maximum coordinates.

In the second cropping step (S354), a plurality of object frames may be obtained by cropping each frame according to the cropping area (CR) set in step S353.

In other words, in step S300, preprocessing is performed differently depending on the number of people identified, and object image data can be obtained.

The behavior recognition step (S400) is a step in which the risk detection device (2) recognizes behavior from object image data according to the identified person, and the object's behavior can be recognized through the SlowFast model among artificial intelligence models.

The situation determination step (S500) can determine whether it is a dangerous situation or a normal situation depending on the behavior recognized in step S400. If the S500 stage is judged to be a dangerous situation, it moves to the S600 stage, and if it is judged to be a normal situation, the process can be returned to the beginning and started again from the S100 stage.

In the alarm and emergency contact step (S600), if a dangerous situation is determined in step S500, the alarm device (W) can be activated or emergency contact can be made to the registered terminal (T).

As described above, the behavior recognition-based risk situation detection system and method according to an embodiment of the present invention uses video to detect a risk situation of a lonely death or home invasion that may occur in a single-person household, and detects the risk situation. Prevent or respond quickly.

Additionally, by blurring images obtained from video, it is possible to detect dangerous situations while protecting individual privacy.

In addition, by recognizing the actions of multiple people rather than just one person, it is possible to detect dangerous situations caused by others, such as home invasions, as well as lonely deaths, making it possible to respond to a variety of dangerous situations.

Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing the technical idea or essential features of the present invention. You will be able to understand it. Therefore, the embodiments described above are illustrative in all respects and are not restrictive.

1: Filming Department
2: Hazard detection device
20: Image extraction unit
21: object detection unit
210: Personnel confirmation department
211: Image preprocessing unit
22: Behavior recognition unit
23: Notification unit
24: database
W: alarm device
T: terminal
BB: Bounding box for the object corresponding to the person
CR: Crop area

Claims (13)

A filming unit installed in a residential space and performing filming, and
A risk detection device that acquires image data from an image captured through the photographing unit, performs a blur process on the acquired image data, and then recognizes the behavior of an object corresponding to a person to determine a risk situation,
The danger detection device,
an image extraction unit that receives the captured image from the photographing unit, obtains image data including a plurality of frames, and performs a blur process on the image data;
An object detection unit that detects the object from blurred image data, identifies people according to the detected object, and generates object image data through preprocessing;
It includes a behavior recognition unit that recognizes behavior from the object image data according to the identified number of people and determines whether or not there is a dangerous situation,
The object detection unit,
A person identification unit that extracts the bounding box for the object for each frame of blurred image data and identifies the number of people, and
An image pre-processing unit that generates the object image data by cropping each frame of the image data according to the personnel information identified from the personnel verification unit to obtain a plurality of object frames,
The image preprocessing unit,
If the above person is identified as one person,
Convert the bounding box extracted for each frame into upper left vertex coordinates and lower right vertex coordinates to obtain box vertex coordinate information,
Compare the box vertex coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinates of the minimum value and the lower coordinates of the maximum value based on the x coordinate,
A behavior recognition-based risk situation detection system that crops each frame by setting a crop area based on the minimum and maximum coordinates.
delete According to paragraph 1,
The personnel verification department said,
If multiple bounding boxes are extracted and it is determined that the number of people is more than two,
A behavior recognition-based risk situation detection system characterized by checking whether there is an intersection area between the bounding boxes.
delete According to paragraph 3,
The image preprocessing unit,
If the number of people is determined to be two or more, but there is an intersection area between the bounding boxes,
Obtain box vertex coordinate information by converting the bounding boxes with intersection areas for each frame into upper left vertex coordinates and lower right vertex coordinates, respectively,
Based on the box vertex coordinate information of the bounding box where the intersection area exists for each frame, box area coordinate information, which is the minimum upper coordinate and maximum value lower coordinate based on the x coordinate of the box area, is extracted,
Compare the box area coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate,
A behavior recognition-based risk situation detection system that crops each frame by setting a crop area based on the minimum and maximum coordinates.
According to paragraph 1,
The behavior recognition unit,
Recognize behavior from the object image data according to the number of people identified through the behavior recognition model to determine whether there is a dangerous situation,
The action recognition model is,
A Slow model that receives some of the plurality of object frames of the object image data and extracts spatial and color features;
A fast model that receives more object frames than the slow model and extracts features according to time changes, and
A behavior recognition-based risk situation detection system that recognizes behavior by combining the features extracted from the slow model and fast model based on the number of people, and includes a prediction unit that determines whether or not there is a risk situation according to the recognized behavior.
An image extraction step in which a risk detection device receives an image captured from a photographing unit and obtains image data including a plurality of frames;
A blur processing step of blurring the acquired image data;
An object detection step of detecting objects corresponding to people in blurred image data to identify people and generating object image data through preprocessing;
A behavior recognition step of recognizing behavior from the object image data according to the identified person, and
It includes a situation determination step to determine whether a dangerous situation exists based on the recognized behavior,
The object detection step is,
An object extraction step of extracting a bounding box for the object for each frame of blurred image data, and
It includes a personnel confirmation step of identifying the number of people according to the extracted bounding box,
The object detection step is,
If the person information identified in the person confirmation step above is one person,
A first preprocessing step of converting the bounding box for each frame to obtain box vertex coordinate information, and cropping each frame using the box vertex coordinate information to obtain a plurality of object frames of the object image data. A behavior recognition-based risk situation detection method including more.
delete delete In clause 7,
The first preprocessing step is,
A first coordinate conversion step of obtaining box vertex coordinate information by converting the bounding box into upper left vertex coordinates and lower right vertex coordinates for each frame;
A first crop area that compares the box vertex coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate, and sets the crop area based on the minimum and maximum coordinates Setting steps and
A behavior recognition-based risk situation detection method comprising a first cropping step of obtaining a plurality of object frames by cropping each frame according to the cropping area.
In clause 7,
The object detection step is,
If the number of people identified in the person confirmation step above is more than two people,
An intersection area determination step of checking whether there is an intersection area between the bounding boxes of the frame, and
If an intersection area between the bounding boxes exists, box vertex coordinate information is obtained by transforming the bounding boxes in which the intersection area exists for each frame, and the box vertex coordinate information of the bounding boxes in which the intersection area exists is utilized for each frame. A behavior recognition-based risk situation detection method further comprising a second preprocessing step of cropping to obtain an object frame of the object image data.
According to clause 11,
The second preprocessing step is,
A second coordinate conversion step of obtaining box vertex coordinate information by converting the bounding boxes with intersection areas for each frame into upper left vertex coordinates and lower right vertex coordinates, respectively;
A box area extraction step of extracting box area coordinate information, which is the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate of the box area, based on the box vertex coordinate information of the bounding box where the intersection area exists for each frame;
A second crop area that compares the box area coordinate information based on a plurality of frames to extract the minimum and maximum coordinates, which are the upper coordinate of the minimum value and the lower coordinate of the maximum value based on the x coordinate, and sets the crop area based on the minimum and maximum coordinates. Setting steps and
A behavior recognition-based risk situation detection method comprising a second cropping step of obtaining a plurality of object frames by cropping each frame according to the cropping area.
In clause 7,
If a dangerous situation is determined in the situation determination step, a behavior recognition-based dangerous situation detection method further includes an alarm and emergency contact step of activating an alarm device or making emergency contact to a registered terminal.