KR20230042926A - Apparatus and Method for Detecting Violence, Smart Violence Monitoring System having the same - Google Patents

Abstract

The present invention relates to a device and method for detecting violence, and a smart violence detection system comprising the same. According to one embodiment of the present invention, provided is the device for detecting violence comprising: an object detection part that detects an object by analyzing a surveillance image, and analyzes at least one among a pose and action of the detected object; an object analysis part that identifies a perpetrator and victim through face detection and face identification, and analyzes a change factor set to reflect a damage of the victim as time goes by; and a violence occurrence determination part that determines whether the analyzed change factor satisfies a violence occurrence condition and transmits a determination result to the outside.

Description

Apparatus and Method for Detecting Violence, Smart Violence Monitoring System having the same}

The present invention relates to a violence detection device and method, and a smart violence monitoring system including the same, and more particularly, a violence detection device capable of detecting violence or abnormal behavior against childcare by analyzing images collected in childcare facilities, and method, and a smart violence monitoring system including the same.

Recently, many cases of child abuse by teachers in child care facilities such as daycare centers and kindergartens have become a social problem.

In order to solve this problem, surveillance cameras are installed in childcare facilities, but constant analysis is not possible, and harsh behavior is only confirmed after parents confirm the child's condition by requesting video from the surveillance camera.

In addition, there is no way to check the teacher's behavior at all times even in the daycare center, and there is a need for a plan to prevent accidents in advance.

Therefore, it is necessary to prepare mutually acceptable prevention measures between child care facilities and parents.

Republic of Korea Patent Publication No. 10-2009-0035379

The technical problem to be achieved by the present invention is to provide a violence detection device and method capable of detecting violence or abnormal behavior against childcare by analyzing images collected in childcare facilities, and a smart violence monitoring system including the same.

In addition, the technical problem to be achieved by the present invention is to provide a violence detection device and method that can reflect the specificity of childcare facilities and can replace high-performance equipment, and a smart violence monitoring system including the same.

The technical problem to be achieved by the present invention is not limited to the above-mentioned technical problem, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

In order to achieve the above technical problem, according to an embodiment of the present invention, an object is detected by analyzing a surveillance image, and through an object detection unit that analyzes at least one of a pose and a motion of the detected object, face detection and face identification. An object analysis unit that identifies perpetrators and victims and analyzes the change factors set to reflect the victim's damage over time, and violence that determines whether the analyzed change factors satisfy the conditions for occurrence of violence and transmits the judgment result to the outside. It provides a violence detection device including an occurrence determination unit.

In an embodiment of the present invention, the object detection unit analyzes the pose of the detected object to recognize the action, analyzes whether the recognized action is related to violence using a set analysis model, and measures the city center distance between the plurality of objects. By measuring , overlapping objects can be identified.

In an embodiment of the present invention, the object analyzer may identify an offender and a victim by performing face detection and face identification of overlapping objects using at least one of a face recognition model and a facial feature information extraction model.

In an embodiment of the present invention, the object analysis unit includes at least one of a scene including a facial state of the victim and a pose of the victim so as to reflect a change in expression and/or emotion over time. can be analyzed.

In an embodiment of the present invention, the object detector may extract audio data from the surveillance image through audio analysis, and the object analyzer may recognize sound by receiving the audio data extracted from the object detector.

In an embodiment of the present invention, the violence occurrence determination unit classifies the facial expressions of the victim, the poses and sounds of the offender and the victim into each case of scene-based violence, character pose-based violence, and sound-based violence, and set for each case. Scores are measured in this way, and the occurrence of violence can be determined by adding up the measured scores and comparing them with a standard score.

In an embodiment of the present invention, the violence occurrence determination unit uses the violence detection model in a scene-based violence case, CNN-based classification to learn intra-frame features, LSTM-based classification to learn inter-frame features by time, invariant/local Conducting optical flow-based classification that learns the characteristics of a score to determine violence or general, or CNN-based pose judgment that learns features within a frame in a character pose-based violence case, distance between objects and calculation of whether a reference value is exceeded The score can be measured to determine violence or general, or the score can be measured to determine normal or abnormal by performing LSTM-based sound classification in sound-based violence cases.

In order to achieve the above technical problem, according to another embodiment of the present invention, a photographing device installed in a child care facility and generating a surveillance image by photographing at least one subject, a violence detecting device, and violence occurring from the violence detecting device Provides a smart violence monitoring system that includes a control server that receives and reconfirms surveillance images of notifications and suspicious signs, and notifies the guardians of victims of violence.

In an embodiment of the present invention, the control server may provide the violence detection device with a determination value for confirming the occurrence of violence through analysis of the surveillance video as additional data for learning.

In order to achieve the above technical problem, according to another embodiment of the present invention, detecting an object from a surveillance video through image analysis, detecting a pose for each object, and recognizing an action to determine whether the recognized action is related to violence. Determining step, measuring the city center distance between objects, identifying overlapping objects using the measured city center distance, identifying the perpetrator and victim through face detection and face identification of the objects identified as overlapping, Analyzing change factors that reflect changes in facial expressions and/or emotions of the victim, recognizing sounds through audio analysis, and determining whether the recognized sounds are related to violence, and analyzing the change factors and the recognized sounds. It provides a method for detecting violence, including the step of determining the occurrence of violence by comparing it with criteria for determining the occurrence of violence.

In an embodiment of the present invention, in the step of determining the occurrence of violence, CNN-based classification for learning intra-frame features using the violence detection model in a scene-based violence case, LSTM-based classification for learning inter-frame features for each time, Measuring scores to determine violence or general by performing optical flow-based classification that learns immutable/local features, CNN-based pose judgment that learns in-frame features in character pose-based violent cases, distance between objects and exceeding reference values Calculating whether or not to measure the score to determine violence or general, performing LSTM-based sound classification in sound-based violence cases to measure scores to determine normal or abnormal, and scoring in a manner set for each case The method may include measuring, summing up the measured scores, and then determining the occurrence of violence by comparing them with a reference score.

According to an embodiment of the present invention, it is possible to detect violence or abnormal behavior against children in child care by analyzing images collected in child care facilities.

In addition, the technical problem to be achieved by the present invention can reflect the specificity of childcare facilities and can replace high-performance equipment.

The effects of the present invention are not limited to the above effects, and should be understood to include all effects that can be inferred from the description of the present invention or the configuration of the invention described in the claims.

1 is a diagram showing the configuration of a smart violence monitoring system according to an embodiment of the present invention.
2 is a diagram showing the configuration of a violence detection device according to an embodiment of the present invention.
3 is a diagram showing an example of a posture and motion analysis model according to an embodiment of the present invention.
4 is a diagram exemplarily illustrating overlap between objects according to an embodiment of the present invention.
5 is a diagram showing an example of a face recognition model according to an embodiment of the present invention.
6 is a diagram illustrating an example of a facial feature information extraction model according to an embodiment of the present invention.
7 is a diagram showing an example of a facial expression analysis model according to an embodiment of the present invention.
8 is a flowchart illustrating a violence detection method according to an embodiment of the present invention.
9 is a flowchart showing in detail the violence occurrence determination step according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and, therefore, is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this is not only "directly connected", but also "indirectly connected" with another member in between. "Including cases where In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing the configuration of a violence detection device and method according to an embodiment of the present invention, and a smart violence monitoring system including the same. 2 is a diagram showing the configuration of a violence detection device according to an embodiment of the present invention. 3 is a diagram showing an example of a posture and motion analysis model according to an embodiment of the present invention. 4 is a diagram exemplarily illustrating overlap between objects according to an embodiment of the present invention. 5 is a diagram showing an example of a face recognition model according to an embodiment of the present invention. 6 is a diagram illustrating an example of a facial feature information extraction model according to an embodiment of the present invention. 7 is a diagram showing an example of a facial expression analysis model according to an embodiment of the present invention.

1 to 7, a violence detection device and method according to an embodiment of the present invention, and a smart violence monitoring system including the same, include a photographing device 100, a violence detection device 200, and a control server 300. can include

The photographing device 100 may be installed in a childcare facility, photograph at least one target of childcare children and childcare workers (teachers), and generate a monitoring image including at least one of images and sounds related to the photographed target. To this end, the photographing device 100 may include a closed circuit (CCTV).

The violence detection device 200 detects an object from a surveillance video, identifies an assailant and a victim through an object detector 210 that analyzes at least one of the pose and action of the detected object, face detection and face identification, and time An object analysis unit 220 that analyzes change factors set to reflect the victim's damage situation over time, and a violence occurrence determination unit 230 that determines whether the analyzed change factors satisfy violence occurrence conditions and transmits the determination result to the outside. ) may be included.

Specifically, the object detector 210 may detect at least one object from a surveillance image captured by the photographing device 100 through image analysis. Here, the object detector 210 uses a detection algorithm including at least one of OpenPose, Codebook-based Background Modeling, and Human Pose Estimation. object can be detected in the image frame. For example, with reference to FIG. 3 , the object detection unit 210 detects key-points of main connecting parts (joints) of the body using the human pose estimation that detects a human pose in an image or video, and It can recognize poses (postures) such as standing, walking, kicking, and punching.

In addition, the object detection unit 210 recognizes an action (action) by analyzing the pose (posture) of each object of the childcare child and the childcare worker (teacher), and the recognized action (action) corresponds to violence such as kicking and punching. You can determine if it is related. Here, the object detector 210 may analyze the behavior of the object using at least one of a posture estimation model (eg, PoseNet, body key point) and a frame analysis model per time (eg, LSTM). For example, the object detection unit 210 recognizes an action in which a childcare worker (teacher) takes a pose (posture) instructing a specific direction, and childcare children suddenly flock to one side and wait, or all bow their heads, and then assault or punish them. It is possible to analyze the childcare worker's (teacher's) behavior related to at least one of , , and harsh behavior.

In addition, the object detection unit 210 may measure the center (centre) distance between each object of the child care child and the child care teacher (teacher). Here, the object detection unit 210 may identify overlapping objects (people) using the measured city center distance with reference to FIG. 4 . For example, the object detection unit 210 may measure the center distance between the upper bodies of each of the plurality of objects based on the upper body, and determine that the plurality of objects overlap when the measured distance is equal to or less than the arm length of the object. there is.

Meanwhile, the object detector 210 may extract audio data from the surveillance image received from the photographing device 100 through audio analysis.

The object analyzer 220 may identify an assailant and a victim through face detection and face identification of objects (people) identified as overlapping. Here, the object analyzer 220 may detect a face using a face recognition model (RetinaFace) as shown in FIG. 5, extract facial features using a facial feature information extraction model (Arcface) as shown in FIG. 6, and , The face can be identified using a feature information similarity analysis algorithm (eg, Cosign Similarity).

In addition, the object analysis unit 220 uses a facial expression model as shown in FIG. 7 to reflect the victim's facial expression change and / or emotion change over time, a scene including the victim's facial state, And it is possible to analyze the change factor including at least one of the victim's pose. Through this, the object analyzer 220 may analyze the change factors to determine whether the change in expression and/or emotion of the victim is related to violence. For example, the object analyzer 220 may check victim emotions such as surprise, sadness, tears, anger, fear, etc. through a change in facial expression of a child who has been assaulted, punished, or abused.

Meanwhile, the object analyzer 220 may recognize sound by receiving audio data extracted from the object detector 210 . Here, the object analyzer 220 may recognize sounds such as screaming, crying, and corporal punishment by applying a learning model learned using an artificial intelligence algorithm to audio data. In addition, the object analyzer 220 may check whether the recognized sound is related to violence, such as a scream, a beating, or a fall sound.

The violence occurrence determination unit 230 may compare the change factors analyzed by the object analysis unit 220 and the recognized sound with a violence occurrence determination standard to determine whether violence has occurred to the childcare child.

Specifically, the violence occurrence determination unit 230 classifies the victim's facial expression, the pose and sound of the offender and the victim into each case of scene-based violence, character pose-based violence, and sound-based violence, and in a manner set for each case. Scores are measured, and scores measured for each case are summed up and compared with a standard score (a criterion for determining the occurrence of violence), and then it can be determined whether the current situation is a violent situation or a general situation.

Here, the violence occurrence determination unit 230 takes the frames of the current time and the previous time as input data and combines deep-learning components CNN (Convolutional Neural Networks) + LSTM (Long Short-Term Memory) It is possible to determine the occurrence of violence using a violence detection model based on . For example, the violence occurrence determination unit 230 uses the violence detection model in a scene-based violence case to use CNN-based classification to learn intra-frame features, LSTM-based classification to learn inter-frame features over time, and immutable/local By performing an optical flow-based classification that learns features, it can be scored to determine violence or generality. Alternatively, the violence occurrence determination unit 230 may measure scores to determine violence or general by performing CNN-based pose determination that learns characteristics within a frame in a person pose-based violence case, distance between objects, and calculation of whether or not a reference value is exceeded. there is. Alternatively, the violence occurrence determination unit 230 may perform LSTM-based sound classification in the sound-based violence case and measure a score to determine whether it is normal or abnormal.

At this time, the violence occurrence determination unit 230 may calculate a score using Equation 1 below.

[Equation 1]

는 분석된 감시 영상의 총 시간이고,

는 피해자의 변화요소(얼굴 표정/감정) 분석에 따른 폭력 여부를 0 또는 1로 표현한 값이고,

는 피해자의 변화요소에 따른 가중치를 수치로 나타낸 값이며,

는 피해자의 변화요소 분석에 따른 폭력 확률을 0.00 내지 1.00로 표현한 값이다.here,

is the total time of the analyzed surveillance video,

is a value expressed as 0 or 1 whether or not there is violence according to the analysis of the victim's changing factors (facial expression/emotion),

is a numerical value representing the weight according to the change factor of the victim,

is a value expressing the probability of violence from 0.00 to 1.00 according to the analysis of the changing factors of the victim.

In addition, the violence occurrence determination unit 230 classifies the violence level into suspicious (score 0 to 10), normal (score 10 to 20), dangerous (score 20 to 40), and serious (score 40 to 200), and sums If the scored score is greater than or equal to danger, it can be determined that the current situation is a violent situation.

When the violence determination unit 230 determines that a violent situation occurs, the surveillance video of the suspicious sign may be transmitted to the control server 300 together with the violence occurrence notification.

In the control server 300 , a controller and/or a worker reconfirms the suspicious symptom monitoring image received from the violence detection device 200 to determine whether violence has occurred, and can notify the guardian of the victim of the occurrence of violence. At this time, the control server 300 may notify the occurrence of violence to the guardian's user terminal 400 in the form of at least one of text, video, and sound.

In addition, the control server 300 may provide the violence detection device 200 with a determination value for confirming the occurrence of violence through analysis of the surveillance video as additional learning data. The violence detection device 200 that has received the additional training data may perform enhancement of the violence detection model through additional AI model learning, and may create, update, and/or distribute a new weight file.

Hereinafter, a violence detection method according to an embodiment of the present invention will be described with reference to FIGS.

FIG. is a flow chart showing a method for detecting violence according to an embodiment of the present invention, and FIG. Here, the violence detection method is described with reference to the smart violence monitoring system according to an embodiment of the present invention described above, but redundant descriptions can be omitted for convenience.

The violence detection method according to an embodiment of the present invention includes detecting an object from a surveillance video through image analysis (S110), detecting a pose for each object (S120), recognizing an action (S130), Determining whether it is related to violence (S140), measuring the distance between objects (S150), checking overlapping objects (people) using the measured distance (S160), overlapping objects (persons) ) through face detection and face identification to identify perpetrators and victims, and analyzing change factors reflecting changes in the victim's expression and/or emotion over time (S170), determining whether the expression is related to violence Step (S180), analyzing audio from surveillance images (S210), recognizing sound using audio data extracted through audio analysis (S220), determining whether the recognized sound is related to violence ( S230), determining the occurrence of violence by comparing the change factor and the recognized sound with a criterion for determining occurrence of violence (S300), and transmitting a notification of occurrence of violence.

In step S110 , the object detector 210 may detect at least one object from the surveillance image captured by the photographing device 100 through image analysis.

In step S120, the object detector 210 may detect a pose by connecting main connection parts in the detected object.

In step S130, the object detector 210 may recognize an action (behavior) by analyzing the pose (posture) of the object.

In step S140, the object detector 210 may determine whether the recognized action (behavior) is related to violence such as kicking or punching.

In step S150, the object detection unit 210 may measure the center (centre) distance between each object of the child care child and the caregiver (teacher).

In step S160, the object detection unit 210 may check overlapping objects (people) using the measured city center distance.

In step S170, the object analyzer 220 may identify an assailant and a victim through face detection and face identification of objects (people) identified as overlapping. Here, the object analyzer 220 may detect a face using a face recognition model (RetinaFace), extract facial features using a facial feature information extraction model (Arcface), and use a feature information similarity analysis algorithm (eg : Faces can be identified using Cosign Similarity. In addition, the object analyzer 220 analyzes a change factor including at least one of a scene including the victim's face state and a victim's pose so as to reflect the victim's expression change and/or emotion change over time. can do.

In step S210, the object detector 210 may extract audio data from the surveillance image received from the photographing apparatus 100 through audio analysis.

In step S220, the object analyzer 220 may receive the audio data extracted from the object detector 210 and recognize the sound.

In step S230, the object analyzer 220 may determine whether the recognized sound is a sound related to violence, such as a scream, a beating, or a fall.

In step S300, CNN-based classification for learning intra-frame features using the violence detection model in a scene-based violence case, LSTM-based classification for learning inter-frame features over time, and optical flow-based classification for learning immutable/local features score to determine violence or general, CNN-based pose judgment that learns features within a frame in case of person pose-based violence, distance between objects and whether a reference value is exceeded, and score to judge violence or general Step of measuring, measuring scores to determine normal or abnormal by performing LSTM-based sound classification in sound-based violence cases, and measuring scores in a manner set for each case, summing the measured scores, and then obtaining a reference score and It may include a step of determining the occurrence of violence by comparing.

According to an embodiment of the present invention, it is possible to detect violence or abnormal behavior against children in child care by analyzing images collected in child care facilities.

In addition, the technical problem to be achieved by the present invention can reflect the specificity of childcare facilities and can replace high-performance equipment.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

100: photographing device
200: violence detection device
210: object detection unit
220: object analysis unit
230: violence occurrence determination unit
300: control server
400: user terminal

Claims (11)

an object detector configured to detect an object by analyzing the surveillance image and to analyze at least one of a pose and a motion of the detected object;
an object analysis unit that identifies an offender and a victim through face detection and face identification, and analyzes change factors set to reflect the victim's damage situation over time; and
a violence occurrence determination unit that determines whether the analyzed change factors satisfy violence occurrence conditions and transmits the determination result to the outside;
Including, violence detection device.
According to claim 1,
The object detection unit,
Characterized by analyzing the pose of the detected object to recognize the action, using a set analysis model to analyze whether the recognized action is related to violence, and measuring the city center distance between a plurality of objects to identify overlapping objects, violence detector.
According to claim 2,
The object analysis unit,
An apparatus for detecting violence characterized by identifying an offender and a victim by performing face detection and face identification of overlapping objects using at least one of a face recognition model and a facial feature information extraction model.
According to claim 3,
The object analysis unit,
A violence detection device characterized by analyzing the change factor including at least one of a scene including a victim's face state and a victim's pose to reflect a change in facial expression and/or emotion over time.
According to claim 1,
The object detection unit,
Extracting audio data from the surveillance video through audio analysis;
The object analysis unit,
Characterized in that the sound is recognized by receiving the audio data extracted from the object detection unit, the violence detection device.
According to claim 5,
The Violence Determination Division,
The facial expression of the victim, the pose and sound of the perpetrator and the victim are classified into each case of scene-based violence, character pose-based violence, and sound-based violence, scores are measured in a method set for each case, and the measured scores are summed up. A violence detection device characterized in that it determines the occurrence of violence by comparing it with a reference score.
According to claim 6,
The Violence Determination Division,
In scene-based violence cases, CNN-based classification that learns intra-frame features using the violence detection model, LSTM-based classification that learns inter-frame features over time, and optical flow-based classification that learns immutable/local features are used to detect violence or Scoring to judge the general, or
In the case of person pose-based violence, CNN-based pose judgment that learns features within a frame, distance between objects, and calculation of whether or not a reference value is exceeded are used to measure scores to determine violence or general,
An apparatus for detecting violence characterized by measuring scores to determine normal or abnormal by performing LSTM-based sound classification in sound-based violence cases.
A photographing device installed in a childcare facility and generating a surveillance image by photographing at least one subject;
A violence detection device according to any one of claims 1 to 7; and
a control server that receives and reconfirms the surveillance video of the violence occurrence notification and suspicious signs from the violence detection device, and notifies the guardian of the victim of the occurrence of violence;
Including, smart violence monitoring system.
According to claim 8,
The control server,
A smart violence monitoring system, characterized in that for providing a determination value confirming the occurrence of violence through analysis of surveillance video to the violence detection device as additional learning data.
Detecting an object from a surveillance image through image analysis;
Detecting a pose for each object, recognizing an action, and determining whether the recognized action is related to violence;
measuring centroid distances between objects and identifying overlapping objects using the measured centroid distances;
Identifying an assailant and a victim through face detection and face identification of objects identified as overlapping, and analyzing change factors reflecting changes in facial expression and/or emotion of the victim over time;
recognizing a sound through audio analysis and determining whether the recognized sound is related to violence; and
determining the occurrence of violence by comparing the change factor and the recognized sound with a criterion for determining occurrence of violence;
Including, violence detection method.
According to claim 10,
In the step of determining the occurrence of violence,
In scene-based violence cases, CNN-based classification that learns intra-frame features using the violence detection model, LSTM-based classification that learns inter-frame features over time, and optical flow-based classification that learns immutable/local features are used to detect violence or measuring the score to judge the general;
Measuring a score to determine violence or general by performing a CNN-based pose judgment that learns characteristics within a frame in a case of person pose-based violence, distance between objects, and calculation of whether a reference value is exceeded;
Measuring a score to determine normal or abnormal by performing LSTM-based sound classification in the sound-based violence case; and
Measuring scores in a manner set for each case, summing up the measured scores, and then comparing them with a reference score to determine occurrence of violence;
Characterized in that it comprises a, violence detection method.

Images