KR102518615B1 - Apparatus and Method for complex monitoring to judge abnormal sound source - Google Patents

Abstract

Disclosed is a device capable of detecting an abnormal situation by collecting images and sound sources using an anomaly detection device and performing complex analysis on the collected information. The device includes an anomaly detector that collects video data and sound source data, an analysis unit that determines an abnormal situation by complexly analyzing the video data and the sound source data, and generates abnormal situation information, and transmits and stores the abnormal situation information. It is characterized by including wealth.

Description

Apparatus and Method for complex monitoring to judge abnormal sound source}

The present invention relates to a technology for determining an abnormal sound source, and more particularly, to a device and method capable of detecting an abnormal situation by collecting images and sound sources using an integrated abnormality detection device and complexly analyzing the collected information. .

Artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning. Machine learning is an algorithm technology that classifies/learns the characteristics of input data by itself, and element technology is a technology that utilizes machine learning algorithms such as deep learning, such as linguistic understanding, visual understanding, inference/prediction, knowledge expression, motion control, etc. consists of the technical fields of

In recent years, as artificial intelligence technology develops, various technologies are pouring out. In particular, in the field of recognizing audio data such as voice and sound, studies on recognizing abnormal situations from audio data are being actively conducted.

This may be a method of taking action according to the situation more effectively by detecting an abnormal sound source in the audio data and notifying the supervisor of the abnormal situation beyond the limits of the abnormal situation recognition method through video data captured by CCTV (Closed-circuit Television).

However, in the case of the existing multi-sensing CCTV, it is determined whether an abnormality has occurred based on sound information. In other words, a sensor matrix map for a plurality of complex sensing CCTVs is created based on the installation location information and installation direction information of the complex sensing CCTV, and when abnormal sound is generated, the location of the abnormal sound is determined based on the sensor matrix map. way to calculate it.

However, such an anomaly detection device has a probability of false positives by using only analysis of acoustic information. In addition, there is a problem in that an abnormal situation in which a small sound is generated cannot be detected because an image is not analyzed.

In addition, there is a problem in that the ability to cope with an actual emergency situation is reduced because the voice such as a rescue request is not recognized.

1. Republic of Korea Patent Registration No. 10-1543561 (registration date: August 5, 2015)

The present invention has been proposed to solve the problems caused by the above background art, and provides a device and method capable of detecting an abnormal situation by collecting images and sound sources using an integrated abnormality detection device and complexly analyzing the collected information. It has a purpose.

In addition, another object of the present invention is to provide a device and method capable of accurately detecting an abnormal situation in which a small sound is generated.

In addition, another object of the present invention is to provide an apparatus and method capable of improving the ability to cope with an actual emergency situation by accurately recognizing a voice such as a rescue request.

In order to achieve the object presented above, the present invention provides a device capable of detecting an abnormal situation by collecting images and sound sources using an anomaly detection device and complexly analyzing the collected information.

The device,

an anomaly detector that collects image data and sound source data;

an analyzer configured to perform a complex analysis of the video data and the sound source data to determine an abnormal situation and generate abnormal situation information; and

It is characterized in that it includes; a storage transmission unit for transmitting the abnormal situation information.

In addition, the anomaly detector may include an image collection unit generating the image data; and a sound source collection unit generating the sound source data.

In addition, the sound source collection unit, two microphones for generating sound source data by collecting the sound source; and a direction detecting module for detecting the direction of the sound source and generating sound source direction information.

In addition, the sound source collection unit, a body in which the direction detection module is configured; and a sound collecting plate integrally disposed on the left and right sides of the body to collect sound sources and having two microphones disposed at edges.

In addition, the sound collecting plate is characterized in that it is a paraboloid type to enhance the sound collecting effect of the sound source.

In addition, the two microphones are characterized in that they are installed in a direction facing the reflected sound source at a sound collecting point where the sound source reflected by the sound collecting plate is focused.

The analysis unit may further include: an image analysis module for recognizing an object by analyzing the image data, detecting an abnormal behavior based on information of the object, and generating abnormal behavior information; a sound source analysis module recognizing an abnormal sound source from the sound source data and generating sound source direction information of the abnormal sound source; and a complex analysis module generating the surrounding abnormal situation information according to a predefined event processing rule using the abnormal behavior information and the sound source direction information.

The image analysis module may include a pre-processing unit pre-processing the image data to generate a standardized standard image frame; an image analysis unit detecting and classifying the object by applying a neural network classification model to the standard image frame; and an abnormal behavior detection unit generating the abnormal behavior information by detecting the abnormal behavior using the recognized object.

In addition, in order to increase the reliability of the detected object, it is characterized in that an ensemble of classifiers learned according to the field situation is analyzed.

In addition, the abnormal sound source is characterized in that it is an abnormal sound including a scream, an explosion sound, a collision sound, or a vehicle sudden stop, or an abnormal sound having a specific word for a rescue request.

In addition, when the first event caused by the occurrence of the abnormal behavior information or the sound source direction information corresponds to a priority condition event predefined according to the event processing rule, the abnormal situation information indicating a first-level alarm is generated. .

In addition, when the second event exceeds a preset first weight, the abnormal situation information indicating a second-stage alarm is generated, and the first weight is a value given to an event according to sound analysis or voice analysis. .

In addition, when the first event does not correspond to the priority condition event and a video event due to the occurrence of the abnormal behavior information occurs, a first set value for which the sum of weights is preset by adding a first weight and a second weight is obtained. If it exceeds, the abnormal situation information indicating the second-stage alarm is generated, and the second weight is a value given to an event according to video analysis.

In addition, when the first event does not correspond to the priority condition event and a video event due to the occurrence of the abnormal behavior information occurs, and the second weight exceeds a preset second set value, the second-stage alarm is displayed. It is characterized in that the abnormal situation information is generated.

In addition, if the first event does not correspond to the priority condition event and no video event occurs due to the occurrence of the abnormal behavior information, and the second weight exceeds a preset third set value, the first-stage alarm When the abnormal situation information indicating is generated and the second weight does not exceed a preset third set value, a search for another device is requested to the operator terminal.

On the other hand, another embodiment of the present invention, (a) the anomaly detector collecting image data and sound source data; (b) generating abnormal situation information by determining an abnormal situation through a complex analysis of the video data and the sound source data by an analysis unit; and (c) transmitting the abnormal situation information by a storage/transmission unit.

According to the present invention, it is possible to detect an abnormal situation by collecting images and sound sources using an integrated anomaly detection device and complexly analyzing the collected information.

In addition, another effect of the present invention is that even in an abnormal situation in which a small sound is generated, it can be accurately sensed.

In addition, another effect of the present invention is that it is possible to improve the ability to cope with an actual emergency situation by accurately recognizing a voice such as a rescue request sound.

1 is a configuration block diagram of a complex monitoring device according to an embodiment of the present invention.
FIG. 2 is an external perspective view of the sound source collection unit shown in FIG. 1 .
FIG. 3 is an enlarged perspective view of a portion of the sound source collection unit shown in FIG. 2 .
4 is a perspective view of the microphone shown in FIG. 3;
FIG. 5 is a block diagram of the image collecting unit shown in FIG. 1 .
FIG. 6 is a detailed block diagram of the image analysis module shown in FIG. 1 .
7 is a flowchart illustrating a process of determining an abnormal sound source according to an embodiment of the present invention.
8 is a flowchart illustrating a process of generating event results by applying weights according to an embodiment of the present invention.
9 is a diagram showing a concept of 3-channel feature point extraction according to an embodiment of the present invention.
10 is a flowchart illustrating a process of generating a classification model according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

In describing each figure, like reference numbers are used for like elements.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term "and/or" includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in this application, they should not be interpreted in ideal or excessively formal meanings. Should not be.

Hereinafter, a complex monitoring apparatus and method for determining an abnormal sound source according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a complex monitoring apparatus 100 according to an embodiment of the present invention. Referring to FIG. 1 , the complex monitoring device 100 includes an anomaly detector 101 that collects video data and sound source data, and an analysis unit that determines an abnormal situation by complexly analyzing the video data and sound source data and generates abnormal situation information. 130, a storage and transmission unit 140 for transmitting abnormal situation information, and the like.

The anomaly detector 101 may include an image collection unit 110 that collects image data, a sound source collection unit 120 that collects sound source data, and the like.

The image collection unit 110 may include a closed-circuit television (CCTV) 111. Of course, in addition to CCTV, it may be a CCD (Charge-Coupled Device) camera, CMOS (Complementary Metal-Oxide Semiconductor) camera, IP (Internet Protocol) camera, etc.

The sound source collecting unit 120 includes a sound collecting plate 121 that collects sound sources, a microphone 122 that collects sound sources collected by the sound collecting plate 121 and generates sound source data, detects the direction of the sound source and obtains sound source direction information. It may be configured to include a direction detection module 123 and the like to generate.

The microphone 122 may consist of two or more microphones. That is, they can be installed side by side side by side.

The direction detection module 123 may include a microcomputer including an algorithm that calculates the arrival time difference of the sound signals received from each of the two or more microphones and estimates the direction of the sound source using the difference. Of course, the direction sensing module 123 can be implemented in other ways as well.

The analysis unit 130 analyzes image data to recognize an object, detects abnormal behavior based on object information, and generates abnormal behavior information. The image analysis module 131 recognizes an abnormal sound source from sound source data, It may be configured to include a sound source analysis module 132 that generates direction information, a complex analysis module 133 that generates surrounding abnormal situation information according to a predefined complex analysis rule using abnormal behavior information and direction information, and the like. there is. Abnormal sound sources may include screams, explosions, crashes, vehicle-level stop sounds, and the like.

The storage transmission unit 140 streams the collected image data and sound source data (141), and stores generated event information in the event DB 142. In addition, event processing rules are predefined and stored in the composite analysis rule DB 143 . In addition, the storage and transmission unit 140 performs a function of generating alarm information according to abnormal situation information and transmitting it to the control system 20 .

The storage transmission unit 140 implements DB (Database), flash memory type for streaming, hard disk type, multimedia card micro type, card type memory ( SD (Secure Digital) or XD (eXtreme Digital) memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium.

In addition, the image analysis module 131, the sound source analysis module 132, and the complex analysis module 133 shown in FIG. 1 refer to units that process at least one function or operation, and are implemented in software and/or hardware. It can be. In hardware implementation, ASIC (application specific integrated circuit), DSP (digital signal processing), PLD (programmable logic device), FPGA (field programmable gate array), processor, microprocessor, other It may be implemented as an electronic unit or a combination thereof.

In software implementation, software component components (elements), object-oriented software component components, class component components and task component components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, data , databases, data structures, tables, arrays, and variables. Software, data, etc. may be stored in memory and executed by a processor. The memory or processor may employ various means well known to those skilled in the art.

The operator can define events through the operator terminal 10, set event processing rules, and adjust them according to the installation environment. The operator terminal 10 may be a mobile phone, a smart phone, a laptop computer, a digital broadcast terminal, a note pad, a desktop computer, and the like.

In addition, the components shown in FIG. 1 are shown as being configured separately, but may be configured integrally.

FIG. 2 is an external perspective view of the sound source collecting unit 120 shown in FIG. 1 . Referring to FIG. 2, the sound source collection unit 120 is integrally disposed on the left and right sides of the circular body 210 and the body 210 to collect sound sources, and is disposed on the edge of the sound collecting plate 121. It may be configured to include a microphone 122 for collecting sound sources. A direction detection module 123 may be configured inside the body 210 .

FIG. 3 is a partially enlarged perspective view of the sound source collection unit 120 shown in FIG. 2 . Referring to FIG. 3 , the sound collecting plate 121 takes on a paraboloid shape to enhance the sound collecting effect of the sound source. In addition, a microphone 122 is installed in a direction facing the reflected sound source at the sound collecting point 310 where the sound source reflected by the sound collecting plate 121 is focused.

FIG. 4 is a perspective view of the microphone 122 shown in FIG. 3 . Referring to FIG. 4 , the microphone 122 is installed on the inner and outer walls of the housing 410 . Of course, a through hole 411 is formed in the housing 410 for fastening with the sound collecting plate 121 .

FIG. 5 is a block diagram of the image collecting unit 110 shown in FIG. 1 . Referring to FIG. 5, an image receiving module 510 for receiving a video stream from a camera, an image decoding module 520 for generating an image frame by decoding the video stream, and converting the image frame into RGB (Red Green Blue) format image data It may be configured to include an image conversion module 530 that converts to .

FIG. 6 is a detailed block diagram of the image analysis module 131 shown in FIG. 1 . Referring to FIG. 6 , the image analysis module 131 includes a preprocessing unit 610 that preprocesses image data to generate a standardized standard image frame, and an image that detects and classifies objects by applying a neural network classification model to the standard image frame. It may be configured to include an analysis unit 620, an abnormal behavior detection unit 630 that detects abnormal behavior using the recognized object and generates abnormal behavior information, and the like.

The pre-processing unit 610 is a part that adjusts the precision of image data to be suitable for analysis, standardizes the pixel values of the image and converts them into 32-bit floating point data between [0,1] so that computational resources can be used efficiently. .

The image analysis unit 620 detects and classifies an object using a neural network classification model (eg, an EfficientDet model) in order to increase the accuracy of prediction classification for standardized image frames standardized through preprocessing. The neural network classification model recognizes objects through deep learning-based machine learning.

In order to increase the reliability of the detected object, an ensemble of object classifiers learned according to the field situation is analyzed. EfficientDet can use a pretrained pretrained model such as EfficientNet with high classification accuracy as a backbone model.

The object classifier is used to improve accuracy deterioration due to false positives occurring in the object detection process, and the frequently occurring false positive situations are as follows.

1) Localization error: when there is no object to be detected but it is detected.

2) Object Classification Error: When the localization is correct but the object classification is incorrect.

To solve this problem, the object classifier receives the pixel value of the detected object area and performs secondary analysis to classify the background or object type to be detected. For example, if the purpose is to classify people and vehicles, three types of objects are classified: background, people, and vehicles. Therefore, when a localization error occurs, the classifier outputs the background as a result, and otherwise outputs the object to be detected as a result.

However, since the background has various characteristics compared to the object to be detected, there is still a possibility of false positives. The ensemble method used to solve this problem outputs the final result through voting on various results obtained by using several classifiers learned under various conditions. Voting-based ensemble methods include hard voting and soft voting, and in this technology, probability-based voting is used to minimize uncertainty due to diversity of backgrounds.

The abnormal behavior detection unit 630 determines the event result by applying weights to the object (human or vehicle) recognized by the video analysis unit 620 invading or wandering in a predefined specific area or traffic accident such as a vehicle collision. send.

The preprocessing unit 610, the image analysis unit 620, and the abnormal behavior detection unit 630 shown in FIG. 6 refer to units that process at least one function or operation, and may be implemented in software and/or hardware. there is. In hardware implementation, ASIC (application specific integrated circuit), DSP (digital signal processing), PLD (programmable logic device), FPGA (field programmable gate array), processor, microprocessor, other It may be implemented as an electronic unit or a combination thereof.

In software implementation, software component components (elements), object-oriented software component components, class component components and task component components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, data , databases, data structures, tables, arrays, and variables. Software, data, etc. may be stored in memory and executed by a processor. The memory or processor may employ various means well known to those skilled in the art.

7 is a flowchart illustrating a process of determining an abnormal sound source according to an embodiment of the present invention. Referring to FIG. 7 , the analyzer 130 analyzes captured image data based on deep learning, and at the same time analyzes acquired sound source data based on deep learning (steps S710 and S720).

Then, image analysis is performed for an abnormal situation from the image data (step S711). To elaborate, object recognition is performed through deep learning-based machine learning, and abnormal behavior is detected based on the information of the recognized object. The object may be, for example, a person or a vehicle, and the abnormal behavior may be trespassing, wandering, or collision.

Meanwhile, sound source recognition is performed from the sound source data (step S721). In other words, the sound source may consist of abnormal sounds such as screams, explosions, crashes, and vehicle sudden stops, and abnormal voices with specific words for rescue requests, such as “help me”, “help me”, and “fire”. . Of course, the sound source may be defined differently.

Therefore, it is possible to recognize abnormal sounds such as screams, explosions, collisions, and sudden vehicle stops, and to recognize the direction of a generated sound source. Of course, by analyzing voices, it is possible to recognize specific words for rescue requests, such as “help me,” “help me,” and “fire,” and recognize the direction of the generated sound source.

An event is generated due to this recognition of the sound source, and the analyzer 130 determines a situation around the multi-sensing CCTV according to a predefined event processing rule for this event (step S730). Event definitions can be set in the operator terminal 10 and can be adjusted according to the installation environment.

Then, the analysis unit 130 first determines whether a condition event has occurred (step S740). In other words, according to predefined event handling rules, it is first checked whether a condition event has occurred. Examples of judgment conditions according to predefined event processing rules are as follows.

event weight preferential conditions selection condition image analysis person - blanket vehicle - blanket crash 50 blanket prowl 50 blanket sound analysis scream 50 blanket belch 100 blanket crash sound 100 blanket sudden stop 50 blanket voice analysis rescue request 200 blanket

Therefore, if the first event caused by the occurrence of the abnormal behavior information or the sound source direction information corresponds to a priority condition event predefined according to the event processing rule, the abnormal situation information indicating the first stage alarm is generated and the control system 20 ) (step S741).

Thereafter, when the preset weight is exceeded, abnormal situation information indicating a second-stage alarm is generated and transmitted to the control system 20 (steps S760 and S770).

Complex events are judged according to weights and complex analysis is performed through events that occur within a preset time (approximately 30 seconds). When a priority condition event occurs, a first-level alarm (warning) and a second-level alarm (warning) are generated when the maximum weight exceeds 200, and transmitted to the control system.

Meanwhile, in step S740, if the first condition event does not occur, it is checked whether a video event has occurred (step S750).

In step S750, when the maximum sum of weights (that is, the sum of the first weights according to sound analysis/voice analysis and the second weights according to video analysis) exceeds 200 in a state where a video event is detected, a second-stage alarm (warning) is transmitted to the control system. (steps S743 to S770).

When the weight exceeds 200 in a state where no video event is detected, it is transmitted to the first-stage alarm control system, and a search request for other CCTV devices is requested to the operator's terminal.

The first weight is a value assigned to an event based on sound analysis or voice analysis, and the second weight is a value assigned to an event based on image analysis.

8 is a flowchart illustrating a process of generating event results by applying weights according to an embodiment of the present invention. Referring to FIG. 8 , image data is acquired and standardized image data is standardized to generate a standard image frame (steps S810 and S820).

Afterwards, the object is detected and classified using a neural network classification model (eg, EfficientDet) to increase the accuracy of prediction classification for the standard video frame. In order to increase the reliability of the detected object, an ensemble of learned classifiers according to the field situation is analyzed (step S830).

Thereafter, the abnormal behavior is analyzed, and an event is generated by giving a weight (steps S840, S850, and S860).

9 is a diagram showing a concept of 3-channel feature point extraction according to an embodiment of the present invention. Referring to FIG. 9 , a raw component source, a harmonic component source, and a percussive component source 930 are obtained through channel division 920 for obtained audio data (ie sound source data) 910 . ), perform -1 to 1 peak normalization (940) for each channel, and use the peak normalization data to log-Mel-spectrogram. ) is extracted, and feature points 950 for each channel are combined (960) to generate channel feature points 970 for the classification model (Efficient-Net).

In the case of the Log-Mel spectrogram, Log-Mel is a scale conversion function that reflects the threshold for recognizing a person's sound found in pitch. For example, it can be Mel(f) = 2595 log(1+f/700). f is the frequency. A spectrogram is a tool for visualizing and grasping sound or waves, and is a combination of characteristics of a waveform and a spectrum.

10 is a flowchart illustrating a process of generating a classification model according to an embodiment of the present invention. Referring to FIG. 10, when audio data is acquired, the audio data is divided into front audio and rear audio (S1010, S1011, S1020, S1030). This is to secure higher accuracy by confirming the correlation between the sound generated from the front and the sound generated from the rear.

Thereafter, feature points for each channel are extracted for front audio and rear audio (steps S1021 and S1031).

After that, a time-distributed method is used to extract an accurate weight for correlation. If you use the TimeDistributed method, it serves to connect the output output at each time with the layer declared inside.

In the case of short sound sources, good results are obtained without using the Timedistributed method, but for long sound sources of more than 5 seconds, better results are obtained by using the Timedistributed method.

Channel data extracted from audio data can be used the same as images, so a CNN-based pretrained model such as Efficient-Net with high image classification accuracy can be used (S1040, S1050).

Weights are extracted from channel feature points through a pre-learning model, spatiotemporal information is added, and weights (Auto-Pool ID) are reconstructed using image + spatiotemporal information to generate a resultant model (steps S1041 to S1060).

In more detail, it is as follows.

The last DNN layer is removed from the pretrained model through image data to use only convolutional weights (steps S1040 and S1050).

After that, dropout is used to prevent overfitting (steps S1041 and S1051). Dropout is one of the methods to solve overfitting of a model by partially omitting neurons in a neural network.

Matrix data is combined into one channel using the flatten layer and expressed (steps S1043 and S1053). The flatten layer is a layer that converts the extracted main features into one-dimensional data to deliver to the fully coupled layer. It flattens the data in the form of an image into an array form.

Learning is performed through the finally combined weights (steps S1045 and S1055). That is, learning proceeds using the dense layer. The dense layer is a layer used in neural networks and connects both input and output. For example, if there are 4 input neurons and 8 output neurons, the total number of connections is 4x8=32. Each connection line contains a weight, which means connection strength.

The learned result is output in a sigmoid form (steps S1047 and S1057). Sigmoid　 is a function that shows a gentle sigmoid curve 　 similar to the letter S.

An optimal result (ie, Auto-pool 1D) is calculated through Max pool and Average pool through the two sigmoid results (step S1060).

Then, weights are extracted by applying the channel feature points to the pre-learned model, and a classification model is generated by applying the weights.

In addition, the steps of a method or algorithm described in connection with the embodiments disclosed herein are implemented in the form of program instructions that can be executed through various computer means such as a microprocessor, processor, CPU (Central Processing Unit), etc. It can be recorded on any available medium. The computer readable medium may include program (instruction) codes, data files, data structures, etc. alone or in combination.

The program (command) code recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs, DVDs, and Blu-rays, and Read Only Memory (ROMs). ), a RAM (Random Access Memory), flash memory, etc., may include a semiconductor storage device specially configured to store and execute program (command) codes.

Here, examples of the program (command) code include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine code generated by a compiler. The hardware devices described above may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

10: operator terminal
20: control system
100: complex monitoring device
101: anomaly detector
110: image collection unit
120: sound source collection unit
121: sound collector 122: microphone 123: direction detection module
130: analysis unit
131: video analysis module 132: sound source analysis module 133: complex analysis module
140: storage sending unit
310l collection point
510: video receiving module
520: image decoding module
530: image conversion module

Claims (16)

an anomaly detector 101 that collects image data and sound source data;
an analyzer 130 that determines an abnormal situation by performing complex analysis of the image data and the sound source data and generates abnormal situation information; and
A storage and transmission unit 140 for transmitting the abnormal situation information; includes,
The anomaly detector 101,
an image collection unit 110 generating the image data; and
A sound source collection unit 120 generating the sound source data;
The sound source collection unit 120,
Two microphones 122 for collecting sound sources and generating sound source data; and
A direction detection module 123 for detecting the direction of the sound source and generating sound source direction information;
The sound source collection unit 120,
a body 210 in which the direction detection module 123 is configured; and
It includes; a sound collecting plate 121 integrally disposed on the left and right sides of the body 210 to collect sound sources and having two microphones 122 disposed at the edges,
It is composed of an integral type including the abnormal detector 101, the analyzer 130, and the storage and transmission unit 140,
Analysis of the video data and the sound source data is performed simultaneously,
The sound source direction information is estimated by calculating the arrival time difference of the sound signals received from each of the two microphones and using the arrival time difference,
The analysis unit 130,
an image analysis module 131 for recognizing an object by analyzing the image data, detecting an abnormal behavior based on information of the object, and generating abnormal behavior information;
a sound source analysis module 132 recognizing an abnormal sound source from the sound source data and generating sound source direction information of the abnormal sound source; and
A complex analysis module 133 generating the surrounding abnormal situation information according to a predefined event processing rule using the abnormal behavior information and the sound source direction information;
The sound source data is classified into abnormal sounds including screams, explosions, crashes, and vehicle sudden stops, and abnormal sounds having specific words for rescue request.
delete delete delete According to claim 1,
The complex monitoring device for determining an abnormal sound source, characterized in that the sound collecting plate 121 is in the form of a paraboloid to enhance the sound collecting effect of the sound source.
According to claim 1,
The two microphones 122 are installed in a direction facing the reflected sound source at the sound collecting point 310 where the sound source reflected by the sound collecting plate 121 is focused.
delete According to claim 1,
The image analysis module 131,
a preprocessing unit 610 for preprocessing the image data to generate a standardized standard image frame;
an image analysis unit 620 for detecting and classifying the object by applying a neural network classification model to the standard image frame; and
An abnormal behavior detection unit (630) for detecting the abnormal behavior using the recognized object and generating the abnormal behavior information;
According to claim 8,
A complex monitoring device for determining an abnormal sound source, characterized in that an ensemble of classifiers learned according to the field situation is analyzed to increase the reliability of the detected object.
delete According to claim 1,
characterized in that the abnormal situation information indicating a first-level alarm is generated when the first event caused by the occurrence of the abnormal behavior information or the sound source direction information corresponds to a priority condition event predefined according to the event processing rule. A complex monitoring device that judges
According to claim 11,
When the second event exceeds the preset first weight, the abnormal situation information indicating the second-stage alarm is generated, and the first weight is a value given to an event according to sound analysis or voice analysis. A complex monitoring device that judges.
According to claim 12,
When the first event does not correspond to the priority condition event and a video event due to the occurrence of the abnormal behavior information occurs, the sum of the first weight and the second weight exceeds a preset first set value. The complex monitoring device for determining an abnormal sound source, characterized in that the abnormal situation information indicating the second-stage alarm is generated, and the second weight is a value given to an event according to video analysis.
According to claim 13,
If the first event does not correspond to the priority condition event and a video event occurs due to the occurrence of the abnormal behavior information, if the second weight exceeds a preset second set value, the abnormality indicating the second-stage alarm A complex monitoring device for determining an abnormal sound source, characterized in that situation information is generated.
According to claim 13,
If the first event does not correspond to the priority condition event and no video event occurs due to the occurrence of the abnormal behavior information, and the first weight exceeds a preset third set value, the first-stage alarm is displayed. The composite monitoring device for determining an abnormal sound source, characterized in that when the abnormal situation information is generated and the first weight does not exceed a preset third set value, a search for another device is requested to an operator terminal.
(a) collecting image data and sound source data by the anomaly detector 101;
(b) generating abnormal situation information by determining an abnormal situation through complex analysis of the video data and the sound source data by the analysis unit 130; and
(c) transmitting, by the storage and transmission unit 140, the abnormal situation information;
The anomaly detector 101,
an image collection unit 110 generating the image data; and
A sound source collection unit 120 generating the sound source data;
The sound source collection unit 120,
Two microphones 122 for collecting sound sources and generating sound source data; and
A direction detection module 123 for detecting the direction of the sound source and generating sound source direction information;
The sound source collection unit 120,
a body 210 in which the direction detection module 123 is configured; and
It includes; a sound collecting plate 121 integrally disposed on the left and right sides of the body 210 to collect sound sources and having two microphones 122 disposed at the edges,
It is composed of an integral type including the abnormal detector 101, the analyzer 130, and the storage and transmission unit 140,
Analysis of the video data and the sound source data is performed simultaneously,
The sound source direction information is estimated by calculating the arrival time difference of the sound signals received from each of the two microphones and using the arrival time difference,
The analysis unit 130,
an image analysis module 131 for recognizing an object by analyzing the image data, detecting an abnormal behavior based on information of the object, and generating abnormal behavior information;
a sound source analysis module 132 recognizing an abnormal sound source from the sound source data and generating sound source direction information of the abnormal sound source; and
A complex analysis module 133 generating the surrounding abnormal situation information according to a predefined event processing rule using the abnormal behavior information and the sound source direction information;
The sound source data is classified into an abnormal sound including a scream, an explosion sound, a crash sound, and a vehicle sudden stop, and an abnormal sound having a specific word for a rescue request.

Images