KR102374144B1 - Abnormaly sound recognizing method and apparatus based on artificial intelligence and monitoring system using the same - Google Patents

Abstract

The present invention relates to an artificial intelligence-based abnormal sound source recognition apparatus, a method therefor, and a control system using the same. a model generation unit for generating a model for an abnormal sound source for classifying an abnormal sound source using the machine learning result; a sound source section extractor for time-dividing and extracting a sound source section to recognize a sound source signal inputted from the outside with a time difference; a sound source feature extraction unit for extracting sound source features of the sound source section; an abnormal sound source classification unit for classifying the sound source characteristics according to a predetermined abnormal sound source type using the abnormal sound source model; and an abnormal sound source recognition unit for notifying an abnormal sound source recognition result of the sound source signal according to the time difference classification result of the sound source characteristics.

Description

AI-based abnormal sound source recognition device, method and control system using the same

The present invention relates to an apparatus for recognizing an abnormal sound source based on artificial intelligence, a method thereof, and a control system using the same, and more particularly, time-division and extraction of a sound source section for an arbitrary sound source signal, and then, based on artificial intelligence, an abnormal sound source section To accurately recognize abnormal sound sources by extracting and classifying sound sources, artificial intelligence-based abnormal sound source recognition device, method, and control system using the same.

An artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns, judges, and becomes smarter by itself.

As these artificial intelligence systems are used, the recognition rate improves and user preferences can be understood more accurately, and the existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

Artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning. Machine learning is an algorithm technology that categorizes/learns characteristics of input data by itself, and element technology is a technology that uses machine learning algorithms such as deep learning, such as language understanding, visual understanding, reasoning/prediction, knowledge expression, motion control, etc It consists of the technical fields of

Recently, with the development of artificial intelligence technology, various technologies are pouring out. In particular, in the field related to the recognition of audio data such as voice and sound, studies for recognizing abnormal situations from audio data are being actively conducted. This can be a way to more effectively take action according to the situation by detecting an abnormal sound source in the audio data and notifying the supervisor, beyond the limit of the method of recognizing an abnormal situation through image data captured by a surveillance camera.

Meanwhile, in the past, abnormal sound sources were classified using SVM (Support Vector Machine). In this case, there is a limitation in that there is a high possibility of misclassification of new audio data because a decision interface can be created including an unobserved region.

Moreover, since the amount of training data is limited to classify an abnormal sound source, it is necessary to prepare a method for generating a model for quickly and accurately classifying an abnormal sound source, and analyze the unnecessary section by constructing a model considering the acoustic characteristics of the abnormal sound source As a result, it is necessary to reduce the possibility of misclassification that may occur.

Therefore, in order to classify the abnormal sound source, it is necessary to implement a method for accurately recognizing the abnormal sound source by extracting and classifying the abnormal sound source based on artificial intelligence.

Korean Patent Publication No. 10-1718073 (registered on Mar. 14, 2017)

An object of the present invention is to accurately recognize an abnormal sound source by extracting and extracting the sound source section by time division for an arbitrary sound source signal and then extracting and classifying the abnormal sound source for the sound source section based on artificial intelligence, an artificial intelligence-based abnormal sound source recognition device , to provide a method and a control system using the same.

The artificial intelligence-based abnormal sound source recognition apparatus according to an embodiment of the present invention generates a model for generating a model for an abnormal sound source for classifying an abnormal sound source using the machine learning result performed based on the data amount of an arbitrary sound source data set wealth; a sound source section extractor for time-dividing and extracting a sound source section to recognize a sound source signal inputted from the outside with a time difference; a sound source feature extraction unit for extracting sound source features of the sound source section; an abnormal sound source classification unit for classifying the sound source characteristics according to a predetermined abnormal sound source type using the abnormal sound source model; and an abnormal sound source recognition unit for notifying an abnormal sound source recognition result of the sound source signal according to the time difference classification result of the sound source characteristics.

The sound source section extractor may extract the sound source signal into a first sound source section in which the first highest point is confirmed and a second sound source section in succession to the first sound source section for a predetermined time.

The first sound source section may represent a section length of 1 second left and right, respectively, based on the first highest point of the song, and the second sound source section may represent a section length four times longer than that of the first sound source section.

In the second sound source section, one or more peaks may appear.

The abnormal sound source recognition unit notifies the first abnormal sound source recognition result for the sound source signal through the first sound source section, and then informs the second abnormal sound source recognition result for the sound source signal through the second sound source section can

The abnormal sound source recognition unit, when recognizing a plurality of abnormal sound sources from the first abnormal sound source recognition result and the second abnormal sound source recognition result, synthesizes the plurality of abnormal sound source recognition results to inform event information according to the abnormal sound source scenario may be

The abnormal sound source scenario may indicate occurrence of a specific event corresponding to the order of occurrence of individual abnormal sound sources when a plurality of abnormal sound sources are recognized.

The model for the abnormal sound source may be a model based on a convolutional neural network.

The sound source feature extraction unit obtains the sound source signal as a spectrum representing a frequency domain, and then applies a filter bank based on a mel scale to the spectrum to derive a mel spectrum And, it may be to use a log mel spectrogram-based extraction technique that extracts a sound source feature based on a log mel spectrogram through cepstrum analysis of the mel spectrum.

In addition, according to an embodiment of the present invention, (a) generating a model for an abnormal sound source for classifying an abnormal sound source by using the machine learning result performed based on the data amount of the arbitrary sound source dataset; (b) time-dividing and extracting the sound source section to recognize the sound source signal inputted from the outside with a time difference; (c) extracting a sound source feature of the sound source section; (d) classifying the sound source characteristics into predetermined abnormal sound source types using the abnormal sound source model; and (e) notifying an abnormal sound source recognition result of the sound source signal according to the time difference classification result of the sound source feature.

The step (b) may include extracting the sound source signal into a first sound source section in which the first highest point is confirmed and a second sound source section in succession to the first sound source section for a predetermined time.

In the step (e), the first abnormal sound source recognition result for the sound source signal is informed through the first sound source section, and then the second abnormal sound source recognition result for the sound source signal is informed through the second sound source section it could be

In step (e), when a plurality of abnormal sound sources are recognized from the first abnormal sound source recognition result and the second abnormal sound source recognition result, event information according to the abnormal sound source scenario is synthesized by synthesizing the plurality of abnormal sound source recognition results. may be informing

In the step (c), after obtaining a spectrum representing the sound source signal in a frequency domain, a filter bank based on a mel scale is applied to the spectrum to obtain a mel spectrum It may be to use a log mel spectrogram-based extraction technique for extracting sound source characteristics based on the derivation and log mel spectrogram through cepstrum analysis of the mel spectrum. .

In addition, in the control system according to an embodiment of the present invention, in the control system for monitoring a control area where an event/accident is expected to occur, an abnormal sound source recognition device for recognizing an abnormal sound source from a sound source signal generated in the control area ; and a control server for controlling the control area to be intensively controlled according to the abnormal sound source recognition information transmitted from the abnormal sound source recognition device; a model generator for generating a model for an abnormal sound source for classifying an abnormal sound source using the machine learning result; a sound source section extractor for time-dividing and extracting the sound source section in order to recognize the sound source signal with a time difference; a sound source feature extraction unit for extracting sound source features of the sound source section; an abnormal sound source classification unit for classifying the sound source characteristics according to a predetermined abnormal sound source type using the abnormal sound source model; and an abnormal sound source recognition unit for notifying an abnormal sound source recognition result of the sound source signal according to the time difference classification result of the sound source characteristics.

The abnormal sound source recognition information may include location information of the control area and direction information in which the abnormal sound source is generated.

The direction information may be detected by a sound source sensor installed in the control area.

The control server may control the camera direction of the CCTV installed in the control area according to the direction information.

According to an embodiment, a control terminal for reproducing in real time the captured image of the CCTV controlled by the control server; may further include.

The present invention can accurately recognize an abnormal sound source by extracting and extracting the sound source section by time division for an arbitrary sound source signal, and then extracting and classifying the abnormal sound source for the sound source section based on artificial intelligence.

In addition, the present invention can speed up machine learning and further increase the accuracy of the abnormal sound source classification result even though the data amount of the abnormal sound source dataset is limited.

In addition, the present invention can configure a model for classifying an abnormal sound source by reflecting the acoustic characteristics of the abnormal sound source and the psychoacoustic characteristics that humans recognize the sound.

In addition, the present invention can minimize the blind spot of the CCTV and actively operate the CCTV.

1 is a view showing an artificial intelligence-based abnormal sound source recognition device according to an embodiment of the present invention;
Figure 2 is a view showing the CNN model generator of Figure 1;
3 is a diagram showing an abnormal sound source waveform;
4 is a view for explaining the extraction of the sound source section extending the abnormal sound source of FIG. 3, and FIG. 5 is a comparison result for the case of analyzing the entire sound source section and the case of analyzing the first and second sound source sections by dividing drawing to do,
6 and 7 are views showing an artificial intelligence-based abnormal sound source recognition method according to an embodiment of the present invention;
8 is a diagram illustrating a control system using an artificial intelligence-based abnormal sound source recognition device according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. Also, it should be noted that, throughout the drawings, the same components are denoted by the same reference numerals as much as possible.

The terms or words used in the present specification and claims described below should not be construed as being limited to conventional or dictionary meanings, and the inventor shall appropriately define his/her invention as terms for the best description of his/her invention. Based on the principle that it can be done, it should be interpreted as meaning and concept consistent with the technical idea of the present invention.

Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all of the technical spirit of the present invention. It should be understood that there may be equivalents and variations.

In the accompanying drawings, some components are exaggerated, omitted, or schematically illustrated, and the size of each component does not fully reflect the actual size. The present invention is not limited by the relative size or spacing drawn in the accompanying drawings.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. Also, when a part is said to be “connected” to another part, it includes not only a case in which it is “directly connected” but also a case in which it is “electrically connected” with another element interposed therebetween.

The singular expression includes the plural expression unless the context clearly dictates otherwise. Terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, and includes one or more other features, number, or step. , it should be understood that it does not preclude in advance the possibility of the presence or addition of an operation, component, part, or combination thereof.

Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a diagram showing an artificial intelligence-based abnormal sound source recognition apparatus according to an embodiment of the present invention, FIG. 2 is a diagram showing the CNN model generator of FIG. 1, FIG. 3 is a diagram showing abnormal sound source waveforms, and FIG. 4 is a diagram for explaining the extraction of the sound source section that extends the abnormal sound source of Fig. 3, and Fig. 5 is a diagram for explaining the comparison results for the case of analyzing the entire sound source section and the case of dividing and analyzing the first and second sound source sections It is a drawing.

As shown in Fig. 1, the artificial intelligence-based abnormal sound source recognition device (hereinafter referred to as 'abnormal sound source recognition device', 100) according to an embodiment of the present invention is extracted by time-divisioning a sound source section for an arbitrary sound source signal. Then, it is possible to accurately recognize abnormal sound sources by extracting and classifying abnormal sound sources for sound source sections based on artificial intelligence.

Here, the abnormal sound source is an atypical sound source that is generated in a specific dangerous situation and is not expressed in words, but can intuitively inform a specific dangerous situation, for example, a scream, a broken window sound, a vehicle horn. It can be classified into types such as sound, sudden vehicle stop sound, vehicle accident sound, explosion sound, and gun sound. The acoustic characteristics of such an abnormal sound source are characterized by a large sound volume of a certain decibel (db) or more and a short sound section.

That is, the abnormal sound source is not recognized as a word or sentence having a meaning of an acoustic signal, but represents a specific dangerous situation. Such an abnormal sound source corresponds to a weak label sound source that is labeled only with the presence or absence of a sound source section in a file unit rather than a dataset labeled for the sound source section.

In addition, artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning, and is a deep learning model for specifically classifying abnormal sound sources. A case in which the model is applied will be described.

On the other hand, the abnormal sound source recognition apparatus 100 includes a CNN model generation unit 110, a sound source section extraction unit 120, a sound source feature extraction unit 130, an abnormal sound source classification unit 140, an abnormal sound source recognition unit 150, It may be configured to include a sound source section storage unit 160 .

First, the CNN model generator 110 generates a CNN model for classifying an abnormal sound source.

In this case, the CNN model generator 110 performs machine learning on the abnormal sound source dataset to generate a CNN model because the amount of data for generating the CNN model is limited. do not create

That is, the CNN model generator 110 first generates a CNN model by performing machine learning based on a sufficient amount of training data of an arbitrary sound source dataset in which various types of sound sources are weakly labeled, and then uses this to generate an abnormal sound source. Create a CNN model for classification.

Hereinafter, for convenience of explanation, a CNN model for classifying an abnormal sound source is referred to as a 'CNN model for an abnormal sound source', and a CNN model for classifying an arbitrary sound source is referred to as a 'CNN model for an arbitrary sound source'.

In this way, when the CNN model generating unit 110 uses a pre-made CNN model for an arbitrary sound source to create a CNN model for an abnormal sound source, even if the data amount of the abnormal sound source dataset is limited, the machine learning is fast and the accuracy of the abnormal sound source classification result is improved. can make it higher.

Here, the CNN model for arbitrary sound sources can be used as long as it is made by securing sufficient amount of training data even if the classification accuracy of arbitrary sound sources does not reach the generally required level. This is to create a CNN model for an abnormal sound source by using a CNN model for an arbitrary sound source as a supplementary method because it is difficult for the abnormal sound source dataset to secure sufficient amount of training data for model generation.

Specifically, the CNN model generator 110 generates a CNN model for an arbitrary sound source as follows.

That is, the CNN model generator 110 divides each sound source signal of the arbitrary sound source data set by frame and applies a Short-Time Fourier Transformer (STFT) to convert each sound source signal in the frequency domain instead of the time domain. A spectrum expressed in (frequency domain) is obtained.

And, the CNN model generating unit 110 is a relationship between a physical frequency and a frequency recognized by a real person in order to reflect the characteristics of the human auditory organ appearing more sensitively in a low frequency band than in a high frequency (high frequency) band A mel spectrum is derived by applying a filter bank based on a mel scale representing .

Then, the CNN model generator 110 performs cepstrum analysis through log and inverse Fourier transform on the Mel spectrum obtained by reflecting the characteristics of the human auditory organ.

Through this, the CNN model generation unit 110 extracts the random sound source feature as a log mel spectrogram for each sound source signal of the arbitrary sound source data set (S1). This is a signal processing process that converts a sound source signal into an image form to classify each sound source signal using a CNN model.

In addition, the CNN model generation unit 110 performs CNN-based machine learning on the features of the arbitrary sound source to generate a CNN model for the arbitrary sound source (S2, S3). At this time, the CNN model generator 110 extracts classification features by determining a weight and a bias for calculation in the convolutional layer, and max pooling (max) in the pooling layer. -pooling) can be used to classify by type of abnormal sound source predetermined for classification characteristics.

Meanwhile, the CNN model generating unit 110 generates a CNN model for an abnormal sound source as follows.

In this case, the CNN model generating unit 110 generates a CNN model for an abnormal sound source by performing fine adjustments to machine learning to classify an abnormal sound source with a CNN model for an arbitrary sound source that has already been machine learning completed in advance.

That is, the CNN model generating unit 110 generates a CNN model for an abnormal sound source by finely adjusting the weights and biases for calculation in the convolutional layer of the CNN model for an arbitrary sound source according to the abnormal sound source. In this case, the CNN model generator 110 may perform fine adjustment only on the last fully connected layer in the CNN model for an arbitrary sound source.

Referring to FIG. 2 , the CNN model generating unit 110 converts the abnormal sound source into a log mel spectrogram for each sound source signal of the abnormal sound source data set, as in extracting the random sound source features from the arbitrary sound source dataset. A feature is extracted (S11).

Then, the CNN model generator 110 re-learns the characteristics of an abnormal sound source from the CNN model for an arbitrary sound source and generates a CNN model for an abnormal sound source through fine adjustment to fit the abnormal sound source (S12, S13). That is, the CNN model generating unit 110 extracts classification features for an abnormal sound source by re-tuning the CNN model for an arbitrary sound source into a CNN model for an abnormal sound source, and classifies the abnormal sound source by a predetermined type of abnormal sound source based on the classification characteristic. A CNN model is prepared.

As such, the CNN model generator 110 generates a CNN model for an abnormal sound source for classifying an abnormal sound source by using the machine learning result performed based on the data range of the arbitrary sound source dataset.

Next, when an arbitrary sound source signal is input from the outside, the sound source section extractor 120 extracts a sound source section that reflects the acoustic characteristics of the abnormal sound source and the psychoacoustic characteristics that a person recognizes from the sound source signal.

In this case, the sound source section extractor 120 time-divisions and extracts the sound source signal input from the outside in order to recognize it with a time difference.

Here, as described above, the acoustic characteristics of the abnormal sound source indicate a characteristic of a large sound volume of a certain decibel (db) or more and a short section. That is, the abnormal sound source has a shape that forms the peak audio of the sound source section with reference to FIG. 3 .

Although the psychoacoustic characteristics that a person recognizes may vary depending on the person's reflexes, sensitivity to sound, or whether or not to learn, the length of sound required for a person to recognize a sound is about 2 seconds on average.

Accordingly, the sound source section extractor 120 extracts a predetermined pre/post section from the sound source signal as a sound source section based on the highest point of the sound source section. As shown in FIG. 3 , the sound source section may be extracted as a section of 2 seconds in total, 1 second each before and after each section, based on the highest point in the time domain according to psychoacoustic characteristics.

However, the analysis result may not be 100% accurate for the entire 2 second sound source section. So, the sound source section extractor 120 performs a pre-recognition process for the sound source section of 2 seconds, and then performs a post-recognition process for the sound source section of 8 seconds continuously with a length that can determine the before and after situations of the abnormal sound source. do.

Here, as a sound source section in which the line recognition process is performed as shown in FIG. 4, the section in which the first peak that appears first is identified is hereinafter collectively referred to as the 'first sound source section', and is a sound source section in which a subsequent recognition process is performed, the first sound source A section that is continuously confirmed for a predetermined time in succession to the section is hereinafter collectively referred to as a 'second sound source section'.

The first sound source section may represent a total section length of 2 seconds, which is a section of 1 second left and right based on the first highest point, and the second sound source section can be expressed as a section length of 8 seconds, which is a predetermined time consecutive to the first sound source section. The length of the first sound source section and the second sound source section can be changed according to the characteristics of the sound.

In addition, one peak (first highest point, W1) may appear in the first sound source section, but more than one peak (second highest point W2, third highest point W3) may appear in the second sound source section. In this case, the second sound source section may represent a section length four times that of the first sound source section.

As such, when the second sound source section includes at least one abnormal sound source, a scenario in which a specific event is notified corresponding to the order of occurrence of individual abnormal sound sources when a plurality of abnormal sound sources are recognized (hereinafter, 'abnormal sound source scenario') ) can be shown.

For example, the abnormal sound source scenario is when abnormal sound source 1 is 'horn sound', abnormal sound source 2 is 'vehicle sudden stop', and abnormal sound source 3 is 'vehicle crash sound', abnormal sound source 1 - abnormal sound source 2 - abnormal sound source 3 It may mean a specific event of 'vehicle traffic accident occurrence' corresponding to the order of occurrence. In this way, the abnormal sound source scenario may be previously determined in consideration of various cases and stored in the sound source section storage unit 160 in advance.

Here, the case in which a plurality of abnormal sound sources are recognized corresponds to a case in which a plurality of abnormal sound sources are recognized in the second sound source section and a case in which a plurality of abnormal sound sources are recognized in the first and second sound source sections.

On the other hand, since the abnormal sound source corresponds to a sound source that can intuitively inform a specific dangerous situation as described above, it is important to recognize it quickly and without omission.

However, referring to FIG. 5 , in the case of recognizing the entire sound source section at once, as in the case of recognizing the first and second sound source sections by time division, even though it is possible to recognize abnormal sound sources without omission, the first and second sound source sections are It may be difficult to quickly recognize an abnormal sound source compared to the case of time division recognition.

That is, in the case of recognizing the entire sound source section at once, after extracting all the sound source sections, recognition of the entire sound source section may be performed to inform the occurrence of an abnormal sound source.

On the other hand, in the case of time-division recognition of the first and second sound source sections, a line recognition process is performed for the first sound source section immediately after the first sound source section is extracted, thereby generating an abnormal sound source for the sound source signal input from the outside It is possible not only to quickly inform the sound source, but also to perform a post-recognition process for the second sound source section, which is shorter than the entire sound source section separately from the first sound source section, to accurately inform the second generation of abnormal sound sources with respect to the sound source signal. Of course, when the post-recognition process is performed for the second sound source section, the recognition process for the first sound source section is excluded, so that the recognition time is shortened.

In the case of recognizing the entire sound source section at once in recognizing an abnormal sound source for an arbitrary sound source signal on the time axis as shown in FIG. 5, and when recognizing the first and second sound source segments by time division, the recognition speed of the abnormal sound source is different from each other. can be easily understood Such a difference may appear more clearly as the length of the first and second sound source sections increases.

Referring back to FIG. 1 , the sound source section extractor 120 extracts the first sound source section and the second sound source section, and transfers the first sound source section to the sound source feature extractor 130 to perform a line recognition process for the first sound source section and (①), and store it in the sound source section storage unit 160 to perform a post-recognition process for the second sound source section (ⓐ).

That is, the sound source section extractor 120 extracts the first sound source section and delivers it to the sound source feature extractor 130, while extracting and recording the second sound source section by a predetermined section length to the sound source section storage unit 160 Save.

Next, the sound source feature extraction unit 130 extracts the sound source characteristics based on the log Mel spectrogram for the first sound source section of the arbitrary sound source signal extracted by the sound source section extraction unit 120 (②). A detailed description thereof will be omitted since it is the same as described above.

Similarly, the sound source feature extraction unit 130 extracts the sound source characteristics based on the log Mel spectrogram for the second sound source section stored in the sound source section storage unit 160 (ⓑ, ⓓ). At this time, the sound source feature extraction unit 130 receives the second sound source section from the sound source section storage unit 160 when there is a request between the abnormal sound source classification units 140 (ⓒ).

Next, the abnormal sound source classification unit 140 is based on the CNN model for the abnormal sound source generated by the CNN model generation unit 110 to the sound source characteristics of the first sound source section extracted by the sound source feature extraction unit 130. Classification features are extracted for the data, and the classification features are classified according to a predetermined abnormal sound source type (③).

Similarly, the abnormal sound source classifying unit 140 is based on the CNN model for the abnormal sound source generated by the CNN model generating unit 110. For the sound source characteristics of the second sound source section extracted by the sound source feature extraction unit 130, Classification features are extracted, and the classification features are classified according to a predetermined abnormal sound source type (ⓔ).

In this way, the abnormal sound source classification unit 140 extracts the classification features for the sound source characteristics of the first sound source section or the second sound source section in the convolutional layer of the CNN model for the abnormal sound source, and simplifies the feature points for classification in the pooling layer. (sub-sampling), and classifies by type of abnormal sound source predetermined for classification characteristics through the final fully connected layer.

Next, the abnormal sound source recognition unit 150 notifies the abnormal sound source recognition result in an arbitrary sound source signal using the classification results for each of the first sound source section and the second sound source section.

Specifically, when one abnormal sound source is recognized in the first sound source section and the second sound source section, the abnormal sound source recognition unit 150 notifies the abnormal sound source classification result as one abnormal sound source recognition result.

In addition, when recognizing a plurality of abnormal sound sources in the first sound source section and the second sound source section, the abnormal sound source recognition unit 150 notifies the abnormal sound source classification result individually for each of the plurality of abnormal sound source recognition results, or By synthesizing the sound source recognition results of

Here, the abnormal sound source recognition unit 150 recognizes an abnormal sound source in the second sound source section or in the first and second sound source sections, except for the case of recognizing the abnormal sound source only in the first sound source section. In some cases, event information according to an abnormal sound source scenario may be notified.

On the other hand, the abnormal sound source recognition apparatus 100 may further include an interface unit (not shown) for receiving an arbitrary sound source signal and outputting an abnormal sound source recognition result according to a classification result for the arbitrary sound source signal. For example, the interface unit includes a microphone to receive an arbitrary sound source signal, and a display device or speaker ( speaker), etc. may be provided.

In addition, the abnormal sound source recognition apparatus 100 includes at least one processor and a memory for storing computer-readable instructions.

At least one processor performs the artificial intelligence-based abnormal sound source recognition method according to an embodiment of the present invention when executing computer readable instructions stored in the memory.

That is, when the processor executes computer readable instructions stored in the memory, the CNN model generator 110, the sound source section extractor 120, the sound source feature extractor 130, It is possible to perform the functions of the sound source classification unit 140 and the abnormal sound source recognition unit 150 .

Here, the processor is at least one or more processors, and may also be referred to as a controller, a microcontroller, a microprocessor, a microcomputer, or the like.

In addition, the processor may be implemented by hardware or firmware, software, or a combination thereof.

Also, a memory may be a single storage device, or may be a collective term for a plurality of storage elements. The computer readable instructions stored in the memory may be executable program code or parameters, data, and the like.

In addition, the memory may include random access memory (RAM), or may include non-volatile memory (NVRAM) such as magnetic disk storage or flash memory. Here, the memory may perform the function of the sound source section storage unit 160 .

6 and 7 are diagrams illustrating an artificial intelligence-based abnormal sound source recognition method according to an embodiment of the present invention.

As shown in FIG. 6 , the abnormal sound source recognition apparatus 100 time-divisions and extracts the first sound source section and the second sound source section in order to recognize the sound source signal inputted from the outside with a time difference.

First, the abnormal sound source recognition apparatus 100 extracts a first sound source section from the sound source signal (S201).

Then, the abnormal sound source recognition apparatus 100 extracts the sound source characteristics of the first sound source section, and classifies the sound source characteristics according to a predetermined abnormal sound source type using the abnormal sound source model (S202).

Then, the abnormal sound source recognizing apparatus 100 checks whether one abnormal sound source is recognized in the first sound source section (S203).

In this case, when the abnormal sound source recognition apparatus 100 does not recognize one abnormal sound source in the first sound source section (S203), and extracts the second sound source section from the sound source signal (S203-1), the sound source of the second sound source section The features are extracted, and the sound source features are classified according to a predetermined abnormal sound source type using the abnormal sound source model (S204).

Thereafter, the abnormal sound source recognizing apparatus 100 checks whether one or more abnormal sound sources are recognized in the second sound source section (S205).

At this time, when recognizing a plurality of abnormal sound sources, the apparatus 100 for recognizing an abnormal sound source synthesizes the plurality of abnormal sound sources and confirms whether the abnormal sound source scenario corresponds to the abnormal sound source scenario (S206). That is, the abnormal sound source recognizing apparatus 100 may inquire and confirm the abnormal sound source scenario corresponding to the generation order of the individual abnormal sound sources constituting the plurality of abnormal sound sources. In step S206, only the abnormal sound source recognition result in the second sound source section is considered to determine whether the abnormal sound source scenario is applicable.

Here, if the abnormal sound source recognition apparatus 100 corresponds to the abnormal sound source scenario (S206), it notifies specific event information according to the abnormal sound source scenario (S207). In this case, the abnormal sound source recognition apparatus 100 recognizes a plurality of abnormal sound sources in the second sound source section.

Otherwise, the abnormal sound source recognition apparatus 100 notifies the individual abnormal sound source recognition result (S208). At this time, the abnormal sound source recognition apparatus 100 notifies the individual abnormal sound source recognition result if it does not correspond to the abnormal sound source scenario even if it recognizes one abnormal sound source in the second sound source section or recognizes a plurality of abnormal sound sources in the second sound source section.

Meanwhile, in step S203 of FIG. 6 , when the abnormal sound source recognizing apparatus 100 recognizes one abnormal sound source in the first sound source section ( S203 ), as shown in FIG. 7 , a subsequent step is performed.

6 shows a case of recognizing an abnormal sound source in the second sound source section, FIG. 7 shows a case of recognizing an abnormal sound source in the first sound source section first, and then recognizing an abnormal sound source in the first and second sound source sections indicates the case.

Referring to FIG. 7 , when the abnormal sound source recognition apparatus 100 recognizes one abnormal sound source in the first sound source section (S251), it notifies the result of the one abnormal sound source recognition (S252). In this way, the apparatus for recognizing an abnormal sound source 100 may perform line analysis on the first sound source section to promptly notify the occurrence of an abnormal sound source for the first sound source section.

At the same time, when the abnormal sound source recognition apparatus 100 extracts the second sound source section from the sound source signal (S252), the sound source feature of the second sound source section is extracted, and the sound source characteristic is determined in advance by using the abnormal sound source model. Classify by type (S253).

Thereafter, the abnormal sound source recognizing apparatus 100 checks whether one or more abnormal sound sources are recognized in the second sound source section (S254).

At this time, when the abnormal sound source recognition apparatus 100 recognizes one or more abnormal sound sources in the second sound source section (S254), it checks the abnormal sound source recognition result in the first sound source section (S255), and at least one or more abnormal sound sources in the second sound source section By synthesizing the sound source recognition results, it is checked whether it corresponds to an abnormal sound source scenario (S256). In step S256, it is checked whether the abnormal sound source scenario corresponds to the result of recognition of one abnormal sound source in the first sound source section and the recognition result of one or more abnormal sound sources in the second sound source section.

Here, if the abnormal sound source recognition apparatus 100 corresponds to the abnormal sound source scenario (S256), it notifies specific event information according to the abnormal sound source scenario (S257). In this case, the abnormal sound source recognition apparatus 100 informs of specific event information according to the abnormal sound source scenario constituted based on the recognition result of one abnormal sound source in the first sound source section and the recognition result of one or more abnormal sound sources in the second sound source section.

Otherwise, the abnormal sound source recognition apparatus 100 notifies the individual abnormal sound source recognition result (S258). At this time, the abnormal sound source recognition apparatus 100 notifies the individual abnormal sound source recognition result if it does not correspond to the abnormal sound source scenario even if it recognizes one abnormal sound source in the second sound source section or recognizes a plurality of abnormal sound sources in the second sound source section. In step S258, the abnormal sound source recognition result is not notified again in the first sound source section.

8 is a diagram illustrating a control system using an artificial intelligence-based abnormal sound source recognition device according to an embodiment of the present invention.

Referring to FIG. 8 , the control system includes an abnormal sound source recognition device 100 , a control server 200 , and a control terminal 300 .

This control system monitors a number of control areas (ie, the 1st control area to the Nth control area) where the possibility of an event/accident is expected. In addition, CCTVs (10-1 to 10-N) and sound source detectors (20-1 to 20-N) are installed in each of the plurality of control areas. At this time, the sound source detectors 20-1 to 20-N are capable of detecting the direction in which the sound source is generated, and a detailed description thereof will be omitted because it is easily understood by those skilled in the art.

On the other hand, the abnormal sound source recognition apparatus 100 is connected to the sound source detectors 20-1 to 20-N, receives the sound source signal collected by the sound source detectors 20-1 to 20-N, and extracts the abnormal sound source and classifying to recognize abnormal sound sources. A detailed description thereof will be omitted because it can be understood through the description of FIGS. 1 to 3 described above.

At this time, the abnormal sound source recognition apparatus 100 transmits the abnormal sound source classification result of a specific control area to the control server 200 . At this time, the abnormal sound source recognition device 100 notifies the abnormal sound source recognition result of the specific control area, and the location information of the specific control area and the sound source detected by the sound source detectors 20-1 to 20-N are generated. The direction information is transmitted together to the control server 200 .

Then, the control server 200 uses the location information and sound source direction information of the specific control area transmitted from the abnormal sound source recognition device 100 to focus the camera direction of the CCTV installed in the specific control area toward the location where the abnormal sound source is generated. It can be controlled to be controlled. At this time, the CCTV is PTZ (Pan/Tilt/Zoom) controlled.

In addition, the control terminal 300 reproduces the captured video of the CCTV controlled by the control server 200 in real time so that the administrator can check it.

In this way, the control system can minimize the blind spot of the CCTV and actively operate the CCTV.

The method according to some embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CDROMs and DVDs, and magneto-optical disks such as floppy disks. hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the foregoing description has focused on novel features of the invention as applied to various embodiments, those skilled in the art will recognize the apparatus and method described above without departing from the scope of the invention. It will be understood that various deletions, substitutions, and changes are possible in the form and details of Accordingly, the scope of the invention is defined by the appended claims rather than by the foregoing description. All modifications within the scope of equivalents of the claims are included in the scope of the present invention.

10-1 to 10-N; CCTV 20-1 to 20-N ; sound detector
110 ; CNN model generator 120 ; sound source section
130 ; sound source feature extraction unit 140 ; Abnormal sound classification section
150 ; Abnormal sound source recognition unit 160 ; music section storage
200 ; control server 300 ; control terminal

Claims (27)

a model generator for generating a model for an abnormal sound source for classifying an abnormal sound source by using the machine learning result performed based on the data amount of the arbitrary sound source dataset;
a sound source section extractor for extracting the sound source section by time division in order to recognize the sound source signal inputted from the outside with a time difference;
a sound source feature extraction unit for extracting sound source features of the sound source section;
an abnormal sound source classification unit for classifying the sound source characteristics according to a predetermined abnormal sound source type using the abnormal sound source model; and
an abnormal sound source recognition unit for notifying an abnormal sound source recognition result of the sound source signal according to the time difference classification result of the sound source characteristics;
The sound source section extraction unit,
Extracting the sound source signal into a first sound source section in which the first highest point is confirmed, and a second sound source section in succession to the first sound source section for a predetermined time period,
The first sound source section,
It represents the length of the section of 1 second on each left and right based on the first highest point of the sing-gi,
The second sound source section,
Represents a section length four times that of the first sound source section,
The second sound source section is an artificial intelligence-based abnormal sound source recognition device in which one or more peaks appear.
delete delete delete The method of claim 1,
The abnormal sound source recognition unit,
An artificial intelligence-based abnormality that informs the first abnormal sound source recognition result for the sound source signal through the first sound source section, and then informs the second abnormal sound source recognition result for the sound source signal through the second sound source section sound recognition device.
6. The method of claim 5,
The abnormal sound source recognition unit,
When a plurality of abnormal sound sources are recognized from the first abnormal sound source recognition result and the second abnormal sound source recognition result, it is based on artificial intelligence that informs event information according to the abnormal sound source scenario by synthesizing the plurality of abnormal sound source recognition results of abnormal sound source recognition device.
7. The method of claim 6,
The above abnormal sound source scenario is,
An artificial intelligence-based abnormal sound source recognition device that notifies the occurrence of a specific event in response to the order of occurrence of individual abnormal sound sources when a plurality of abnormal sound sources are recognized.
The method of claim 1,
The model for the above abnormal sound source,
An artificial intelligence-based abnormal sound source recognition device that is a convolutional neural network-based model.
The method of claim 1,
The sound source feature extraction unit,
After obtaining the sound source signal as a spectrum representing the frequency domain, a filter bank based on a mel scale is applied to the spectrum to derive a mel spectrum, and the mel spectrum is An artificial intelligence-based abnormal sound source recognition device that uses a log mel spectrogram-based extraction technique that extracts log mel spectrogram-based sound source features through cepstrum analysis .
(a) generating a model for an abnormal sound source for classifying an abnormal sound source by using the machine learning result performed based on the data amount of the arbitrary sound source dataset;
(b) time-dividing and extracting the sound source section to recognize the sound source signal inputted from the outside with a time difference;
(c) extracting a sound source feature of the sound source section;
(d) classifying the sound source characteristics into predetermined abnormal sound source types using the abnormal sound source model; and
(e) notifying an abnormal sound source recognition result of the sound source signal according to the time difference classification result of the sound source feature;
includes,
The step (b) is,
Extracting the sound source signal into a first sound source section in which the first highest point is confirmed and a second sound source section in succession to the first sound source section for a predetermined time period,
The first sound source section,
It represents the length of the section of 1 second on each left and right, based on the first highest point,
The second sound source section,
Represents a section length four times that of the first sound source section,
The second sound source section is an artificial intelligence-based abnormal sound source recognition method in which one or more peaks appear.
delete delete delete 11. The method of claim 10,
Step (e) is,
An artificial intelligence-based abnormality that informs the first abnormal sound source recognition result for the sound source signal through the first sound source section, and then informs the second abnormal sound source recognition result for the sound source signal through the second sound source section How to recognize a sound source.
15. The method of claim 14,
Step (e) is,
When a plurality of abnormal sound sources are recognized from the first abnormal sound source recognition result and the second abnormal sound source recognition result, it is based on artificial intelligence that informs event information according to the abnormal sound source scenario by synthesizing the plurality of abnormal sound source recognition results method of recognizing abnormal sound sources.
16. The method of claim 15,
The abnormal sound source scenario is,
An artificial intelligence-based abnormal sound source recognition method that notifies the occurrence of a specific event in response to the order of occurrence of individual abnormal sound sources when a plurality of abnormal sound sources are recognized.
11. The method of claim 10,
The model for the above abnormal sound source,
An artificial intelligence-based abnormal sound source recognition method, a convolutional neural network-based model.
11. The method of claim 10,
Step (c) is,
After obtaining the sound source signal as a spectrum representing the frequency domain, a filter bank based on a mel scale is applied to the spectrum to derive a mel spectrum, and the mel spectrum is An artificial intelligence-based abnormal sound source recognition method using a log mel spectrogram-based extraction technique that extracts the log mel spectrogram-based sound source characteristics through cepstrum analysis .
delete delete delete delete delete delete delete delete delete

Images