KR102292479B1 - Violent section detection apparatus and method for voice signals using vector quantization information - Google Patents

Abstract

Provided are an apparatus for detecting a violent section from voice signals using vector quantization information, which accurately detect a violent section, and a method thereof. According to the present invention, the apparatus comprises: a section generation unit sequentially performing a set of operations, which generates a plurality of sections by dividing m-second voice signal for determining violent language into units of time, n (n is a positive number greater than or equal to one) times; a vector quantization wave-to-vector (VQ-Wav2Vec) processing unit sequentially performing a set of operations, which inputs the generated voice signal for each section into a VQ-Wav2Vec model to vector quantization information for each section, n times; and a violence determination unit performing artificial intelligence (AI) learning through the vector quantization information for each section input from the VQ-Wav2Vec processing unit to output information on whether or not violent language exists.

Description

TECHNICAL FIELD [0002] Violent section detection apparatus and method for voice signals using vector quantization information}

The present invention relates to an apparatus and method for detecting a section of violence in a voice signal using vector quantization information, and more particularly, to a voice signal using vector quantization information that can detect a section of violence using vector quantization information for a voice signal. It relates to an apparatus and method for detecting a violent section of

Recently, in the case of language modeling in the field of natural language processing, a method using a pre-trained language model is used. The most representative examples are Embeddings from Language Model (ELMo), OpenAI Generative Pre-training (GPT), and Bidirectional Encoder Representations from Transformers (BERT). In particular, BERT achieved SOTA (State Of The Art) in all natural language processing tasks by utilizing MLM (Masked Language Model) and Next Sentence Prediction techniques.

In addition, recently, a wav2vec model that converts a speech signal, not text information, into a vector representation for speech recognition was developed by FAIR (Facebook AI Research), and the wav2vec model shows high performance in speech recognition rate.

Meanwhile, in the past, a log mel-spectrogram or MFCC (Mel Frequency Cepstral Coefficient) algorithm is mainly used to analyze a voice signal. A voice signal contains various information such as phoneme/language, meaning, tone, speaker's characteristics, and emotions, but the results analyzed by existing algorithms do not sufficiently include various information of these voice signals, and therefore, the voice It is less accurate in determining whether it is violent or non-violent language.

Domestic Registered Patent No. 10-1779361 (Registered on Sept. 12, 2017)

The technical object of the present invention to solve the above problems is to provide an apparatus and method for detecting a violence section of a voice signal using vector quantization information capable of discriminating a section of violence included in the voice signal.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above technical problem, according to an embodiment of the present invention, an apparatus for detecting a violent section of a voice signal using vector quantization information divides an m-second voice signal for determining a violent language into a plurality of time units. a section generator that sequentially performs one set operation for generating sections of n (n is a positive number equal to or greater than 1) times; a VQ-Wav2Vec processing unit that sequentially performs a set operation of outputting vector quantization information for each section by inputting the generated speech signal for each section to a VQ-Wav2Vec (Vector Quantization Wave to Vector) model n times; and a violence determining unit configured to perform artificial intelligence learning of vector quantization information for each section input from the VQ-Wav2Vec processing unit through a deep learning model, and output determination information on whether or not violent language exists for each section.

A time unit applied each time the section generator performs a set operation of dividing the voice signal into time units n times is different from each other.

The time unit applied in the one set operation may be changed to one of equal and non-uniform.

The violence determination unit may include: a learning unit that artificially learns Vector Quantization information for each section input from the VQ-Wav2Vec processing unit and outputs temporary determination information on whether or not violent language exists for each section; and a cumulative determination unit that finally determines whether or not violence against the voice signal is made by accumulating the provisional determination information for each of the n times when the provisional determination information for each section n times is input by the learning unit.

On the other hand, according to another embodiment of the present invention, in the method of detecting a violent section of a voice signal using vector quantization information, (A) the electronic device divides the m-second voice signal for determining the violent language into units of time, sequentially performing one set operation of generating sections n times (n is a positive number greater than or equal to 1); (B) One set operation in which the electronic device inputs the speech signal for each section generated in step (A) into the VQ-Wav2Vec (Vector Quantization Wave to Vector) model and outputs Vector Quantization information for each section sequentially performing n times; and (C), by the electronic device, AI-learning the vector quantization information for each section input from the step (B) through a deep learning model and outputting discrimination information on whether or not violent language is present for each section. .

In step (A), a time unit applied every time one set operation of dividing the voice signal into time units is performed n times is set differently.

The time unit applied in the one set operation may be changed to one of equal and non-uniform.

The step (C) includes the steps of: (C1) performing artificial intelligence learning of the vector quantization information for each section input from the step (B) and outputting temporary identification information on whether or not violent language is present for each section; and (C2) arranging the temporary determination information for each section n times input from the step (C1) so that the time axis coincides with each other; and (C3) accumulating the provisional identification information for each section n times arranged to coincide with the time axis in step (C2) at the same time to finally determine whether violence against the voice signal is present.

According to the present invention, the violence section in the voice signal can be detected more accurately than before using the VQ-wav2vec model that discretely represents embedding information using the Vector Quantization module.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating an apparatus for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention;
2 is a diagram showing the architecture of the VQ-Wav2Vec model;
3 is a view showing a case in which the section generator divides the discrimination target voice signal into different time units and divides it into a plurality of sections;
4 is a view showing a temporary determination result of the learning unit;
5 is a view showing the determination result of the cumulative determination unit;
6 is a flowchart schematically illustrating a method for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention;
7 is a flowchart showing in detail step S630 of FIG. 6, and,
8 is a block diagram illustrating a computing system for executing a method for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention.

Before describing the specific content for carrying out the present invention, the terms or words used in the specification and claims may properly define the concept of the term in order for the inventor to best describe his or her invention. Based on the principle that there is, it should be interpreted as meaning and concept consistent with the technical matters of the present invention.

In addition, when it is determined that the detailed description of the well-known functions related to the present invention and its configuration may unnecessarily obscure the gist of the present invention, it should be noted that the detailed description is omitted.

When it is stated that any element, component, device, or system includes a component consisting of a program or software, even if not explicitly stated, that element, component, device, or system means that the program or software executes or operates It should be understood to include hardware (eg, memory, CPU, etc.) or other programs or software (eg, drivers required to run an operating system or hardware) necessary for the operation.

In addition, it should be understood that, unless specifically stated in the implementation of an element (or component), the element (or component) may be implemented in software, hardware, or any form of software and hardware.

Hereinafter, specific technical contents to be practiced in the present invention will be described in detail with reference to the accompanying drawings.

The present invention indicates that each configuration of the devices shown in FIG. 1 can be functionally and/or logically separated, and does not necessarily mean that each configuration is divided into a separate physical device or generated with a separate code. An average expert in the field of technology can easily infer.

1 is a block diagram illustrating an apparatus 100 for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention.

The apparatus 100 for detecting a violent section of a voice signal using Vector Quantization information shown in FIG. 1 is a device for determining whether violent language is included by analyzing a voice signal, and is a VQ-Wav2Vec (Vector Quantization Wave to Vector) model. And, deep learning models or machine learning methods can be used.

Referring to FIG. 1 , the apparatus 100 for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention includes a section generator 110 and a Vector Quantization Wave to Vector (VQ-Wav2Vec) processor 120 . and a violence determination unit 130 .

The section generating unit 110 may sequentially perform one set operation of generating a plurality of sections by dividing the same m-second voice signal for determining violent language in units of time n (n is a positive number equal to or greater than 1) times.

The section generator 110 may set different time units applied every n times of one set operation of dividing an m-second voice signal into time units. Also, the section generator 110 may change the time unit applied in one set operation to either an equal section unit or an unequal section unit.

The reason for dividing the discrimination target voice signal into a plurality of sections is to accurately discriminate a specific section corresponding to a violent language among the voice signals.

Hereinafter, a voice signal for discriminating a violent language is referred to as a 'discrimination target voice signal'.

In detail, the section generating unit 110 receives an input voice signal to be determined having a length of m seconds, divides it into equal time units or unequal time units, and generates a plurality of sections corresponding to the first set operation. can be done

In addition, the section generating unit 110 divides the same discrimination target voice signal by time unit, but divides the same time unit into a time unit different from the time unit applied in the above set to generate a plurality of sections corresponding to the second set operation. can be done That is, the section generator 110 may generate a plurality of sections by applying different time units to every n sets.

For example, when a time unit equal to n=2 is set, the section generating unit 110 divides the discrimination target voice signal for 20 seconds into 2 second units in the first set to generate 10 sections, 2 In the first set, 5 sections can be created by dividing by 4 seconds. Here, it is obvious that 2 seconds and 4 seconds are examples and are not limited thereto.

Alternatively, when a time unit unequal to n=2 is set, the section generating unit 110 generates a discrimination target voice signal for 20 seconds in the first set for 2 seconds, 4 seconds, 2 seconds, 4 seconds, 2 seconds, 7 sections are generated by dividing by 4 seconds and 2 seconds, and in the second set, 6 sections can be generated by dividing by 5 seconds, 3 seconds, 2 seconds, 4 seconds, 4 seconds, and 2 seconds units. The non-uniform time unit applied here is an example and is not limited thereto.

The VQ-Wav2Vec processing unit 120 inputs the speech signal for each section generated by the section generator 110 to the VQ-Wav2Vec model and outputs a set of vector quantization information for each section sequentially n times. can be done

2 is a diagram showing the architecture of the VQ-Wav2Vec model.

2, the VQ-Wav2Vec model is an encoder network (encoder

network, f), a Vector Quantization module (q) and a context network (c). Both the encoder network and the context network may be convolutional neural networks.

The encoder network (f) encodes the input discrimination target voice signal (X) into a hidden representation (Z). The hidden representation (Z) is also called the continuous representation (Z).

)로 변환한다.The Vector Quantization module (q) converts a continuous representation (Z) into a discrete representation (

) is converted to

)를 만든다.For the Vector Quantization module, for example, one of Gumbel Softmax or K-means Clustering methods may be used. When using Gumbel Softmax, the Vector Quantization module makes a logit by linearly transforming Z, then applies Gumbel Softmax and argmax to create a one-hot vector. This transforms the continuous variable Z into a discrete variable, which is Vector Quantization. After that, the Vector Quantization module does the dot product of the embedding matrix to obtain a discrete representation (

) to make

)를 context representation(C)로 변환한다. VQ-Wav2Vec 학습을 마치면 C는 음성 피처로 사용될 수 있다.The context network (g) is a discrete representation (

) to the context representation (C). After completing VQ-Wav2Vec training, C can be used as a voice feature.

For example, if a voice signal to be identified between about 10 and 30 seconds sampled at 16 kHz is input to the VQ-Wav2Vec model, the VQ-Wav2Vec model can output vector quantized information of [T, 2]. . Here, T is a time step, information coming out in two groups corresponds to vector quantization information, and the VQ-Wav2Vec model may provide a result of accumulating information corresponding to all time steps. The accumulated information in the VQ-Wav2Vec model can be regarded as a discrete variable. T is also changed according to the length of the voice signal. For example, if T=100 when a 30-second wave is input, T=200 when a 60-second wave is input.

The violence determination unit 130 may perform artificial intelligence learning of vector quantization information for each section input from the VQ-Wav2Vec processing unit 120 to output discrimination information on whether or not violent language exists for each section.

Referring to FIG. 1 , the violence determining unit 130 includes a learning unit 132 and an accumulation determining unit 134 .

The learning unit 132 may artificially learn vector quantization information for each section input from the VQ-Wav2Vec processing unit 120 , and output n sets of temporary determination information on whether or not violent language exists for each section. In order to determine a specific section corresponding to a violent language among the discrimination target voice signals, the learning unit 132 may divide the discrimination target voice signal into several segments to temporarily determine whether the discrimination target voice signal is a violent voice. Among the results of artificial intelligence learning of vector quantization information for each section, 1 is a violent section, and 0 is a non-violent section.

The learning unit 132 uses a deep learning model such as a Fully Connected Network (FNN), a machine learning method such as a decision tree or LGBM (Light GBM) to learn artificial intelligence using a small amount of violent voice DB (Database). ) alone can be learned.

In addition, since the learning unit 132 learns the information accumulated by the time step, it has the advantage of being able to determine whether or not violent language is present even if the length of the voice signal to be determined is variable.

When the result of learning n sets (that is, n times) of vector quantization information for each section from the learning unit 132 is input, the accumulation determining unit 134 returns a learning result corresponding to the same time among the n times of learning results for each section. By accumulating, it is possible to finally determine whether there is violence against the voice signal.

That is, when n sets of results of learning vector quantization information for each section (that is, provisional discrimination results) are generated, the cumulative determination unit 134 arranges the n sets of learning results for each section based on the time axis, and corresponds to the same time. accumulating learning results.

And, the cumulative determination unit 134 compares the learning result accumulated for the same time with a preset determination reference value, and if the accumulated learning result is greater than or equal to the determination reference value, it corresponds to the violent section, and if it is less than, it is determined as the non-violent section. can

In addition, it is possible to set the determination reference value of the accumulation determination unit 134 according to the application of the present invention. When the violence discrimination performance increases the precision, it is set as the first discrimination standard value, and when the sensitivity (recall) is increased, it can be used as a second discrimination standard value that is relatively lower than the first discrimination standard value. The first discrimination reference value and the second discrimination reference value can be changed by an administrator. The higher the precision, the greater the first discrimination reference value, and the higher the sensitivity, the lower the second discrimination reference value.

3 is a diagram illustrating a case in which the section generating unit 110 divides the discrimination target voice signal into different time units and divides it into a plurality of sections.

Referring to FIG. 3 , seven sets of operations for dividing the 40-second discrimination target voice signal into different time units were performed, respectively, in units of 2 seconds, units of 4 seconds, units of 5 seconds, units of 8 seconds, units of 10 seconds, and units of 20 seconds. unit and 40 seconds. Accordingly, the sections generated for each set are 20, 10, 8, 5, 4, 2, and 1, respectively.

4 is a diagram showing a result of the provisional determination of the learning unit 132 .

Referring to FIG. 4 , the VQ-Wav2Vec processing unit 120 receives a discrimination target voice signal for each divided section and generates accumulated vector quantization information for each section, and the learning unit 132 uses the accumulated vector quantization information for artificial intelligence. By learning, it is temporarily determined whether or not it corresponds to violent language by section. As for the provisional determination result, the section corresponding to violent language is expressed as 1, and the section corresponding to non-violent language is expressed as 0. The VQ-Wav2Vec processing unit 120 and the learning unit 132 perform 7 sets of these operations as shown in FIG. 3 to provide 7 temporary determination results.

5 is a diagram showing a determination result of the accumulation determination unit 134 .

Referring to FIG. 5 , the cumulative determination unit 134 accumulates the seven sets of provisional determination results (0 or 1 in time series for each section) shown in FIG. 4 at the same time, and the accumulated provisional determination result and the determination reference value (dotted line) is compared to finally determine the violent section and the non-violent section.

6 is a flowchart schematically illustrating a method for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention.

The electronic device for performing the method for detecting a section of violence in a voice signal using the vector quantization information shown in FIG. 6 is the device for detecting a section of violence 100 described with reference to FIGS. 1 to 5 or a computing system to be described later with reference to FIG. 800).

Referring to FIG. 6 , the electronic device sequentially performs n (n is a positive number greater than or equal to 1) a set operation of generating a plurality of sections by dividing an m-second voice signal for determining violent language into time units (S610). ).

The electronic device sequentially performs one set operation of outputting vector quantization information for each section by inputting the voice signal for each section generated in step S610 to the VQ-Wav2Vec model n times (S620).

In step S610, a time unit applied to division may be set differently every time a set operation of dividing a voice signal into time units is performed n times. In addition, time units applied in one set operation may be uniform or non-uniform, and may be changed. The electronic device performs AI-learning of vector quantization information for each section input from step S620 through a deep learning model, and outputs discrimination information on whether or not violent language is present for each section (S630).

7 is a flowchart illustrating in detail step S630 of FIG. 6 .

Referring to FIG. 7 , the electronic device may perform artificial intelligence learning of vector quantization information for each section input from step S620 to output temporary determination information on whether or not violent language exists for each section ( S632 ).

The electronic device arranges the temporary determination information for each section n times input from step S632 so that the time axis coincides with each other (S634).

The electronic device may finally determine whether violence against the voice signal is present by accumulating information corresponding to the same time among the n-time temporary determination information for each section arranged so that the time axis coincides in step S634 ( S636 ).

8 is a block diagram illustrating a computing system 800 for executing a method for detecting a violent section of a voice signal using vector quantization information according to an embodiment of the present invention.

Referring to FIG. 8 , the computing system 800 includes at least one processor 810 , a memory 830 , a user interface input device 840 , a user interface output device 850 , and storage connected through a bus 820 . 860 , and a network interface 870 .

The processor 810 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 830 and/or the storage 860 . The memory 830 and the storage 860 may include various types of volatile or non-volatile storage media. For example, the memory 830 may include a read only memory (ROM) 831 and a random access memory (RAM) 832 . The method for detecting a violent section of a voice signal using vector quantization information according to the present invention may be provided by being included in a storage medium that can be read through the computing system 800 by tangibly implementing a program of instructions for implementing it. can be easily understood by those skilled in the art.

Accordingly, the steps of the method or algorithm described in relation to the embodiments disclosed herein may be directly implemented in hardware, software module, or a combination of the two executed by the processor 810 . A software module resides in a storage medium (ie, memory 830 and/or storage 860) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM. You may. An exemplary storage medium is coupled to the processor 810 , the processor 810 capable of reading information from, and writing information to, the storage medium. Alternatively, the storage medium may be integral with the processor 810 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

In the above, even though all the components constituting the embodiment of the present invention are described as being combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, although all of the components may be implemented as one independent hardware, some or all of the components are selectively combined to perform some or all functions of the combined components in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer readable storage medium (Computer Readable Media), read and executed by the computer, thereby implementing the embodiment of the present invention.

On the other hand, although described and illustrated in relation to a preferred embodiment for illustrating the technical idea of the present invention as described above, the present invention is not limited to the configuration and operation as shown and described as such, and deviates from the scope of the technical idea. It will be apparent to those skilled in the art that many changes and modifications to the present invention are possible without the above. Accordingly, all such suitable alterations and modifications and equivalents are to be considered as falling within the scope of the present invention. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: Violence section detection device of voice signal
110: section generator
120: VQ-Wav2Vec processing unit
130: violence discrimination unit
132: study unit
134: cumulative discrimination unit

Claims (8)

a section generating unit that sequentially performs a set operation of generating a plurality of sections by dividing an m-second voice signal for determining violent language into time units n (n is a positive number of 2 or more) times;
a VQ-Wav2Vec processing unit that sequentially performs a set operation n times for inputting the generated speech signal for each section into a VQ-Wav2Vec (Vector Quantization Wave to Vector) model and outputting vector quantization (vector quantization) information for each section; and
Including; a violence determination unit that artificially learns n sets of vector quantization information for each section input from the VQ-Wav2Vec processing unit through a deep learning model and outputs discrimination information on whether or not violent language is present for each section;
The violence determination unit,
AI-learning the Vector Quantization information for each section input from the VQ-Wav2Vec processing unit outputs n sets of temporary identification information on whether or not violent language is used for each section, and the temporary identification information is 1 if the section corresponds to violent language and non-violent language a learning unit that is 0 if the section corresponds to ; and
When n sets of provisional determination information for each section are input in the learning unit, the n sets of provisional determination information for each section are arranged based on the time axis, and n sets of provisional determination information for each section arranged based on the time axis are displayed in the same time zone. A cumulative determination unit that accumulates for each time period and compares the results accumulated for each time period with a preset determination reference value, and determines that the accumulated result corresponds to a violent section if it is greater than or equal to the determination standard value, and is determined to correspond to a non-violent section if it is less than; and
The time unit applied every time the section generator performs one set operation of dividing the voice signal into time units n times is different, and the time unit applied in the one set operation is changeable to either equal or non-uniform. A device for detecting the violence section of a voice signal using vector quantization information. delete delete delete (A) sequentially performing, by the electronic device, n (n is a positive number equal to or greater than 2) times of one set operation of generating a plurality of sections by dividing an m-second speech signal for determining violent language;
(B) One set operation in which the electronic device inputs the speech signal for each section generated in step (A) into the VQ-Wav2Vec (Vector Quantization Wave to Vector) model and outputs Vector Quantization information for each section sequentially performing n times; and
(C) the electronic device, through the deep learning model, artificial intelligence learning the n sets of vector quantization information for each section input from the step (B) and outputting discrimination information on whether or not violent language is provided for each section; including,
The step (C) is,
(C1) AI learns Vector Quantization information for each section input from step (B) and outputs n sets of temporary determination information on whether or not violent language is used for each section, and the provisional determination information is 1 if the section corresponds to violent language , including 0 if the section corresponding to non-violent language; and
(C2) arranging the n sets of temporary determination information for each section input from the step (C1) so that the time axis coincides with each other; and
(C3) accumulating n sets of temporary discrimination information for each section arranged so that the time axis coincides with each other in step (C2) at the same time, comparing the accumulated result at the same time with a preset discrimination reference value, and comparing the accumulated result Determining that corresponds to a violent section if is greater than or equal to the determination standard value, and is less than or equal to a non-violent section;
The step (A) is,
A time unit applied every time one set operation of dividing the voice signal into time units is performed n times, and the time unit applied in the one set operation is changeable to either equal or non-uniform. A method of detecting the violence section of a voice signal using vector quantization information. delete delete delete

Images