JP2020140050A - Caption generation device, caption generation method and program - Google Patents

Abstract

To provide a caption generation device, a caption generation method and a program that enable generation of a caption using a neural network for an acoustic signal.SOLUTION: A caption generation device comprises a caption generation unit which divides, when generating a spectrogram for an acoustic signal, the spectrogram to a fixed length into one or more blocks, inputs the blocks to a convolutional neural network to extract a feature quantity vector, and inputs the extracted feature quantity vector to a recursion type neutral network to generate a caption for the acoustic signal.SELECTED DRAWING: Figure 1

Description

The present invention relates to a caption generator, a caption generator, and a program.

Some TV programs display telops and subtitles. Such telops and subtitles are created, for example, by human input using a keyboard based on a manuscript. The live news program is created by typing on the keyboard while listening to the words spoken by the announcer.

In addition, a method of generating and adding a caption to an image is being studied. For example, it has been proposed to obtain image features by inputting an image into a convolutional neural network and generate a caption using a recurrent neural network (RNN) (for example, Patent Document 1, Non-Patent). Reference 1).

A caption generative model from an image using deep learning such as the techniques described in Patent Document 1 and Non-Patent Document 1 converts the image into a fixed-length intermediate vector representation using a convolutional neural network (CNN). (See, for example, Non-Patent Document 2), and its intermediate vector representation is converted into a caption using RNN (see, for example, Non-Patent Document 3).

JP-A-2018-1249969

"Show and Tell: A Neural Image Caption Generator", Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan ", IEEE, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, p3156-3164 "Overfeat: Integrated recognition, localization and detection using convolutional networks", Sermanet, Pierre, et al., ArXiv preprint arXiv: 1312.6229, 2013 "Neural machine translation by jointly learning to align and translate", Bahdanau, Dzrnitry, et al, arXiv preprint arXiv: 1409.0473, 2014.

There is a desire to generate and add captions to acoustic signals as well. There is a problem that the contents of data containing general acoustic signals cannot be understood until they are actually heard. However, even if an attempt is made to apply the image caption generation method of each conventional technique to an acoustic signal, the input signal is a one-dimensional time-series signal, not two-dimensional like an image, and the acoustic signal has a variable length, so it looks like an image. It is not possible to make a fixed length expression by resizing to.

The present invention has been made in view of the above problems, and provides a caption generation device, a caption generation method, and a program capable of generating captions for an acoustic signal using a neural network. The purpose is.

(1) In order to achieve the above object, the caption generator <1> according to one aspect of the present invention divides the spectrogram into a fixed length and one or more blocks when generating a spectrogram for an acoustic signal. Then, the block is input to the convolutional neural network <CNN> to extract the feature amount vector, and the extracted feature amount vector is input to the recurrent neural network <RNN> to generate a caption for the acoustic signal. A generation unit <12> is provided.

(2) Further, in the caption generator according to one aspect of the present invention, the spectrogram may be a logarithmic mel frequency spectrogram.

(3) Further, in the caption generator according to one aspect of the present invention, the recurrent neural network is composed of a multi-layered long-term short-term storage layer <LSTM>, and the recurrent neural network is a first recurrent neural network. A network <RNN unit 1222> and a second recurrent neural network <RNN unit 1231> are provided, and the first recurrent neural network inputs the extracted feature quantity vector into the multi-layered long-term and short-term storage layer. The intermediate representation vector may be generated, and the second recurrent neural network may input the intermediate representation vector into the multi-layered long-term and short-term storage layer to generate the caption.

(4) Further, in the caption generation device according to one aspect of the present invention, the block may be a grayscale image of the spectrogram.

(5) In order to achieve the above object, the caption generation method according to one aspect of the present invention is a procedure in which the acquisition unit acquires an acoustic signal and when the caption generation unit generates a spectrogram for the acoustic signal. , The procedure of dividing the spectrogram into one or more blocks into one or more blocks, inputting the blocks into a convolutional neural network to extract a feature amount vector, and a recursive type of the extracted feature amount vector by the caption generator. It includes a procedure for generating a caption for the acoustic signal by inputting it into a neural network.

(6) In order to achieve the above object, the program according to one aspect of the present invention uses the spectrogram in the procedure of acquiring an acoustic signal and in causing the computer of the caption generator to generate a spectrogram for the acoustic signal. The acoustic signal is divided into one or more blocks by dividing into a fixed length, and the block is input to a convolutional neural network to extract a feature amount vector, and the extracted feature amount vector is input to a recursive neural network. And perform the steps to generate a caption for.

According to (1), (3), (5), and (6) described above, a one-dimensional acoustic signal can be converted into two dimensions, and a caption can be generated for the acoustic signal using a neural network. ..

Further, according to (2) described above, a caption can be generated by using a caption that matches the human hearing.
Further, according to (4) described above, the amount of calculation can be reduced.

It is a figure which shows the configuration example of the caption generation apparatus which concerns on embodiment. It is a figure which shows the outline of the process performed by the caption generation apparatus which concerns on embodiment. It is a figure which shows the outline of the process performed by the caption generation apparatus which concerns on embodiment. It is a figure which shows the process performed by the pre-processing part and the encoder which concerns on embodiment. It is a figure which shows the processing example of the decoder which concerns on embodiment. It is a figure which shows the learning process example in the encoder which concerns on embodiment. It is a figure which shows the structure of LSTM and the processing example. It is a flowchart of the processing procedure example of the caption generation apparatus which concerns on embodiment. It is a figure which shows the example of the acoustic signal used for evaluation. It is a figure which shows the architecture of a learning model. It is a figure which shows the configuration example and the processing procedure example of each module. It is a figure which shows the output example of the output caption which did not match the correct answer. It is a figure which shows the evaluation result example.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, integrating and recognizing a large amount of information contained in an acoustic signal is referred to as acoustic scene understanding.

<Caption generator>
FIG. 1 is a diagram showing a configuration example of the caption generation device 1 according to the present embodiment. As shown in FIG. 1, the caption generation device 1 includes an acoustic signal acquisition unit 11, a caption generation unit 12, and an output unit 13.
Further, the caption generation unit 12 includes a preprocessing unit 121, an encoder 122, and a decoder 123.
The preprocessing unit 121 includes a cutting unit 1211, a normalizing unit 1212, and a short-time Fourier transform unit 1213.
The encoder 122 includes a CNN unit 1221 and an RNN unit 1222 (first recurrent neural network).
The decoder 123 includes an RNN unit 1231 (second recurrent neural network).

The caption generation device 1 generates a caption for the acquired acoustic signal. That is, the caption generation device 1 understands the acoustic scene for the acoustic signal and generates a caption.

The acoustic signal acquisition unit 11 acquires an acoustic signal and outputs the acquired acoustic signal to the caption generation unit 12. The acoustic signal may be a sound picked up by a microphone or a recorded sound signal.

The caption generation unit 12 generates a caption for the acoustic signal output by the acoustic signal acquisition unit 11.

The preprocessing unit 121 performs preprocessing on the acquired acoustic signal to convert the acoustic signal, which is one-dimensional information, into two-dimensional information.

The cutout unit 1211 cuts out an acoustic signal from the acoustic signal output by the acoustic signal acquisition unit 11 using a window having a predetermined time width. The method of cutting out the acoustic signal will be described later. The cutting unit 1211 outputs the cut out acoustic signal to the normalizing unit 1212.

The normalization unit 1212 normalizes the acoustic signal cut out by the cutout unit 1211, and outputs the normalized acoustic signal after the cutout to the short-time Fourier transform unit 1213.

The short-time Fourier transform unit 1213 performs a short-time Fourier transform (SFTT) on the normalized and cut-out acoustic signal output by the normalization unit 1212. By this processing, a two-dimensional spectrogram image for each cut-out acoustic signal is generated, and the one-dimensional acoustic signal is converted into two-dimensional image data. The short-time Fourier transform unit 1213 sequentially outputs the converted two-dimensional spectrogram image to the encoder 122.

The encoder 122 uses the two-dimensional spectrogram image output by the preprocessing unit 121 to generate a vector which is an intermediate representation. The method for processing the encoder 122 will be described later.

The CNN part 1221 is a convolutional neural network (CNN). The CNN unit 1221 inputs a two-dimensional spectrogram image sequentially output by the short-time Fourier transform unit 1213 into the CNN to generate a feature vector. The CNN unit 1221 outputs the generated feature quantity vector to the RNN unit 1222. The CNN unit 1221 extracts the feature amount of the two-dimensional spectrogram image from the two-dimensional spectrogram image by, for example, a convolution process or a pooling process using a kernel. The CNN unit 1221 combines the extracted features with a binding layer to generate a feature vector. The CNN unit 1221 generates one feature amount vector for one image. The configuration and operation of the CNN will be described later.

The RNN unit 1222 is a recurrent neural network (RNN). The RNN unit 1222 collects the feature quantity vectors output by the CNN unit 1221 by RNN to generate a vector which is an intermediate representation. The generated intermediate representation vector has a pseudo variable length. The RNN unit 1222 outputs the generated vector, which is an intermediate representation, to the decoder 123. The RNN unit 1222 has a structure in which RNNs are stacked in multiple layers. The RNN is a neural network having a network structure in which the value of the hidden layer is input to the hidden layer again. By inputting the feature vector generated by the CNN unit 1221 one by one into the LSTM (Long short-term memory) described later of the RNN, words are sequentially output by using it as a trigger. The configuration and operation of the RNN will be described later.

The decoder 123 generates a caption using a vector which is an intermediate representation generated by the encoder 122. The method for processing the decoder 123 will be described later.

The RNN unit 1231 is a recurrent neural network (RNN). The RNN unit 1231 inputs a vector, which is an intermediate representation output by the encoder 122, to the RNN to generate a caption, and outputs the generated caption information (for example, text information) to the output unit 13. The RNN unit 1231 has a structure in which RNNs are stacked in multiple layers.

The output unit 13 converts the caption information output by the decoder 123 into image information, and outputs the converted image information to an external device (not shown) such as an image display device (not shown). The external device may be a tablet terminal, a smartphone, or the like. Further, the output unit 13 may also output the recorded acoustic signal to the external device.

<Outline of processing flow>
First, an outline of the processing flow will be described with reference to FIGS. 2 and 3.
2 and 3 are diagrams showing an outline of the processing performed by the caption generation device 1 according to the present embodiment.
As shown in FIGS. 2 and 3, the acoustic signal acquisition unit 11 acquires the acoustic signal (reference numeral g1 in FIG. 2 and reference numeral g11 in FIG. 3). In reference numerals g1 in FIG. 2 and reference numeral g11 in FIG. 3, the horizontal axis represents time and the vertical axis represents the magnitude of amplitude.
As shown by the reference numeral g12 in FIG. 3, the preprocessing unit 121 performs a cutting process, a normalization process, and a short-time Fourier transform process. As a result, spectrograms which are a plurality of images are generated as shown by the reference numeral g13 in FIG. 3, and the generated spectrograms are input to the encoder (reference numeral g2 in FIG. 2 and reference numeral g13 in FIG. 3).

Next, as shown by the reference numeral g14 in FIG. 3, the encoder 122 inputs the spectrogram to the CNN to generate a vector, and inputs the generated vector to the RNN. The RNN unit 1222 is provided with a multi-stage LSTM layer as described later. The encoder 122 generates an intermediate representation vector (reference numeral g3 in FIG. 2 and reference numeral g15 in FIG. 3) by this processing.

Next, as shown by the reference numerals g4 in FIG. 2 and the reference numeral g16 in FIG. 3, the decoder 123 inputs the vector of the intermediate representation into the RNN to generate a caption. The RNN unit 1222 is provided with a multi-stage LSTM layer as described later. The generated caption is "ringing the bell" as shown by reference numeral g5 in FIG.

<Processing performed by the pre-processing unit 121 and encoder 122>
The processing performed by the preprocessing unit 121 and the encoder 122 will be described in detail with reference to FIG.
FIG. 4 is a diagram showing processing performed by the preprocessing unit 121 and the encoder 122 according to the present embodiment.
Reference numeral g101 is an acquired acoustic signal. In reference numeral g101, the horizontal axis is time (frame) and the vertical axis is the magnitude of amplitude.

As shown by reference numerals g102 and reference numeral g104, the cutting unit 1211 cuts out an acoustic signal using a window having a predetermined value (for example, 204480 (reference numeral g103)). As shown by the reference numeral g103, the cutting unit 1211 sequentially cuts out the acoustic signals so that the predetermined times overlap (overlap). By this process, a plurality of waveform data are generated.

Next, as shown by reference numerals g105 and reference numeral g106, the normalization unit 1212 normalizes the amplitude of the cut out acoustic signal in the range of -1 to 1. As shown by the reference numeral g107, the normalization unit 1212 normalizes the FFT (Fast Fourier Transform) window with 1024 and the overlap (overlap) with 512.

Next, as shown by reference numerals g108 and reference numerals g109, the short-time Fourier transform unit 1213 sequentially performs a short-time Fourier transform on the normalized and cut out acoustic signal. By this process, a plurality of spectrograms are generated from the waveform data. Reference numeral g109 is a spectrogram image, in which the horizontal axis represents time and the vertical axis represents frequency (Mel bin).
In the present embodiment, the acoustic signal is converted into a log-scaled mel frequency spectrogram by this preprocessing to obtain a 1-channel grayscale image. As a result, it is possible to input to the CNN section 1221 of the encoder 122 in the subsequent stage. As described above, in the present embodiment, the amount of calculation can be reduced by grayscale the spectrogram image.

Next, as shown by the reference numeral g110, the CNN section 1221 generates a vector (reference numeral g111) by sequentially inputting a plurality of spectrograms into the CNN. Each image of the elements of the sequence is passed through CNN. CNN makes the acoustic signal a vector sequence. Note that one feature vector corresponds to a spectrogram image corresponding to one cut out waveform data. n is the number of spectrograms and is the number of cut out waveforms.

Next, as shown by the reference numeral g112, the RNN unit 1222 inputs the vector output by the CNN unit 1221 into the RNN to generate a vector (reference numeral g113) which is an intermediate representation. The vector sequence is input to the RNN on the encoder side step by step, and the final state is taken out as an intermediate representation.

As described above, in the present embodiment, when taking a spectrogram of an acoustic signal, the spectrogram is divided (blocked) into a fixed-length spectrogram by taking a plurality of fixed-length spectrograms on the waveform data while shifting the window. Generate one or more blocks (spectogram images). That is, when performing the short-time Fourier transform, the cutout process and the normalization are performed.
Further, in the present embodiment, the feature quantity vector is extracted by the encoder 122 to identify the type of sound source included in the acoustic signal, that is, the sound source identification process is performed.

<Logarithmic mel frequency spectrogram>
Here, the logarithmic mel frequency spectrogram will be described. The logarithmic frequency spectrogram is an amplitude spectrogram obtained by performing an FTFT on an acoustic signal and converting it according to human hearing. In addition, the logarithmic mel frequency spectrogram is obtained by expanding / compressing the frequency direction of the amplitude spectrogram and converting the amplitude value in order to match the human perception.

First, the short-time Fourier transform unit 1213 adjusts the frequency characteristics to the auditory sense using the Mel scale. The Mel scale is a frequency axis that reflects human speech perception. The short-time Fourier transform unit 1213 converts f [Hz] into a mel scale m by the following equation (1).

Next, the short-time Fourier transform unit 1213 applies a plurality of bandpass filters arranged at equal intervals on the mel scale to the amplitude spectrum in order to convert the frequency axis of the amplitude spectrum obtained by the STFT into the mel scale. To do. These plurality of bandpass filters are called a mel filter bank and are a set of filters having a triangular window.
Next, the short-time Fourier transform unit 1213 performs an operation of adjusting the loudness of the sound to the auditory sense. Humans perceive loudness on a log scale. Therefore, the short-time Fourier transform unit 1213 performs an operation of adjusting the loudness to the auditory sense by taking a log of the amplitude after applying the mel filter bank.

By the above processing, a logarithmic mel frequency spectrogram can be obtained by applying a mel filter bank in the frequency direction to a normal spectrogram and converting the amplitude into a log scale.

<Structure of neural network>
Next, the structure of the neural network will be described.
v The word dictionary _{size, I = (I 0, ...} I M-1) log mel-frequency spectrogram of the sequence calculated from the waveform data cut _{out, S = (S 0, ...} S N-1) and. Also, each word in the caption is represented by v dimensions of one-hot vector _{S t.} The one-hot vector is a vector in which only one element of the vector is 1 and the other elements are 0. However, S ₀ is a special start word indicating the start of the caption, _SN is a special end word indicating the end of the caption, and _{St, t} {1, ..., N-1} are the actual caption words. Correspond. The neural network of the encoder (encoder) and the decoder (decoder) can be expressed by the following equations (2) to (7).

In the formulas (2) to (7), CNN represents CNN processing, RSTM represents LTSM processing, and Softmax represents processing. Moreover, e is embedded vector, _{h t} is the state of LSTM the encoder side, _{s t} is the state of LSTM the decoder side. W _e (∈ R ^{e × v} ) is the word embedding matrix on the encoder side, and W _p is the output of the LSTM on the decoder side. LSTM _h and LSTM _y are functions that determine the output from the state. _pt is a v-dimensional vector whose elements represent the output probabilities of each word.

LSTM ^encoder and ^{LSTM decoder} is a zero-th step. In addition, the input is a sequence of images, which is the input for each step of the LSTM ^encoder . Final state _{h M-1} of LSTM ^encoder is an intermediate representation, which is used as the initial state s-1 of ^{LSTM decoder} (Equation (4)), the corresponding caption is generated.
A model based on an RNN that outputs such a variable length sequence is referred to as a Seq-to-Seq (Sequence-to-Sequence) model in the embodiment.

Learning is performed using an error function. The error function learns the caption S corresponding to the time-division spectrogram sequence I of the acoustic signal by using the following equation (8) as in the model in the vision and minimizing it.

In the present embodiment, in the Seq-to-Seq model, the performance is improved by stacking RNNs in multiple layers. Here, in the case of a two-layer LSTM, it is represented by the following equations (9) to (12).

However, in the equations (9) to (12), x _t is an input and y _t ² is an output. In the case of a two-layer LSTM, the output sequence y _t ¹ of the first layer LSTM ¹ becomes the output sequence of the second layer LSTM ² , and the output sequence y _t ² of the second layer becomes the output of the entire multi-layer LSTM.
When used in the Seq-to-Seq model, the intermediate representation can be passed by aligning the number of intermediate layers between the LSTM on the encoder side and the LSTM on the decoder side.

Further, in the RNN on the encoder side of the present embodiment, a bidirectional RNN (Bidirectional RNN) capable of inputting a sequence from the past to the future direction (forward direction) and also from the future to the past direction (reverse method). It consisted of. The LSTMs of the forward and reverse encoders are calculated by the following equations (13) and (14), and the final states h _M-1 ^f and h _M-1 ^b are combined to form the intermediate representation h _M-1 = [h. _M-1 ^f ; h _M-1 ^b ].

In the equation (13), the LSTM ^f is a forward LSTM having a state h _t ^f . Further, in the equation (14), the LSTM ^b is a reverse LSTM having a state h _t ^b .

<Attention mechanism>
In the Seq-to-Seq model, the inputs are grouped into one vector for each step in the RNN state, so that when the input sequence becomes long, it becomes difficult to convey the first input information of the sequence to the decoder. Therefore, in the present embodiment, the attention mechanism (Attention Mechanism) represented by the following equations (15) to (18) is provided so that the input can be directly referred to at the time of decoding. In the attention mechanism, when the state s _i is obtained in each step i of the decoder, the caption generation unit calculates the score α _ij between the state s _i-1 of the current decoder and the state h _{j in} each step j of the past encoder. 12 calculates. The input can be referred to by using this score in the calculation of the state s _j in step i.

However, in the equations (15) to (18), e _ij is a score between the state s _i-1 of the i-1 step of the decoder and the state h _j of the j step of the encoder. Further, α _ij is obtained by normalizing this e _ij in the step j direction of the encoder (Σ _j α _ij = 1.0 ≦ α _ij ≦ 1). Also, taking the weighted average c _i the alpha _ij as a weight, thereby reflecting the c _i in the calculation of the state s _i for the next decoder. Further, the inner product s _i-1 · h _j is used for the score function a (s _i-1 , h _j ).

<Decoder processing>
Next, a processing example of the decoder 123 will be described.
FIG. 5 is a diagram showing a processing example of the decoder 123 according to the present embodiment. Note that h ₁ , ..., H ₄ (reference numerals g307, g310, g313, g316) are states of each layer.
First, the RNN unit 1231 sets the intermediate representation vector (reference numeral g201) output by the encoder 122 as the initial state of the RNN (reference numeral g301).
Then, the RNN unit 1231 inputs the special word START representing the beginning of the caption as the input of the first step (reference numeral g211) of the RNN (reference numerals g302, g303). As a result, by taking the word corresponding to the index having the maximum probability p1 (codes g305, g306), the first word of the caption is output as the output of the first step.

Next, the RNN unit 1231 inputs a word corresponding to the output word of the first step as the input of the second step (reference numeral g308). As a result, the second word is output in the output of the second step (reference numeral g212) of the RNN (reference numeral g309).
Hereinafter, the caption is generated by repeating the process of setting the output of the second step to the input of the third step (reference numerals g311 to g318, g321).
The sentence generation ends when the output of the step outputs a special word END (reference numeral g318).

<Learning process>
Next, an example of learning processing in the encoder 122 will be described.
FIG. 6 is a diagram showing an example of learning processing in the encoder 122 according to the present embodiment. Reference numeral g401 is a spectrogram generated by the short-time Fourier transform unit 1213, and the horizontal axis is time and the vertical axis is mel frequency. This spectrogram is input to the CNN included in the CNN unit 1221 (reference numeral g402).

Here, the learning in the CNN is divided into two stages, a learning stage of the CNN part and a learning stage of the entire model.

In the learning stage of the CNN part, a network for classifying acoustic signals is created using a classifier after the CNN, this network is trained, and only the CNN part in the trained network is extracted. Here, the classifier (classifier) is an algorithm for inputting a feature amount and classifying what feature it represents. Further, the classification is a classification to the label given to the acoustic signal. For example, when captions are generated for 33 types of environmental sounds, each of these environmental sounds is a class, and the total number of classes is 33. The teacher data in the learning stage of the CNN part is, for example, an environmental sound (knock, flute, cow, bark, chime, etc.) class label.

Here, in the learning stage of the entire model, the CNN learned in the learning stage of the CNN part is brought in to learn the entire model. The teacher data in the learning stage of the entire model is, for example, sentence data such as "Snare Drum is followed by Gong and then Bark".

<LSTM>
Next, the LSTM will be supplemented.
FIG. 7 is a diagram showing a configuration of an LSTM and a processing example. In FIG. 7, x _t is the input at the t step, y _t is the output at the t step, _ct is the memory cell, i is the input gate, f is the forgetting gate, and o is the oblivion gate. It is an output gate, and × is an element product.
With the configuration and processing of FIG. 7, it is possible to determine how much current and past information is used.

Further, the value used in the LSTM is calculated by the RNN unit 1222 or the RNN unit 1231 using the following equations (19) to (24).

In equations (19) to (21), σ is a sigmoid function.

<Processing procedure of caption generator>
Next, an example of the processing procedure of the caption generation device 1 will be described.
FIG. 8 is a flowchart of a processing procedure example of the caption generator according to the present embodiment.

(Step S1) The acoustic signal acquisition unit 11 acquires an acoustic signal and outputs the acquired acoustic signal to the caption generation unit 12.

(Step S2) The cutting unit 1211 cuts out an acoustic signal from the acoustic signal output by the acoustic signal acquisition unit 11 by using a window having a predetermined time width.

(Step S3) The normalization unit 1212 normalizes the acoustic signal cut out by the cutout unit 1211.

(Step S4) The short-time Fourier transform unit 1213 performs a short-time Fourier transform on the normalized and cut-out acoustic signal output by the normalization unit 1212 to generate a two-dimensional spectrogram image for each cut-out acoustic signal. To do.

(Step S5) The CNN unit 1221 inputs the two-dimensional spectrogram image sequentially output by the short-time Fourier transform unit 1213 into the CNN to generate a feature vector. The CNN unit 1221 generates one feature amount vector for one image.

(Steps S6 and S7) The RNN unit 1222 collects the feature quantity vectors output by the CNN unit 1221 by RNN to generate a vector which is an intermediate representation.

(Steps S8 and S9) The RNN unit 1231 inputs a vector, which is an intermediate representation output by the encoder 122, to the RNN to generate a caption.

<Evaluation result>
Next, an example of the evaluation result of evaluating the caption generation device 1 of the present embodiment will be described with reference to FIGS. 9 to 13.
A dataset of acoustic signals and captions is required for learning and evaluation. For this reason, in the evaluation, we created a mixed sound that randomly synthesized sound sources containing only a single class, and a data set of captions corresponding to it. FIG. 9 is a diagram showing an example of an acoustic signal used for evaluation. In FIG. 9, the horizontal axis is time (seconds) and the vertical axis is frequency [Hz]. In the example shown in FIG. 9, the SnareDrum (reference numeral g501) sounds between 0 and 2 seconds, the Kong (reference numeral g502) sounds between 2 and 4 seconds, and the Bark (reference numeral g503) sounds between 4 and 6 seconds. It has become. The correct data for the caption in this example is "Snare Drum followed by Gong, followed by Bark." In this way, in caption generation, it was evaluated whether it was possible to express what sounds were sounding and in what order.

33 class (33 types) of environmental sounds (knock, flute, cow, bark, chime, etc.) were used as sound sources, and mixed sounds were created by connecting about 3 sound sources so that there was no opalap. However, a silent signal having a random length of about 0 to 0.6 seconds was inserted between the sound source signals. In the evaluation, it was assumed that a caption explaining the type and order of the monaural input acoustic signal was generated. 18,000 pairs of mixed sounds and captions were used as the learning data set, and 2000 pairs were used as the evaluation data set.

For the training of CNN, the existing trained models shown in FIGS. 10 and 11 were used (see, for example, Reference 1). FIG. 10 is a diagram showing the architecture of the learning model. FIG. 11 is a diagram showing a configuration example and a processing procedure example of each module. In FIG. 11, k and n indicate the size of the convolution filter and the number of softmax layers. Also, BN is batch normalization, Concat is feature concatenation, Relu is a modified linear unit, Conv is a linear convolution, MaxPool is the maximum pooling, and GAP is the global average pooling. Reference numeral g511 is a module configuration and processing procedure of Low-level k. Reference numeral g512 is a module configuration and processing procedure of Dense Net-k. Reference numeral g513 is an n-head classifier module configuration and processing procedure. The accuracy of classifying the learned CNN part is about 90%.
References; “Audio tagging system for DCASE 2018: Focusing on label noise, data augmentation and its efficient learning”, Il-Young, Jeong, et al. DCASE2018 Challenge., 2018

The learning of the model was carried out in two stages. First, prepare a CNN for sound source identification. For this, the existing trained model described above was used. Next, transfer learning of the entire model was performed on this CNN using the training data set described above. The waveform data was cut out for 1.28 seconds so as to overlap by 50%.

FIG. 12 is a diagram showing an output example of an output caption that did not match the correct answer. In FIG. 12, the sentences indicated by the reference numerals 601 and g611 are the correct answer captions, and the sentences indicated by the reference numerals g602 and g612 are the output captions. Further, reference numeral g602 and reference numeral g612 are examples in which the order is the same but the existence (class) is not completely the same. Also, if the correct caption "Fireworks is followed by Bark, then Gunshot is ringing" while "Gunshot is followed by Bark, then Fireworks is ringing" is the output caption, the existence is the same, but the order is Is an example that does not match.

FIG. 13 is a diagram showing an example of the evaluation result.
As shown in FIG. 13, the percentage of captions that completely matched the correct answer was 73.20%. On the other hand, regardless of the order of the sound sources, 75.90% of the captions had the same type of sound source. In addition, regardless of the type of sound source, 75.75% had the same order of sound sources included in the caption. In this way, 0.15% of the cases are existence match but not order match, and the ordering itself is completed.

The causes of errors that do not match the correct answer can be divided into sound source identification errors and sound source number errors. Therefore, only the label words included in the caption were extracted to create a label string, and the number of insertion errors, the number of deletion errors, and the number of replacement errors were calculated for the label sequence. Of all 5969 evaluation data, the number of insertion errors was 40 and the number of deletion errors was 20, which were low values.

As described above, in the present embodiment, the one-dimensional acoustic signal can be treated as a two-dimensional grayscale image having one channel by performing a short-time Fourier transform.
Also, while the input image in vision was fixed length data, it can be variable length in sound. However, the width of the spectrogram of the acoustic signal changes greatly depending on its length, and when it is resized, the aspect ratio collapses significantly. Therefore, in the present embodiment, when taking the spectrogram of the acoustic signal, a plurality of fixed-length spectrograms are taken on the waveform data while shifting the window, and they are put together by the RNN on the encoder side.

As described above, in the present embodiment, in order to apply the caption generation model in the image to the model in the acoustic signal, the spectrogram representation for the acoustic signal and the fixed-length vector representation for the variable-length acoustic signal using a plurality of spectrograms are used. The model was extended by introducing it. That is, in the present embodiment, the sound is spectrogrammed into a two-dimensional image. Then, in the present embodiment, this image is input to a convolutional neural network (CNN) for learning. At this time, in the present embodiment, the spectrogram is divided (blocked) into fixed-length spectrograms, and the blocked spectrogram is input to the CNN and at the same time input to the recurrent neural network (RNN) to generate a pseudo time-series signal. Made it possible to handle.

Thereby, according to the present embodiment, it is possible to understand the acoustic scene for the acoustic signal and generate a caption.

A program for realizing all or a part of the functions of the caption generator 1 in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by the computer system. By executing it, all or a part of the processing performed by the caption generation device 1 may be performed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. In addition, the "computer system" shall also include a WWW system provided with a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, it shall include those that hold the program for a certain period of time.

Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Caption generator, 11 ... Acoustic signal acquisition unit, 12 ... Caption generation unit, 13 ... Output unit, 121 ... Preprocessing unit, 122 ... Encoder, 123 ... Decoder, 1211 ... Cutout unit, 1212 ... Normalization unit, 1213 ... Short-time Fourier transform unit, 1221 ... CNN part, 1222 ... RNN part, 1231 ... RNN part

Claims (6)

When generating a spectrogram for an acoustic signal, the spectrogram is divided into fixed lengths into one or more blocks, and the blocks are input to a convolutional neural network to extract a feature amount vector, and the extracted feature amount vector. A caption generator that generates captions for the acoustic signal by inputting to a recursive neural network.
A caption generator equipped with. The caption generator according to claim 1, wherein the spectrogram is a logarithmic mel frequency spectrogram. The recursive neural network is composed of multiple layers of long-term and short-term memory.
The recursive neural network includes a first recurrent neural network and a second recurrent neural network.
The first recursive neural network inputs the extracted feature vector into the multi-layered long-term and short-term storage layer to generate an intermediate representation vector.
The caption generation device according to claim 1 or 2, wherein the second recursive neural network inputs the vector of the intermediate representation into the multi-layered long-term and short-term storage layer to generate the caption. The caption generator according to any one of claims 1 to 3, wherein the block is a grayscale image of the spectrogram. The procedure for the acquisition unit to acquire the acoustic signal,
When the caption generator generates a spectrogram for an acoustic signal, the spectrogram is divided into fixed lengths into one or more blocks, and the blocks are input to a convolutional neural network to extract a feature vector. ,
A procedure in which the caption generation unit generates a caption for the acoustic signal by inputting the extracted feature quantity vector into the recursive neural network.
Caption generation method including. On the computer of the caption generator,
Procedures for acquiring acoustic signals and
When generating a spectrogram for an acoustic signal, the spectrogram is divided into fixed lengths into one or more blocks, and the blocks are input to a convolutional neural network to extract a feature vector.
A procedure for generating a caption for the acoustic signal by inputting the extracted feature vector into the recursive neural network, and
A program that runs.