WO2020162048A1

WO2020162048A1 - Signal conversion system, machine learning system, and signal conversion program

Info

Publication number: WO2020162048A1
Application number: PCT/JP2019/049337
Authority: WO
Inventors: 博光西▲崎▼; 直輝澤田
Original assignee: 国立大学法人山梨大学
Priority date: 2019-02-07
Filing date: 2019-12-17
Publication date: 2020-08-13
Also published as: JP7286894B2; JPWO2020162048A1

Abstract

[Problem] To provide a technology for generating data with which it is easy to identify features of temporal variation of a signal. [Solution] This signal conversion system is provided with: a signal acquisition unit for acquiring a time-variable signal in which values at multiple positions on the time axis are each represented by a single component; a conversion unit for converting each of the values at said multiple positions into a multidimensional quantity represented by a plurality of component values; an output unit for outputting said multidimensional quantity in the form of L pieces (L is an integer not less than 1 but not more than the number of components in the multidimensional quantity) of time-series data which at least include the values of identical components of the multidimensional quantity in a consecutive series of the multiple positions; and a feature output unit for outputting information concerning features of the signal on the basis of a machine-learned model which receives the L pieces of the time-series data as an input.

Description

Signal conversion system, machine learning system and signal conversion program

The present invention relates to a signal conversion system, a machine learning system, and a signal conversion program.

Conventionally, machine learning has been used for various types of classification and recognition. For example, Patent Document 1 discloses a technique of converting a numerical value indicating input information into a binary bit sequence. Specifically, a configuration is disclosed in which the numerical value of an ID indicating a word is converted into a binary bit sequence.

Japanese Patent No. 6259946

In order to accurately classify or recognize a time-varying signal or predict future signal changes, it is necessary to represent the signal in a format that can capture the characteristics of the time-varying signal in detail. Conventionally, such an expression has not been known. For example, when the amplitude of a time-varying waveform signal is sampled and expressed by a decimal value, it is difficult to directly capture a minute amplitude change or a global amplitude time change. Even if an attempt is made to obtain frequency space information by Fourier transform or the like, if a short-term signal is used in the time space, the accuracy of the frequency space information decreases. Although the accuracy of frequency space information is improved by using a long-term signal in the time space, the signal becomes unsuitable for short-term signal classification and recognition.
The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for generating data that makes it easy to capture the characteristics of the temporal change of a signal.

In order to achieve the above-mentioned object, the signal conversion system includes a signal acquisition unit that acquires a signal that is a time-varying signal and in which each of the values at a plurality of positions on the time axis is represented by one component, A conversion unit that converts each of the values for each of the plurality of positions into a multidimensional amount expressed by the values of the plurality of components, and the multidimensional amount is at least the value of the same component of the multidimensional amount at a plurality of consecutive positions. Based on an output unit that outputs L pieces (L is an integer that is 1 or more and is equal to or less than the number of components of a multidimensional quantity) of time series data, and a machine learning model that inputs L pieces of time series data, And a feature output unit that outputs information about the feature.

That is, in the signal conversion system, the conversion unit converts a signal in which the value of one position on the time axis is expressed by one component into a multidimensional amount expressed by the values of multiple components. As a result, the signal can be expressed in a format capable of capturing more various characteristics than the original signal. Then, the output unit outputs the multidimensional amount as L pieces of time-series data including at least values of the same component of the multidimensional amount at a plurality of consecutive positions on the time axis. Further, by applying the L time-series data to a machine learning model, information regarding the characteristics of the signal is output. That is, in the time-series data, the time change of the values of the same component of the multidimensional amount is shown, and the feature of the time change of the signal can be grasped for each component. With such a configuration, it is possible to generate data in which it is easy to capture the characteristics of the time change of the signal.

It is a figure showing the system configuration of one embodiment. It is a figure explaining conversion and output of data. It is a figure which shows the model of learning object. It is a flowchart of a machine learning process. It is a flowchart of a characteristic output process. It is a figure which shows the example of time series data. It is a figure which shows the model of a learning target in other embodiment.

Here, an embodiment of the present invention will be described in the following order.
(1) System configuration:
(1-1) Machine learning processing:
(1-2) Feature output processing:
(2) Other embodiments:

(1) System configuration:
FIG. 1 is a diagram showing a schematic configuration of a computer 10 that functions as a signal conversion system, a feature output system, and a machine learning system according to an embodiment of the present invention. The computer 10 includes a microphone 10a, an A/D conversion unit 10b, a control unit 20, a storage medium 30, and a display unit 40. The control unit 20 includes a CPU, a RAM, and a ROM (not shown), and can execute various programs stored in the storage medium 30 or the like. The microphone 10a, the A/D conversion unit 10b, the control unit 20, the storage medium 30, and the display unit 40 may be configured as an integrated computer, or at least a part of them may be different devices, such as a USB cable. It may be connected. The computer 10 may have various modes, for example, a stationary general-purpose computer or a portable computer such as a smartphone or a tablet terminal.

The microphone 10a collects ambient sound and outputs an analog sound wave signal indicating a temporal change in sound pressure. The A/D conversion unit 10b is a device that converts an analog signal into a digital signal, and samples the analog sound wave signal at the sampling frequency instructed by the control unit 20 and outputs it as a digital sound wave signal. In the present embodiment, the digital sound wave signal is composed of sound pressure amplitude values at a plurality of positions on the time axis, and the amplitude values are represented by decimal numbers. Therefore, the digital sound wave signal is a signal in which each of the values at a plurality of positions on the time axis is represented by one component.

The storage medium 30 is a medium capable of storing various kinds of information, and in the present embodiment, the teacher data 30a for machine learning is stored in the storage medium 30. When the machine-learned model is generated by machine learning, information indicating the model is stored in the storage medium 30 as the machine-learned model 30b. The display unit 40 is a display that displays various types of information.

In the present embodiment, the control unit 20 can execute the signal conversion program. The signal conversion program is a program that causes the control unit 20 to execute the function of converting and outputting a time-varying signal. Further, in the present embodiment, the signal conversion program has a function of performing machine learning based on the converted signal, and a function of outputting a characteristic of sound based on the converted signal.

When the signal conversion program is executed, the control unit 20 functions as the signal acquisition unit 20a, the conversion unit 20b, the output unit 20c, the machine learning unit 20d, and the feature output unit 20e. In the present embodiment, the type of sound source is output as a feature of the sound source. That is, the control unit 20 can classify the types of sound sources. Although the number of classifications may be arbitrary, here, an example of classifying whether or not the sound source is music will be described.

In this embodiment, machine learning is used to classify the types of sound sources. That is, the machine-learned model 30b is generated in advance before the classification, and the classification result is output by inputting the digital signal indicating the sound to be classified into the machine-learned model 30b. In order to perform such classification accurately, it is preferable to use a digital signal capable of capturing detailed characteristics of sound as an input value to the machine-learned model 30b.

Therefore, in this embodiment, the control unit 20 has a function of converting a digital sound wave signal indicating a sound into a multidimensional amount and outputting it as time series data capable of capturing detailed characteristics of the sound. An example of the time-series data is a data string surrounded by a dotted line in FIG. Details will be described later. In the present embodiment, the machine learning is performed using the time series data, and the sound is classified with high accuracy by performing the classification using the time series data. That is, in the present embodiment, time series data is used as input data when learning a machine learning target model, and time series data is also used as input data for performing classification based on the machine learned model 30b. Is used.

In the present embodiment, the generation of the time series data is realized by the control unit 20 executing the functions of the signal acquisition unit 20a, the conversion unit 20b, and the output unit 20c. Further, the machine learning using the time series data is realized by the control unit 20 executing the function of the machine learning unit 20d. Furthermore, the classification of sound source types using time-series data is realized by the control unit 20 executing the function of the feature output unit 20e. In the following, machine learning and sound source type classification (feature output) will be described in order.

(1-1) Machine learning processing:
In this embodiment, the teacher data 30a is prepared in advance for machine learning. The teacher data 30a is information in which a sound wave signal is associated with a sound source type of a sound indicated by the sound wave signal. For example, a set of data in which a sound wave signal indicating music is associated with music, and a set of data in which a sound wave signal indicating a sound other than music (human utterances, etc.) is associated with not music The data can be the teacher data 30a. Of course, as the teacher data 30a, a sufficient amount of data for machine learning is prepared in advance.

The teacher data 30a may be prepared by various methods, for example, data acquired by a plurality of clients may be collected by a server or the like. In this embodiment, the computer 10 can also generate the teacher data 30a. That is, the sound is collected by the microphone 10a included in the computer 10, the information indicating the type of the sound source is associated with the digital sound wave signal converted by the A/D conversion unit 10b, and is stored as the teacher data 30a in the storage medium 30. May be remembered. In any case, in the teacher data 30a, the values indicating the sound pressures at a plurality of positions on the time axis are expressed in decimal numbers.

2 is a diagram for explaining data conversion. In FIG. 2, a part of the sound wave signal Ss is shown on the graph. When the sound wave signal Ss is collected by the microphone 10 a, the A/D conversion unit 10 b converts the sound wave signal into a digital sound wave signal at the sampling frequency instructed by the control unit 20. In FIG. 2, the black circles superimposed on the sound wave signal Ss indicate the sampled amplitude, and the sampling value is shown below it. For example, in FIG. 2, the first sampling value is 0 in decimal and the second sampling value is 12 in decimal. In the teacher data 30a, information (label) indicating the type of sound source is associated with the set of values at a plurality of positions on the time axis thus obtained.

In the present embodiment, the machine learning process is a process of optimizing a training model forming a neural network, and in the present embodiment, the machine learning process is executed according to the flowchart shown in FIG. Here, the model is information indicating an equation for deriving the correspondence between the classification target data and the classification result data, and in the present embodiment, the classification result is whether or not music. The classification target is the time-series data converted and output from the sound wave signal. That is, in the present embodiment, when a digital sound wave signal (decimal number expression) that has been A/D converted by the A/D conversion unit 10b is not directly input to the model of machine learning, but is converted and output. Enter the series data.

The control unit 20 includes a signal acquisition unit 20a, a conversion unit 20b, and an output unit 20c in order to perform processing using time-series data as described above. The signal acquisition unit 20a causes the control unit 20 to perform a function of acquiring a signal that is a time-varying signal and in which each of the values at a plurality of positions on the time axis is expressed by one component. In the present embodiment, the time-varying signal is a signal in which a sound wave signal is sampled at a predetermined sampling frequency and expressed as a decimal value, and in machine learning, the teacher data 30a corresponds to the signal. That is, when the machine learning process is started, the control unit 20 refers to the storage medium 30 and acquires a sound wave signal from each teacher data 30a (step S100). As a result, as shown in FIG. 2, a plurality of data sets in which the amplitudes of the sound wave signal at a plurality of positions on the time axis are expressed in decimal are obtained.

The conversion unit 20b is a program module that causes the control unit 20 to execute a function of converting each value at each of a plurality of positions into a multidimensional amount expressed by a plurality of component values. That is, since the expression of the amplitude by the decimal number indicates the magnitude of the amplitude at a certain position on the time axis by one component, the control unit 20 expresses the sound wave signal by a larger number of components, and thus the sound wave is generated. The signal is converted into a multidimensional quantity (step S105).

In the conversion into a multidimensional quantity, it is sufficient that the expression with one component can be converted into the expression with a plurality of components. In the present embodiment, the control unit 20 determines the magnitude of the amplitude at each position on the time axis. Is converted to a binary value. That is, each digit of the converted binary number becomes a multidimensional component. In FIG. 2, the converted binary value (8 bits) is shown below the value indicating the amplitude of the sound pressure. In this example, the lower digits to the upper digits are displayed so as to be arranged from the upper side to the lower side. For example, the first sampled value of decimal 0 is binary number 00000000, and the second sampled value of decimal 12 is binary number 00001100. The time length of the actual digital sound wave signal is longer than the time length shown in FIG. 2, and for example, the digital sound wave signal is composed of 1 second of data (10000 pieces of data) sampled at 10 kHz. The digital sound wave signal is a conversion target.

The output unit 20c is a program module that causes the control unit 20 to execute a function of outputting a multidimensional amount as time-series data including at least values of the same component of the multidimensional amount at a plurality of consecutive positions. In the present embodiment, the digit of the binary value converted in step S105 is regarded as a component, and the control unit 20 extracts the value of each digit by the function of the output unit 20c to obtain the time series data. Output (step S110).

For example, in the example shown in FIG. 2, the least significant digit of the multidimensional amount represented by a binary number is surrounded by a dashed line. The control unit 20 generates data in which the respective values existing at different positions on the time axis in the least significant digit are arranged in the order on the time axis as the time series data D ₁ of the least significant digit component. As described above, in the present embodiment, the time-series data is one-dimensional data indicating the time change of the value of the same component. One-dimensional data may be implemented for multi-dimensional quantities of each component. That is, when the time series data is extracted for the upper digit of the binary number, the time series data shows a large change in the amplitude of the digital sound wave signal. On the other hand, when the time series data is extracted for the lower digit of the binary number, the time series data indicates a fine change in the amplitude of the digital sound wave signal.

Therefore, the number of components to be extracted as time-series data may be selected from an integer equal to or greater than 1 and equal to or less than the number of components of a multidimensional amount according to the purpose of machine learning or a classification target. In the present embodiment, time-series data is output for all 8-digit binary components (all digits). Therefore, FIG. 2 shows that the time series data D ₂ to D ₈ are output in addition to the time series data D ₁ .

In the present embodiment, the sound is classified based on the time series data output as described above. That is, a machine learning process for optimizing a model for inputting time series data by machine learning is performed. Machine learning may be performed by various methods, but in the present embodiment, machine learning is performed by a neural network including a convolutional neural network (CNN: Convolutional Neural Network) and a recurrent neural network (RNN: Recurrent Neural Network). An example will be described.

3 is a diagram showing an example of a model applicable to the example shown in FIG. In FIG. 3, the model is described so that the calculation proceeds from the lower part to the upper part. In the example shown in FIG. 3, the initial three layers are composed of CNN. Further, in the example shown in FIG. 3, the output result by CNN is input to RNN, and the output by RNN reaches the output layer through full coupling.

At the bottom of FIG. 3, time-series data D ₁ to D _{8 for} 8 channels which are input data are illustrated. Here, an example is shown in which digital sound wave signals obtained by sampling 10,000 positions on the time axis, that is, 10,000 times at 10 kHz are converted and output as time-series data. In the present embodiment, a calculation for convoluting the time-series data D ₁ to D _{8 in} the time direction is performed.

That is, each of the time series data D ₁ to D ₈ is one-dimensional time series data and has a total of 8 channels. Therefore, in the present embodiment, a two-dimensional (time axis direction and channel direction) filter is prepared and the convolution operation is performed. In the example shown in FIG. 3, the filter is two-dimensional, and is a filter that convolves 30×8 values (values of 30 consecutive positions on the time axis for 8 channels). In FIG. 3, the size of the filter is shown as 1×30×8. Of course, the size of the filter is not limited to 30, and may be, for example, 50, 40, 20 or 10, or may be in a range between any two of these exemplified numerical values. The shorter the filter size, the earlier the processing after signal acquisition can be started. Further, in FIG. 3, the input data is eight channels, but it is not limited to eight. Although the number of filters is not limited, the number of filters in the first layer of CNN is 128 in the example shown in FIG. In FIG. 3, these filters are shown as F _1-1 to F _1-128 . In the present embodiment, since a filter having a short size in the time axis direction can be used as described above, it is possible to capture a minute change in time series data as compared with the conventional technique. In the prior art, when the amplitude of a time-varying waveform signal is sampled and expressed as a decimal value, a filter having a short size in the time axis direction directly causes a minute amplitude change or a global amplitude time change. Is difficult to capture. In the Fourier transform or the like, if a short-term signal is used in the time space, the accuracy of the information in the frequency space will decrease. That is, more time is required from the acquisition of the signal to the output of the feature.

Each filter is applied to each of the time series data D ₁ to D ₈ with a predetermined padding or stride. For example, if the stride 10 is appropriately padded, the 10,000 data will be 1,000. The output result of one filter is obtained by adding these. Therefore, when the number of filters is 128, 128 one-dimensional data having 1000 data are generated. In FIG. 3, the 1×1000×128 data is indicated by a rectangle. Of course, hyperparameters such as stride and padding are examples, and the number of output data (1000 pieces, etc.) is also an example (same for other layers).

In any case, when 1×1000×128 data is output by the calculation by the first layer of CNN, this data becomes the input data to the next layer of CNN. FIG. 3 shows an example in which 256 filters of size 1×30×128 are prepared in the second layer, and 512 filters of size 1×30×256 are further prepared in the third layer. ing. Stride and padding are similar to the first layer. In this example, 1×10×512 pieces of data D _CNN are output by the above-described calculation of the three layers.

The data D _CNN becomes an input to the RNN. Although the element forming the RNN shown in FIG. 3 is an LSTM (Long Short-Term Memory), the element may be a GRU (Gated Recurrent Unit) or a bidirectional model may be adopted. A configuration can be adopted. Of course, the hyperparameter may be various parameters.

In any case, in the example shown in FIG. 3, since the data D _CNN input to the RNN is 1×10×512 data, 512 data are input 10 times to the LSTM in this example. Calculation is performed. In FIG. 3, the first 512 input values are input as X _1-1 to X _1-512 , the next 512 input values are input as X _2-1 to X _2-512 , and so on. Is shown.

Furthermore, in the LSTM shown in FIG. 3, it is assumed that the LSTM outputs _{1024, and the} data Y ₁ to Y ₁₀₂₄ output from the last LSTM are input to the next layer. That is, the data Y ₁ to Y ₁₀₂₄ are input to the fully connected layer, and the classification result indicating whether or not the layer is the music next to the fully connected layer is output.

In the present embodiment, the control unit 20 executes the machine learning process based on the above model by the function of the machine learning unit 20d. That is, the control unit 20 machine-learns a machine-learned model that receives time-series data and outputs information related to the characteristics of the sound source of the signal by the function of the machine learning unit 20d. Specifically, the control unit 20 acquires the training model by the function of the machine learning unit 20d (step S115). That is, various kinds of information corresponding to the structure of the neural network as shown in FIG. 3 (information such as a filter indicating a model and an activation function) is defined in advance, and the control unit 20 acquires the information. Get the training model.

Next, the control unit 20 acquires the label of the teacher data 30a by the function of the machine learning unit 20d (step S120). That is, the control unit 20 acquires the label indicating the type of the sound source of each sound wave signal acquired in step S100. For example, in the example shown in FIG. 3, the label indicates whether the output value of the node in the output layer is 1, 0, and when the sound source of the sound wave signal acquired in step S100 is music. 1, 0 is acquired when it is a sound other than music.

Next, the control unit 20 acquires test data by the function of the machine learning unit 20d (step S125). In the present embodiment, the control unit 20 extracts a part of the data acquired in step S110, and associates the label acquired in step S120 with it to make it test data. The test data is data for confirming whether or not learning generalization has been performed, and is not used for machine learning.

Next, the control unit 20 determines the initial value by the function of the machine learning unit 20d (step S130). That is, the control unit 20 gives initial values to the variable parameters (filter weight, bias, etc.) to be learned in the training model acquired in step S115. The initial value may be determined by various methods. Of course, the initial values may be adjusted so that the parameters are optimized during the learning process, or the learned parameters may be acquired from various databases and used.

Next, the control unit 20 performs learning by the function of the machine learning unit 20d (step S135). That is, the control unit 20 inputs the time-series data output in step S110 to the training model acquired in step S115, and outputs information indicating the classification result. When the information indicating the classification result is output, the control unit 20 specifies the error by the loss function indicating the error between the output and the label acquired in step S120. When the loss function is obtained, the control unit 20 updates the parameter by a predetermined optimization algorithm, for example, the stochastic gradient descent method. That is, the control unit 20 repeats the process of updating the parameter based on the differentiation of the loss function by the parameter for a predetermined number of times.

When the parameters have been updated the predetermined number of times as described above, the control unit 20 determines whether or not the generalization of the training model has been completed (step S140). That is, the control unit 20 inputs the test data acquired in step S125 into the training model and outputs the sound wave signal classification result. Then, the control unit 20 obtains the number of coincidences between the output classification result and the classification result associated with the test data (the number of errors between the classification result and the label is equal to or less than the default value), The classification accuracy is obtained by dividing by the number of samples. In the present embodiment, the control unit 20 determines that the generalization is completed when the classification accuracy is equal to or higher than the threshold value.

-In addition to the evaluation of generalization performance, the validity of hyperparameters may be verified. That is, in a configuration in which a hyperparameter that is a variable amount other than the variable parameter to be learned, such as the filter size and the number of nodes, is tuned, the control unit 20 determines the validity of the hyperparameter based on the verification data. You may verify. The verification data may be acquired by preliminarily extracting the verification data by the same process as in step S125 and securing it as data not used for training.

If it is not determined in step S140 that the generalization of the training model is completed, the control unit 20 repeats step S135. That is, the process of further updating the variable parameter to be learned is performed. On the other hand, when it is determined in step S140 that the generalization of the training model is completed, the control unit 20 records the machine-learned model (step S145). That is, the control unit 20 records the training model in the storage medium 30 as the machine-learned model 30b.

(1-2) Feature output processing:
Next, the characteristic output processing for classifying the types of sound sources will be described based on the flowchart shown in FIG. While sound is being output from the sound source to be classified in the vicinity of the computer 10, the user of the computer 10 operates an input unit (not shown) such as a keyboard and a mouse to give an instruction to start execution of the characteristic output process. When the user gives the execution start instruction, the control unit 20 starts execution of the characteristic output process.

When the execution of the characteristic output process is started, the control unit 20 acquires a sound wave signal (step S200). That is, when a sound wave is output from the sound source, the microphone 10a acquires the sound wave and outputs an analog sound wave signal. The control unit 20 preliminarily instructs the A/D conversion unit 10b to have the same sampling frequency as that used for machine learning. The A/D converter 10b samples the analog sound wave signal at the sampling frequency and converts it into a digital sound wave signal. Then, when the digital sound wave signal is output from the A/D conversion unit 10b, the control unit 20 acquires the digital sound wave signal by the function of the signal acquisition unit 20a. For example, in the case of the sound wave signal Ss shown in FIG. 2, since the A/D conversion unit 10b outputs a decimal sound wave signal (0, 12, 6,,... ), the control unit 20 controls the digital sound signal Ss. Acquire sound wave signals.

Next, the control unit 20 converts the sound wave signal into a multidimensional amount by the function of the A/D conversion unit 10b (step S205). That is, the control unit 20 acquires the digital sound wave signal acquired in step S200, and converts the decimal value indicating the amplitude of each of a plurality of positions on the time axis into a binary number. For example, in step S200, when the digital sound wave signals (0, 12, 6,,,) shown in FIG. 2 are acquired, the binary multidimensional quantity (000000000001100,,,) shown in FIG. 2 is obtained. To be acquired.

Next, the control unit 20 outputs the time series data by the function of the output unit 20c (step S210). That is, the control unit 20 extracts a value for each digit of the multidimensional amount and generates time-series data of each digit. For example, in the example shown in FIG. 2, the control unit 20 acquires the time series data D ₁ to D ₈ .

Next, the control unit 20 determines the feature of the sound by the function of the feature output unit 20e (step S215). That is, the control unit 20 acquires the machine-learned model 30b and inputs the time-series data D ₁ to D ₈ output in step S210 to the machine-learned model 30b. The control unit 20 uses the parameters indicated by the machine-learned model 30b to calculate CNN, RNN, etc. shown in FIG. As a result, the value of the output layer is determined. In the examples shown in FIGS. 2 and 3, the time-series data D ₁ to D ₈ which are _eight- dimensional one-dimensional data are input to the machine-learned model 30b, but the number of channels is not limited to eight. It may be an integer greater than or equal to 1 and less than or equal to the number of multidimensional components.

The control unit 20 compares the value of the output layer with the threshold value, and determines that the sound source type is music if the threshold value or more, and determines that the sound source type is not music if it is smaller than the threshold value. For example, when the value of the output layer changes in the range of 0 to 1 and the threshold value is 0.9 and the value of the output layer is 0.9 or more, the control unit 20 determines that the sound source type is music. To consider.

Next, the control unit 20 outputs the determination result by the function of the feature output unit 20e (step S220). That is, the control unit 20 controls the display unit 40 to display information indicating the type of sound source. With the above configuration, the characteristic of the sound can be determined based on the result of machine learning by converting the amplitude represented by the decimal number into the multidimensional amount represented by the value of a larger number of components. Therefore, it is possible to determine the characteristics of the sound by performing machine learning that captures more various characteristics than the original signal represented by a decimal number, and it is possible to determine the characteristics of the sound with high accuracy.

Further, in the present embodiment, the multidimensional amount is decomposed for each same component (each same digit), and machine learning and feature acquisition are performed using time-series data showing a time change of the value of the same component. As a result, the time change of the sound wave signal including both the global time change and the minute time change can be captured in detail. Therefore, it is possible to determine the characteristics of the sound by performing machine learning that captures various temporal changes more than the original signal represented by a decimal number, and it is possible to determine the characteristics of the sound with high accuracy. When the minute time change is noise and the signal features appear in the portion excluding the minute time change, learning is performed so that the weight given to the classification by the minute time change becomes small. Therefore, the characteristics of the sound can be determined with high accuracy even if noise is included.

Here, the classification accuracy is compared between the case where environmental sound event classification is performed using this embodiment and the case where environmental sound event classification is performed using a known method. For environmental sounds, we used a database containing 28 types of events listed in Table 2 of the reference. Based on this database, as described in the reference document, three channels of the power spectrum (256 dimensions) and their variations Δ and ΔΔ are input to the classification model, and an example of classifying the environmental sound event is made proportional. The classification model in this case is a model in which the result of passing through 6 layers of CNN is passed through through 3 layers of fully connected layers. The classification accuracy when machine learning was performed using the classification model was 80.3%.

On the other hand, when machine learning was performed based on this database based on FIG. 3 of the present embodiment, the classification accuracy was 88.3%. As described above, according to this embodiment, the classification accuracy is significantly improved as compared with the known method. (Reference) N.Takahashi, et al., "Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detection," Proc. of INTERSPEECH2016, pp.2982-2986, 2016

(2) Other embodiments:
The above embodiment is an example for carrying out the present invention, in which the value at each position on the time axis of a signal that changes with time is represented by a multidimensional amount, and a time series indicating the time change for each component of the multidimensional amount. Various other embodiments can be adopted as long as data is output. For example, although the signal conversion system, the machine learning system, and the feature output system are configured by one computer 10, they may be different systems, or two systems are configured as one system. May be done.

Needless to say, the method of linking separate systems may be various methods, and may be configured as a client and a server. Furthermore, at least a part of the signal acquisition unit 20a, the conversion unit 20b, the output unit 20c, the machine learning unit 20d, and the feature output unit 20e may be present separately in a plurality of devices. For example, the configuration may be such that the process of acquiring the teacher data 30a by the signal acquisition unit 20a and the process of acquiring the classification target digital sound wave signal are performed by different devices. Of course, a part of the configuration of the above-described embodiment may be omitted, or the order of processing may be changed or omitted.

For example, a system may be configured in which the information obtained in the process of processing by the machine learning model shown in the above embodiment is output. Specifically, the data D _CNN of CNN is obtained in the course of the model shown in FIG. 3, the data D _CNN may be output as the information indicating the characteristics of the sound. Since the information indicates the feature of the sound, if the pattern of the data D _CNN in the case of the specific sound type is specified in advance, the sound type can be classified. Of course, the model for machine learning is not limited to the example shown in FIG. 3. For example, the RNN is omitted, the CNN data D _CNN is input to the fully connected layer, and the information indicating the classification result is output in the subsequent layers. It may be configured to be.

The mode of machine learning is not limited. For example, when machine learning is performed by a neural network, the number of layers and nodes in the model, the type of activation function, the type of loss function, the type of gradient descent method, the gradient Type of optimization algorithm for descent method, presence/absence of mini-batch learning, number of batches, learning rate, initial value, type of presence/absence of overlearning suppression method, presence/absence of convolutional layer, filter size in convolution operation, filter type, padding Machine learning may be performed by appropriately selecting various elements such as the type of a stride, the type of stride, the presence or absence of a pooling layer, the presence or absence of a fully connected layer, and the presence or absence of a recursive structure. Of course, learning may be performed by other machine learning, for example, deep learning, support vector machine, clustering, reinforcement learning, or the like. Furthermore, machine learning may be performed in which the structure of the model (for example, the number of layers or the number of nodes for each layer) is automatically optimized.

The classification mode is not limited to music and non-music, and may be classification of music and voice, or classification of more kinds of sound sources. Of course, the uses of the signal conversion system, the machine learning system, and the feature output system are not limited to the classification of sound source types. For example, it may be used for abnormal sound detection such as tap sound inspection, may be used for detailed classification of music (genre classification, song name classification, etc.), may be used for voice recognition, and may be used for sound recognition. May be used to classify related events (classification of types of environmental sounds). The use scene is not limited, and may be used for statistical management of broadcast contents by, for example, classifying broadcast audio. The present invention can also be applied to sound source separation. For example, a technique is known in which a sound source in which voice is superimposed on music is input to a neural network to separate only music or only voice. However, by inputting the time-series data of the present invention to the neural network, the time of the signal is Since the characteristics of changes can be easily captured, highly accurate and highly efficient separation can be realized.

The signal acquisition unit should be able to acquire a signal that is a time-varying signal and in which each of the values at a plurality of positions on the time axis is represented by one component. That is, the signal acquisition unit acquires a signal in which the value at each position on the time axis is represented by a single component. The component indicates the feature of the signal, and if the component is one, the signal is represented by one feature. Therefore, as in the above-described embodiment, the amplitude of the sound wave signal is sampled at regular intervals, and a signal having one value (scalar amount) at one location on the time axis is represented by one component. It is a signal that has been processed. In addition, other various modes can be adopted as the signal represented by one component.

Also, the time-varying signal is not limited to a sound wave, as long as the signal can have different values at different positions on the time axis. For example, a biological signal such as an ultrasonic wave, an electroencephalogram, an electrocardiogram, an electromyogram, or an arbitrary wave such as an environmental signal such as temperature, humidity, or atmospheric pressure can be a time-varying signal. Furthermore, when a target of interest changes over time, such as a stock price or an exchange chart, the information indicating the change over time of the target may correspond to a signal that changes over time. Furthermore, it is possible to adapt to image signals such as moving images. In the image signal of the moving image, the moving image is represented by the time-series change of a plurality of images. In each image, in the case of a color image, the color of one pixel is expressed by three channels such as red, green, and blue, and in the case of a gray image, the brightness of one pixel is expressed by one channel. Has a gradation value (usually a value of 0 to 255). Therefore, it is also possible to adopt a configuration in which the gradation values of these pixels are converted into a multidimensional amount and the time-series data indicating the values of the respective converted components are output. Of course, the features output in the embodiment of the present invention may be used for various purposes other than classification and recognition, and may be used for prediction of future signal changes, for example. As described above, the feature of the signal that changes with time is not limited to the configuration in which the type of the sound source is used, and various features of the signal may be output by the feature output unit.

-The position on the time axis may be a discrete position, and the interval may be any interval. For example, when a waveform signal is sampled to acquire a signal, the sampling cycle is arbitrary, and the sampling cycle may be fixed or may change with time.

The conversion unit only needs to be able to convert each of the values at each of the multiple positions into a multidimensional amount represented by the values of multiple components. That is, the conversion unit converts one value at each position on the time axis into a plurality of values at each position on the time axis. The conversion into the multidimensional amount may be performed by various methods. That is, various configurations may be adopted other than the configuration in which one value at each position on the time axis is converted into a binary number and each digit of the converted value becomes each component of the multidimensional amount.

For example, if a graph showing a signal waveform is regarded as a two-dimensional image and the image is binarized, data at a position where a signal wave exists and a position where a signal wave does not exist are obtained. FIG. 6 shows an example in which a signal similar to the sound wave signal Ss shown in FIG. 2 is represented by a graph having a time axis and an amplitude axis. In addition, FIG. 6 shows an example in which this graph is regarded as an image and binarized by setting a portion having an amplitude value at a plurality of positions in the time axis direction as 1 and a portion having no amplitude value as 0. Even with such a conversion, the value at each position on the time axis of the time-varying signal is represented by a multidimensional amount, and various types of conversion to a multidimensional amount including this example are possible. May be adopted.

Needless to say, the number of digits for expressing the value of each component of a multidimensional amount is not limited to 1. For example, the value of a plurality of binary digits may be the value of one component. Furthermore, the method of interpreting one component before conversion may be performed by various methods. For example, when one component before conversion is one decimal value and it spans multiple digits, this value is decomposed into values for each digit and then n digits (n is an integer of 2 or more) for each digit. The value may be converted into a value, and the converted value may be the value of each component of the multidimensional amount, or each digit of the converted value may be the value of each component of the multidimensional amount.

As a more specific example, for example, when the value of one component before conversion is the decimal number "806", each of "8", "0", and "6" is converted into the binary number "1000". ", "0000", "0110" may be set, and "1000", "0000", and "0110" may be used as the respective values of the three components of the multidimensional amount. Further, each digit of “100000000110” may be set as each value of 12 components of the multidimensional amount.

Multi-dimensional quantity may be a value expressed by multiple components and may be determined by various methods. Also, the relationship for each component may be various relationships. For example, as in the above-described embodiment, each digit of the binary number expression may be a component, or each component may be an amount expressed so as to be linearly independent.

The output unit is only required to be able to output the multidimensional amount as time series data including at least the values of the same component of the multidimensional amount at a plurality of consecutive positions. That is, the time-series data need only show how the values of the same multidimensional component change over time. The number of positions on the time axis (time length indicated by the time series data) represented by the time series data may be any number as long as it is plural. The number may be selected.

Further, since the time-series data only needs to show how the values of the same component of a multidimensional amount change with time, the number of components is an integer in the range of 1 or more and the number of components of the multidimensional amount or less. Is. Here again, the number of components may be selected depending on the use of the time series data.

-If there are multiple multidimensional components that are time-series data, the time-series data will be two-dimensional data. In this case, for example, from the multidimensional amount, the values of different components at the same position on the time axis are arranged in the first axis direction, and the values of the same component at different positions on the time axis are different from the first axis direction. The configuration may be one in which two-dimensional data arranged in two axial directions is extracted.

In the case of the example shown in FIG. 2, two-dimensional data including all 8-digit binary digits shown in the lower left may be extracted. That is, in the multidimensional amount shown in FIG. 2, the values of different components (digits) at the same position on the time axis are lined up when viewed in the vertical direction of the figure, so the vertical direction can be regarded as the first axis. Furthermore, when viewed in the left-right direction in the figure, the values of the same component are arranged at different positions on the time axis, so the left-right direction (time direction) can be regarded as the second axis.

Even in the case of such two-dimensional data, when focusing on each component, since the values of the same component of a multidimensional amount at a plurality of consecutive positions are included, such data is output by the output unit 20c. Embodiments may be configured. In this case, the machine-learned model 30b including the convolutional neural network that inputs the two-dimensional data is machine-learned. Further, based on the machine-learned model 30b including the convolutional neural network that inputs the two-dimensional data, information regarding the characteristics of the sound source of the signal is output.

Such a configuration is realized by changing the structure of the machine learning model and the processes of the machine learning unit 20d and the feature output unit 20e in the example shown in FIG. FIG. 7 is an example of a machine learning model for inputting two-dimensional data. In FIG. 7, the model is described so that the calculation proceeds from the lower part to the upper part. In the example shown in FIG. 7, the initial three layers are composed of CNN. Further, in the example shown in FIG. 7, the output result by CNN is input to RNN, and the output by RNN reaches the output layer through full coupling.

Two-dimensional data D _2D that is input data is illustrated at the bottom of FIG. 7. Here, the number of positions on the time axis is arbitrary. For example, when the digital sound wave signal obtained by sampling 10000 times at 10 kHz is converted and data about 10000 positions on the time axis is obtained, the data can be the two-dimensional data D _2D. ..

In this example, an operation of convolving the information in the time direction and the component direction (digit direction) of the two-dimensional data D _2D is performed. That is, a two-dimensional filter is prepared and applied to the two-dimensional data D _2D to perform the convolution operation. In the example shown in FIG. 7, the size of the filter is represented by a cube having a size x in the time direction, a size y in the component direction, and a size ch in the channel direction. That is, the size of the filter in the CNN of the first layer is the size x ₁ in the time direction, the size y _{1 in} the component direction, the size _{1 in} the channel direction, and the size of the filter in the CNN of the second layer is the size in the time direction. Size x ₂ , component direction size y ₂ , channel direction size 128, filter size in CNN of the third layer is time direction size x ₃ , component direction size y ₃ , channel direction size The size is 256.

In the example shown in FIG. 7, the number of filters in the CNN of the first layer is 128, the number of filters in the CNN of the second layer is 256, and the number of filters in the CNN of the third layer is 512. Of course, the size of the filters and the number of filters can be adjusted appropriately. In any case, in the calculation of each filter, if the padding or stride is predetermined as a hyper parameter and the convolution calculation is performed, the calculation of the neural network is sequentially performed. For example, when the convolution operation of the CNN of the first layer is performed, the output becomes i ₁ ×j ₁ ×128 pieces of data, which becomes the input of the CNN of the second layer.

When such CNN calculation is repeated for the second and third layers, the output of the second layer becomes i ₂ ×j ₂ ×256 data, and the output of the third layer is i ₃ ×j ₃ ×512 data. Data D _CNN of Also in the model shown in FIG. 7, the data D _CNN becomes an input to the RNN. Although the element forming the RNN shown in FIG. 7 is a BiGRU (Bidirectional Gated Recurrent Unit), the element may be an LSTM or the like, and various configurations can be adopted. Of course, the hyperparameter may be various parameters. In any case, in the example shown in FIG. 7, the data D _CNN input to the RNN is input to the fully connected layer and the feature output at the layer next to the fully connected layer, for example, whether it is music or not. The classification result may be output.

When the machine learning process shown in FIG. 4 is executed by the configuration shown in FIG. 1 in a state in which the machine learning model is predetermined, a machine learned model 30b that outputs sound characteristics with two-dimensional data as an input is obtained. To be However, the data output as time series data in step S110 shown in FIG. 4 is two-dimensional data. In this state, the control unit 20 acquires information indicating a model as shown in FIG. 7 as a training model in step S115. Then, if the control unit 20 performs the learning in step S135 through steps S120 to S130, the machine-learned model 30b that receives the two-dimensional data as an input and outputs the sound feature is obtained.

The same applies to the feature output processing shown in FIG. 5, and the time-series data acquired in step S210 becomes two-dimensional data. Then, in step S215, when the control unit 20 inputs the two-dimensional data into the machine-learned model 30b, the characteristic of the sound is determined. Even with the above-described two-dimensional data, the value of each position on the time axis is decomposed into a plurality of components and includes the time change of each component. Therefore, by performing the machine learning and the feature determination based on the two-dimensional data, it is possible to determine the feature of the sound by performing the machine learning that captures various temporal changes, and to accurately determine the feature of the sound. Can be judged.

Note that the output by the output unit may be performed for the same device or different devices. That is, as in the above-described embodiment, the output unit may be configured to output the result of converting the multidimensional amount into time-series data and further perform the calculation, or the output unit may be another device (for example, an external device). Output to a server or the like connected to the server).

Further, as in the present invention, the method of expressing the value at each position on the time axis of a time-varying signal in a multidimensional amount and outputting time-series data indicating the time change for each component of the multidimensional amount is a program. It can also be applied as a method. The system, program, and method described above can be realized as a single device or a plurality of devices, and include various aspects. Further, it is possible to change as appropriate, such as a part being software and a part being hardware. Further, the invention can be realized as a recording medium of a program for controlling the system. Of course, the recording medium of the software may be a magnetic recording medium or a semiconductor memory, and any recording medium developed in the future can be considered in exactly the same way.

As described above, in the above example, the signal conversion system is configured as a system including at least the signal acquisition unit 20a, the conversion unit 20b, and the output unit 20c. In addition, the signal conversion system is provided with a characteristic output unit 20e that outputs information related to the characteristics of the input signal based on the machine-learned model 30b, and the characteristic output system is also provided to the signal conversion system. The machine learning system is configured so as to further include a machine learning unit 20d that performs machine learning on the machine learned model 30b that outputs information regarding the characteristics of. The signal conversion system according to this embodiment may be executable as a signal conversion program as described above.

10... Computer, 10a... Microphone, 10b... A/D conversion part, 20... Control part, 20a... Signal acquisition part, 20b... Conversion part, 20c... Output part, 20d... Machine learning part, 20e... Feature output part, 30 ... storage medium, 30a ... teacher data, 30b ... machine-learned model, 40 ... display unit

Claims

A signal acquisition unit that acquires the signal, which is a time-varying signal and in which each of the values at a plurality of positions on the time axis is represented by one component;
A conversion unit that converts each of the plurality of values for each position into a multidimensional amount represented by a plurality of component values,
A time series of L (L is an integer not less than 1 and not more than the number of components of the multidimensional amount) including at least values of the same component of the multidimensional amount at a plurality of consecutive positions, the multidimensional amount. An output unit that outputs as data,
A feature output unit that outputs information about the feature of the signal based on a machine learning model that inputs the L time-series data;
A signal conversion system including.
The machine learning model is
Including a convolutional neural network,
The signal conversion system according to claim 1.
The conversion unit is
Each of the plurality of values for each position is converted into an n-ary number (n is an integer of 2 or more), and each digit of the converted value is used as a component of the multidimensional amount.
The signal conversion system according to claim 1.
n is 2,
The signal conversion system according to claim 3.
The signal indicates sound,
The signal conversion system according to any one of claims 1 to 4.
The time series data is
From the multidimensional quantity, L pieces of one-dimensional data (L is an integer not less than 1 and not more than the number of components of the multidimensional quantity) indicating one-dimensional changes in the value of the same component are data extracted.
The signal conversion system according to any one of claims 1 to 5.
The time series data is
From the multidimensional quantity,
Values of different components at the same position on the time axis are arranged in the first axis direction,
Two-dimensional data in which the values of the same component at different positions on the time axis are arranged in a second axis direction different from the first axis direction is extracted data.
The signal conversion system according to any one of claims 1 to 5.
A feature output unit that outputs information about the feature of the sound source of the signal based on a machine-learned model including a convolutional neural network that inputs the two-dimensional data.
The signal conversion system according to claim 7.
A signal conversion system of an input signal to a neural network used in a machine learning system,
A signal acquisition unit that acquires the signal, which is a time-varying signal and in which each of the values at a plurality of positions on the time axis is represented by one component;
A conversion unit that converts each of the plurality of values for each position into a binary value, and converts into a multidimensional amount in which each digit of the converted value is a component;
From the multi-dimensional amount, L pieces of one-dimensional data (L is 1 or more and the number of components of the multi-dimensional amount or less, which indicates the time change of the value of the same component of the multi-dimensional amount at a plurality of consecutive positions, An integer) an output part that extracts and outputs,
A feature output unit that outputs information related to the feature of the signal based on a machine learning model that inputs L pieces of the one-dimensional data;
A signal conversion system comprising:
A signal conversion system according to any one of claims 1 to 7,
A machine learning unit that inputs the time-series data and machine-learns a machine-learned model that outputs information regarding the characteristics of the signal,
A machine learning system.
Computer,
A signal acquisition unit that acquires the signal, which is a time-varying signal and in which each of the values at a plurality of positions on the time axis is represented by one component,
A conversion unit that converts each of the plurality of values for each position into a multidimensional amount represented by a plurality of component values,
A time series of L (L is an integer not less than 1 and not more than the number of components of the multidimensional amount) including at least values of the same component of the multidimensional amount at a plurality of consecutive positions, the multidimensional amount. Output section that outputs as data,
A feature output unit that outputs information regarding the feature of the signal based on a machine learning model that inputs the L time-series data,
Signal conversion program to function as.