JP6783443B2 - Information processing equipment, information processing systems, information processing methods, programs, and recording media - Google Patents

Description

The present invention relates to an information processing device, an information processing system, an information processing method, a program, and a recording medium.

Conventionally, a clustering method for classifying a large amount of data into characteristic categories has been known. As a recent clustering method, the Affinity Propagation (AP) method is known (see, for example, Non-Patent Documents 1 and 2).

In the AP method, the parameter r (i, k) is a score indicating the appropriateness of the data point i in the cluster being assigned to the candidate k of the center point (exemplar) of the cluster, and is sent from the data point i to the candidate k of the exemplar. Is considered to be the message to be sent. The parameter a (i, k) is a score indicating the appropriateness of the data point i to belong as a member in the cluster of the exemplar candidate k, and the data point i expected to become a cluster member from the exemplar candidate k. Is considered to be the message sent by. In other words, the Associated Press method is a message-switched clustering method.

Brendan J. Frey and Delbert Dueck, "Clustering by Passing Messages Between Data Points", University of Toronto Science 315, 972-976, February 2007 Inmar E. Givoni and Brendan J. Frey, "A Binary Variable Model for Affinity Propagation", Neural Computation, Vol 21, issue 6, pp 1589-1600, June 2009

However, in the above-mentioned conventional technique, there is a possibility that the clustered cluster contains outliers different from the representative values in the class, and there is room for improvement in the classification accuracy. Here, when the AP method is applied using environmental sounds as an example of data, different sounds may be classified into the same cluster depending on the representative sound in the cluster and the sound in the same cluster. In this way, the AP method is required to improve the classification accuracy.

Therefore, the present invention provides an information processing device, an information processing system, an information processing method, a program, and a recording medium capable of improving the classification accuracy into clusters with respect to the AP method.

The information processing device according to one aspect of the present invention includes an acquisition unit that acquires a plurality of data and
An extraction unit that extracts the features of each data,
It is a classification unit that generates a final cluster by repeatedly updating each of the first evaluation value and the second evaluation value based on the similarity matrix between each data generated using the extracted feature amount. The autocorrelation of the similarity matrix is set using the similarity regarding the deviation data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value, and the first evaluation value is the cluster center. The candidate data is a value indicating the appropriateness of the member data belonging to the cluster to be the cluster center, and the second evaluation value indicates the appropriateness of the member data belonging to the cluster of the candidate data. It includes the classification unit, which is a value.

It is a conceptual diagram which shows an example of the information processing system in 1st Embodiment. It is a block diagram which shows an example of the hardware composition of the information processing apparatus in 1st Embodiment. It is a block diagram which shows an example of the functional structure of the information processing apparatus in 1st Embodiment. It is a figure which shows the place where the environmental sound was measured. It is a figure which shows an example of a microphone array. It is a figure which shows the position of a microphone. It is a figure which shows an example of a power spectrum. It is a figure which shows an example of the basis vector. It is a figure which shows an example of a coefficient vector. It is a figure which shows an example of the power spectrum reconstructed using the basis vector and the coefficient vector. It is a figure which shows an example of the result of clustering sound data using the improved AP method. It is a figure which shows an example of the accuracy result (the 1) of the improved AP method. It is a figure which shows an example of the accuracy result (the 2) of the improved AP method. It is a figure which shows an example of the generation rate of the environmental sound of one day. It is a figure which shows an example of the environmental sound generation rate in the event A from 10:00 to 13:00 on a certain event day. It is a figure which shows an example of the environmental sound generation ratio in the event A from 14:00 to 18:00 on a certain event day. It is a figure which shows an example of the environmental sound generation ratio in the event B from 14:00 to 18:00 on a certain event day. It is a figure which shows an example of the environmental sound generation ratio in the event C from 10:00 to 13:00 on a certain event day. It is a figure which shows an example of the environmental sound generation ratio in the event C from 14:00 to 18:00 on a certain event day. It is a flowchart which shows an example of the data analysis processing in 1st Embodiment. It is a flowchart which shows an example of the classification process in 1st Embodiment. It is a flowchart which shows an example of the tagging process in 1st Embodiment. It is a flowchart which shows an example of the analysis processing in 1st Embodiment. It is a block diagram which shows an example of the functional structure of the information processing apparatus in 2nd Embodiment. It is a figure which shows an example of a power spectrum. It is a figure which shows an example of the power spectrum of edge detection. It is a figure which shows an example of each of the classified categories. It is a figure for demonstrating the calculation of the loudness of a sound. It is a figure for demonstrating the calculation of the continuity of a sound. It is a figure which shows the predetermined area which expresses a sound pattern in an experimental place. It is a figure which shows the sound pattern in normal times. It is a figure which shows the sound pattern at the time of an event. It is a flowchart which shows an example of the sound pattern expression processing in 2nd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention of excluding the application of various modifications and techniques not specified below. That is, the present invention can be implemented in various modifications without departing from the spirit of the present invention. Further, in the description of the following drawings, the same or similar parts are represented by the same or similar reference numerals. The drawings are schematic and do not necessarily match the actual dimensions and ratios. Even between drawings, parts with different dimensional relationships and ratios may be included.

[First Embodiment]
Hereinafter, the information processing system according to the first embodiment of the present invention will be described. The information processing system according to the first embodiment can appropriately classify the acquired plurality of data. In addition, the information processing system can use appropriately classified clusters and data to build a model suitable for the data and analyze the data.

<System overview>
FIG. 1 is a conceptual diagram showing an example of an information processing system according to the first embodiment. As shown in FIG. 1, for example, in the information processing system 1, information processing devices (server devices) 10A and 10B for processing data and acquisition devices 20A, 20B, and 20C for acquiring data are connected via a network N. To. The information processing devices 10A and 10B may distribute the processing, or may be processed by any one of the devices.

Each information processing apparatus uses reference numerals 10A and 10B when they are individually distinguished and described, and reference numerals 10 are used when they are described collectively without needing to be individually distinguished. The usage of the code of the acquisition device is the same as that of the information processing device.

The acquisition device 20 acquires the data to be classified. For example, if the data is sound data, the acquisition device 20 is a microphone or microphone array capable of collecting sound, and if the data is image data, the acquisition device 20 is an image pickup device such as a camera, and the data is If it is character data, it is a device that acquires character data.

The information processing device 10 acquires a plurality of data from the acquisition device 20 and performs classification processing, so-called clustering, on the plurality of data. In addition, the information processing apparatus 10 may use the result of clustering to perform labeling or build a model of target data. Further, the information processing apparatus 10 may perform various analyzes of the target data using this model. The information processing device 10 may perform clustering, model construction, and various analyzes on the data stored in its own storage unit.

The network N is the Internet or the like, and includes a wireless LAN access point, a mobile phone base station, and the like.

<Hardware configuration>
Next, the hardware configuration of the information processing device 10 will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing apparatus 10 according to the first embodiment.

As shown in FIG. 2, the information processing device 10 includes a control unit 102, a communication interface 106, a storage unit 108, a display unit 114, and an input unit 116, and each unit is connected via a bus line. Will be done.

The control unit 102 includes a CPU, a ROM, a RAM 104, and the like. By executing the processing program 110 or the like stored in the storage unit 108, the control unit 102 realizes, for example, a classification function, a labeling function, a model construction function, and the like, in addition to the function as a general server device. It is configured as follows.

Further, the RAM 104 temporarily holds various information and is used as a work area when the CPU executes various processes.

The communication interface 106 controls communication with the acquisition device 20 via the network N.

The storage unit 108 is composed of, for example, an HDD or the like, and stores the processing program 110 in addition to storing applications and data (not shown) for realizing a function as a general information processing device. Further, the storage unit 108 has an information storage unit 112.

The processing program 110 is a program for performing processing such as classification, and is a program that acquires each data from the acquisition device 20 and processes each data. The processing program 110 may be stored in a computer-readable recording medium, read from the recording medium, and stored in the storage unit 108.

The information storage unit 112 stores, for example, data acquired from each acquisition device 20, information on classification, information on labels, information on models, and the like in association with each other.

The display unit 114 displays information to the administrator of the information processing device 10. The input unit 116 accepts an input from an administrator or the like, or receives an instruction from the administrator or the like. Further, the information processing device 10 does not necessarily have to be provided with the display unit 114 and the input unit 116, and the display unit 114 and the input unit 116 may be connected to the information processing device 10 from the outside.

<Functional configuration>
FIG. 3 is a block diagram showing an example of the functional configuration of the information processing apparatus 10 according to the first embodiment. The information processing device 10 shown in FIG. 3 includes a transmission unit 202, a reception unit 204, a processing control unit 206, and a storage unit 240.

The transmission unit 202 can be realized by, for example, a communication interface 106 or the like. The transmission unit 202 may transmit the classification result, the labeling result, the constructed model, and the like to each device connected via the network.

The receiving unit 204 can be realized by, for example, a communication interface 106 or the like. The receiving unit 204 receives, for example, a plurality of data from each acquisition device 20. The received data is, for example, the sound data of the environmental sound collected by a microphone or the like. It is output to the processing control unit 206.

The storage unit 240 stores various data such as data used for classification, data used for tagging, and data used for model construction. For example, the storage unit 240 stores characteristic data for labeling, identification information to be labeled, and the like.

The processing control unit 206 can be realized by, for example, the control unit 102 or the like. The processing control unit 206 executes classification processing, labeling processing, model construction processing, and the like on a plurality of data received by the receiving unit 204, for example, sound data.

The processing control unit 206 includes an acquisition unit 210, an extraction unit 212, a classification unit 214, a search unit 224, a grant unit 226, an analysis unit 228, and a visualization unit 234 in order to execute the above-described processing.

The acquisition unit 210 acquires a plurality of data from the reception unit 204. The acquisition unit 210 may acquire target data such as classification from a storage unit or the like in the own device in addition to the reception unit 204.

The extraction unit 212 extracts the feature amount of each data. When the data is sound data, the extraction unit 212 converts the sound data into a frequency domain and extracts one or more feature quantities related to the power spectrum in this frequency domain. The extraction unit 212 may calculate a feature amount suitable for the data.

More specifically, the extraction unit 212 analyzes the basis vector obtained by applying the non-negative matrix factor decomposition to the power spectrum, the dispersion matrix having the same dimensions of the power spectrum, and the melfilter bank analysis on the power spectrum as feature quantities. Includes at least one of the vector obtained by performing the above and the first principal component vector of the power spectrum reconstructed using the coefficient vector and the basis vector obtained by the non-negative matrix factor division. The feature amount of each extracted data is output to the classification unit 214.

For example, the classification unit 214 classifies the data by using the feature amount extracted by the extraction unit 212 according to an improved procedure of the AP method. The classification unit 214 generates a final cluster by repeating updating each of the first evaluation value and the second evaluation value, which will be described later. For example, the classification unit 214 determines cluster-centered candidate data and generates clusters until the values based on the first and second evaluation values updated using the updated values of the autocorrelation of the similarity matrix converge. I do. The classification unit 214 sets the autocorrelation value of the similarity matrix by using the similarity with respect to the outlier data in the cluster generated in the process of calculating the first evaluation value and the second evaluation value. The classification unit 214 includes a generation unit 216, a calculation unit 218, an update unit 220, and a determination unit 222 in order to use this classification method.

The generation unit 216 calculates a similarity matrix between each data using the feature amount extracted by the extraction unit 212. The generation unit 216 generates a similarity matrix by calculating the similarity between two data by a predetermined method.

For example, when the above-mentioned vector or variance matrix is extracted from the sound data, the generation unit 216 weights the similarity included in the similarity matrix to the similarity of each vector or variance matrix included in the feature quantity. It may be calculated by performing. The generated similarity matrix is output to the calculation unit 218.

The calculation unit 218 calculates, for example, the responsibility and availability used in the AP method of the classification method. Here, the responsibility is represented by r (i, k) and is referred to as the first evaluation value, and the availability is represented by a (i, k) and is referred to as the second evaluation value.

The calculation unit 218 calculates a first evaluation value indicating the appropriateness of the cluster-centered candidate data based on the similarity matrix generated by the generation unit 216 to be the cluster center of the member data belonging to the cluster. In addition, the calculation unit 218 calculates a second evaluation value indicating the appropriateness of the member data belonging to the cluster of candidate data based on the autocorrelation of the similarity matrix. Further, when the calculation unit 218 acquires the update value of the autocorrelation of the similarity matrix from the update unit 220, the calculation unit 218 newly calculates the first evaluation value and the second evaluation value based on the update value of the autocorrelation and the new cluster. And update.

Unlike the conventional AP method, the update unit 220 uses a new method for setting the value of s (k, k) indicating the autocorrelation of the similarity matrix. For example, the update unit 220 sets the autocorrelation using the similarity with respect to the outlier data in the cluster generated in the process of calculating the first evaluation value and the second evaluation value.

Because of the nature of clustering, even data with low similarity needs to be classified into one of the clusters, so data with low similarity is also included in the cluster. Therefore, in the first embodiment, an appropriate number of clusters is generated and the classification accuracy is improved by performing appropriate processing for outliers having low similarity.

In the conventional AP method, the autocorrelation value s (k, k) of the similarity matrix, which is a parameter that influences the number of clusters, is not updated. Further, even when the first evaluation value r (i, k) and the second evaluation value a (i, k) are updated, the value of s (k, k) is not updated. Therefore, in the conventional AP method, only the number of clusters based on the value of autocorrelation s (k, k) can be obtained, and there is a great possibility that outliers exist in the obtained clusters.

On the other hand, in the proposed method, outliers of clusters required in the process of updating the first evaluation value r (i, k) and the second evaluation value a (i, k) are detected, and the corresponding similarity is detected. A technique for updating the autocorrelation of a matrix is used. That is, in the process of updating the first evaluation value r (i, k) and the second evaluation value a (i, k), the value of s (k, k) of the outlier data that affects the determination of the number of clusters is also updated. Therefore, there is a possibility that clusters will be generated around the outlier data, and in the clustering result when the update finally converges, the possibility that outliers exist in each cluster is smaller than in the conventional method. Therefore, in the first embodiment, the classification accuracy can be improved.

In addition, the update unit 220 may update the autocorrelation of the similarity matrix with respect to the outlier data in the cluster to another value. Further, the update unit 220 may set a value based on the column vector regarding the out-of-similarity data of the similarity matrix to another value. Further, the update unit 220 may rearrange the column vectors in a predetermined order and set the predetermined th value to another value. This makes it possible to classify outliers once classified into a cluster into other clusters, and it becomes easier to generate a cluster composed of more appropriate data while updating the outlier data.

Further, the update unit 220 selects the similarity in this column vector based on the standard deviation of the maximum value in each column vector of the similarity matrix, and sets the selected similarity as the initial value of the autocorrelation of the similarity matrix. It may be set. Thereby, the deviation data can be identified in advance by using the standard deviation of the maximum value of the column vector, and the initial value of the autocorrelation for the deviation data can be set to a more appropriate value. Even if the initial value is set appropriately, the amount of outlier data in the cluster is reduced and the classification accuracy is improved.

The determination unit 222 determines whether or not the value based on the first evaluation value calculated by the calculation unit 218 and the second evaluation value converges. For example, the classification unit 214 uses the cluster center data at that time as the representative data when the added value of the first evaluation value and the second evaluation value converges. If the added values have not converged, the classification unit 214 redetermines the candidate data, reorganizes the cluster, and classifies the data again.

By performing the above classification process, the outliers in the cluster are easily excluded from the cluster by updating the autocorrelation that affects the number of classifications in the cluster, and the classification accuracy is improved. be able to.

Hereinafter, it is considered to construct a new model using the data appropriately classified by the above processing. At this time, modeling can be performed by labeling the classified data with the characteristics of the data.

For example, in the first embodiment, a new model is created by labeling the representative sounds of the clusters classified with respect to the sound data of the environmental sounds and learning using the feature amount of the labeled data. To build. This new model is a model that represents the entire environmental sound, not a conventional model that represents the estimation of a specific sound or the estimation of the direction of sound.

Next, the labeling process and the process related to modeling will be described. The storage unit 240 stores each type of data in association with the identification information indicating this type. For example, in the case of sound data, various characteristic sound data such as train sound, applause sound, event sound, and wind sound are associated with identification information indicating the type of the sound data. The identification information is information indicating a train sound, information indicating an applause sound, and the like, and is a kind of label.

The search unit 224 searches for the data stored in the storage unit 240, which matches the cluster-centered data generated by the classification unit 214. For example, which sound data stored in the storage unit 240 matches with the representative sound at the center of the cluster is searched for by using the similarity or the like. Note that the cluster-centered data may be other data as long as it is representative data indicating the characteristics of the cluster.

The assigning unit 226 assigns a label including the identification information associated with the data searched by the search unit 224 to the cluster-centered data. For example, if the data acquired by the acquisition unit 210 is sound data acquired by the microphone array, the direction of the sound and / or the generation time of the sound are associated with each other. Therefore, the addition unit 226 provides such information. It may be included in the label.

Although the representative data can be automatically given a label by using the search unit 224 and the giving unit 226, an appropriate label may be given by the user actually examining the data in the cluster. For example, the user may determine what the representative sound is by actually listening to the representative sound in the cluster, and label the representative sound.

The analysis unit 228 analyzes a plurality of data using the identification information included in the label. For example, the analysis unit 228 may calculate the ratio of each identification information in a plurality of data and use this ratio as the analysis result. In addition, the analysis unit 228 may subdivide the analysis result by using the direction of the sound and / or the generation time of the sound.

Further, the analysis unit 228 includes a learning unit 230 and a construction unit 232 in order to construct an analysis model using the classified data.

The learning unit 230 performs ensemble learning using the labeled data and the feature amount. Ensemble learning is a learning method in which a plurality of individually learned classifiers are prepared, and they are put together to construct one classifier by simply using the average of the outputs. For ensemble learning, Random Subspace Method (Reference TKHo, "The Random Subspace Method for Constructing Decision Forests," IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol. 20, No. 8, pp. 832-844, 1998 ) Is applied (reference "D. Opitz and R. Maclin," Popular ensemble methods: An empirical study, "Journal of Artificial Intelligence Research 11, pp. 169-196, 1999).

The construction unit 232 constructs a model related to data by learning by the learning unit 230. For model construction, for example, the MATLAB function (reference http://jp.mathworks.com/help/stats/random-subspace.html) is used.

The visualization unit 234 processes the analysis result analyzed using the constructed model so that the analysis result is visualized in the display unit. For example, the visualization unit 234 can graph the analysis result and make it easier for the user to understand.

From the above, by using this model, it is possible to investigate the atmosphere (activity) of the environment such as the generation ratio with respect to the environmental sound.

<Specific example>
Here, as a specific example of the first embodiment, the data is sound data of the environmental sound obtained from the environmental measurement. In the analysis method, the main problem is how to organize various environmental sounds and convert them into easy-to-understand information, and the method for solving the problem will be described below.

≪Environmental sound measurement location and acquisition device≫
FIG. 4 is a diagram showing a place where the environmental sound is measured. As shown in FIG. 4, the acquisition device 20 (microphone array) is installed at a predetermined height of the building BL1 on the right side of the station ST. In addition, there is a plaza AR between the building BL1 and the building BL2, and events and the like are frequently held in this plaza AR. The building BL1 and the building BL2 are shopping centers.

FIG. 5 is a diagram showing an example of a microphone array. FIG. 6 is a diagram showing the position of the microphone. The microphone arrays 20A to D (hereinafter, simply referred to as the microphone array 20) shown in FIGS. 5 and 6 are used for the environmental sound measurement.

Here, since the microphone array 20 is a sound measuring device using a plurality of microphone MCs, it is possible to estimate the sound source direction and the sound source position, and the information can also be utilized as data. The environmental sound collected by the microphone array 20 is the sound generated in the shopping center. More specifically, the environmental sounds are train sounds, open space maintenance sounds, event sounds, hashagi sounds, wind sounds, and the like. The maximum length of sound measured by the microphone array 20 is, for example, 2 seconds. The measured environmental sound is converted into the frequency domain and the power spectrum is calculated. Features are extracted from the calculated power spectrum, and the extracted features are used to construct an analysis model for environmental sound identification.

≪Overview≫
First, the outline of the environmental sound analysis method will be described. The processing control unit 206 applies a short-time Fourier transform to the generated sound data having a certain amplitude, and decomposes the obtained power spectrum Ps into a non-negative matrix factorization (reference: Hiroshi Sawada: "Non-negative matrix factorization NMF". Basics and application to data / signal analysis, "Journal of the Society of Electronics, Information and Communication Engineers, Vol.95, No.9, pp.829-833, 2012) is applied to obtain the basis vector Pv and the coefficient vector Pc.

Here, the processing control unit 206 regards the obtained value P (P includes Ps, Pv, and Pc) as the feature amount of the environmental sound. The processing control unit 206 calculates and generates a similarity matrix using these features. Next, the processing control unit 206 classifies the data into categories (clusters) using the improved Affinity Propagation (AP) method. Thereby, in this specific example, an appropriate category can be generated and the classification accuracy can be improved.

In addition, the processing control unit 206 labels each of the classified categories. The processing control unit 206 performs ensemble learning using the random subspace method using the labeled data, and constructs an analysis model for identifying environmental sounds. Here, in ensemble learning, for example, the k-nearest neighbor method (k-NN method) is used for the weak learner.

Further, in this specific example, it is possible to grasp what kind of sound environment the measurement environment forms by the analysis model of environmental sound identification. In addition, as an application example of this analysis method, for example, by distinguishing event sounds, cheers, applause, etc. at the time of an event, it is possible to provide a service that can measure the activity such as the excitement of an event by sound. ..

≪Analysis model≫
Next, the classification process, the labeling process, and the model building process for constructing the analysis model will be described in order.

(1) Classification process (1.1) Feature amount extraction The feature amount extraction unit 212 extracts the feature amount from the sound data of the environmental sound. For example, the extraction unit 212 performs a short-time Fourier transform on a certain environmental sound to obtain a power spectrum.

FIG. 7 is a diagram showing an example of a power spectrum. In the example shown in FIG. 7, the vertical axis represents frequencies (N), and the horizontal axis represents the number of frames (M frames) determined by the number of shifts of the short-time Fourier transform. At this time, the value of this power spectrum is considered to be a matrix element, and an N × M matrix Ps is considered. Where N is fixed and has 256 dimensions. M changes according to the length of the sound to be cut out.

The extraction unit 212 applies the non-negative matrix factorization to the matrix Ps to obtain the basis vector and the coefficient vector.

FIG. 8 is a diagram showing an example of a basis vector. FIG. 9 is a diagram showing an example of a coefficient vector. For non-negative matrix factorization, for example, Gamma Process Non-negative Matrix Factorization (Gap-NMF) (Reference M. Hoffman, D. Blei, and P. Cook, "Bayesian Nonparametric Matrix Factorization for Recorded Music," Proceedings of the 27 ^th International Conference on Machine Learning, Haifa, 2010) is used.

Gap-NMF is a factorization method that determines the basis vector so that characteristic frequencies are selected. At this time, the number of basis vectors differs depending on the shape of the power spectrum. Further, the coefficient vector is a vector having a value representing the temporal change of each basis vector as an element. In the examples shown in FIGS. 8 and 9, the dark part represents a large value, and the value decreases as the color gradually changes to lighter.

FIG. 10 is a diagram showing an example of a power spectrum reconstructed using a basis vector and a coefficient vector. The power spectrum shown in FIG. 10 is generated by multiplying the basis vector shown in FIG. 8 by the coefficient vector shown in FIG. That is, the power spectrum shown in FIG. 10 is a reconstruction of the power spectrum shown in FIG. 7.

Due to the characteristics of Gap-NMF, the reconstructed power spectrum (see FIG. 10) may have a power spectrum shape in which the characteristic frequency is emphasized in the shape of the original power spectrum (see FIG. 7). I understand. Hereinafter, the reconstructed power spectrum is referred to as Pr _s .

The extraction unit 212 calculates, for example, the following four values as the feature amount of the environmental sound.
・ Matrix Π ₁ , which is a square matrix of the original power spectrum by the following calculation.
・ Mean value of basis vector, vector f,
-Principal component analysis is applied to the square matrix Π ₂ obtained by the following calculation, and the first principal component vector e, obtained
・ Amplitude spectrum fa of P _s compressed to 20 dimensions by mel filter bank analysis
Here, fa is treated as a feature amount that enables the time change of the power spectrum to be compared in the same dimension (20 dimensions) for the power spectra having different Ms.

(1.2) Classification Method The classification unit 214 classifies the environmental sounds generated in one day by using the above-mentioned four feature quantities. Here, let L be the number of daily sound data measured by the microphone array 20. The classification unit 214 uses the same method as the AP method as the classification method. The AP method is a clustering method that can obtain a representative sound (exemplar) and an environmental sound similar to it by using the following two equations according to the similarity of each sound data. The calculation unit 218 calculates the first evaluation value and the second evaluation value using the following equations (3) and (4).

Here, the first evaluation value r (i, k) represents a message sent from the environmental sound data i to the exemplar candidate k in the AP method. The second evaluation value a (i, k) represents a message sent from the exemplar candidate k to the environmental sound data i that may be included in the cluster. At this time, the parameter s (i, k) represents the relationship (for example, similarity) between each element. The parameter a (k, k) is updated using the following equation.
In this way, the AP method exchanges messages between the exemplar candidate k and the sound data i. That is, the classification unit 214 is a message exchange type clustering method in which the exemplar is obtained by alternately obtaining the equation (3) and the equation (4), and the sound data classified into the exemplar is determined. At this time, the initial value of a (i, k) is set to 0.

Here, the AP method is a conventional clustering method, for example, a method in which the number of exemplars is determined depending on the data without requiring information on the number of clusters in advance unlike the k-means method. It is known that the number of exemplars is determined by s (k, k), that is, the magnitude of autocorrelation of environmental sound data. Further, in the conventional method, there are many cases where the AP method is operated by setting the median value of the total similarity s (i, k) to s (k, k).

However, for example, when the classification target data is sound data, the conventional s (k, k) setting method often fails in classification. In other words, due to the nature of clustering, data is classified into some cluster, so some of the data in the cluster has a low degree of similarity to many data (hereinafter, also referred to as outlier data). Will be included.

In order to solve this drawback, an improved s (k, k) setting method is used in the first embodiment. Here, the similarity matrix having s (i, k) as an element is expressed as S. Further, it denoted the vertical vector of the similarity matrix _{s k (k = 1,2, ...} , L) and. In this case, representing the maximum value for each of _{s k s maxk (k = 1,2} , ..., L) and.

Here, the following two examples will be described as the setting method (update method) of s (k, k). Equation (6) is used as the first update method. The update unit 220 obtains the out-of-order data by the equation (6).

Here, abs (x) represents the absolute value of x, mean (x) represents the mean value of x, std (x) represents the standard deviation of x, and the parameter α represents a positive number. .. That is, the out-of-cluster data X can be detected by the equation (6). The update unit 220 applies the following rules to update s (k, k) of the data point k of the detected outlier data.

Here, s _k denotes the k column vectors of the similarity matrix, sort (s _{k) nth} represents the n-th value from the top of the descending k column vector s _k. In this specific example, for example, the fourth value is used, but which value is used depends on the classification form of the given data.

Further, the parameter α of the equation (6) is changed depending on how much deviation data is assumed and the classification is performed. The initial value of s (k, k) is set to the median value of all the elements of the similarity matrix used in the conventional AP method.

Next, the second update method will be described. The update unit 220 uses a method of determining the initial value of s (k, k) based on the standard deviation σ _s of the maximum value s _maxk (k = 1,2, ..., L). That is, the update unit 220 determines s (k, k) using the following equation (8) as the threshold value Th.
Th = σ _s / a (8)

Here, the constant a is an arbitrary positive integer. Threshold value Th, in the elements of s _k, attempt to set a value close to the largest possible value s (k, k), the threshold was considered by the inventors.

Updating unit 220, the elements of each s _k, with larger elements than sigma _s / a, sets the value of the closest element to the sigma _s / a to an initial value of s (k, k). If you do not have an element that s _k is the case, the update unit 220 sets the second value of s _k to the initial value of s (k, k).

In the case of the second update method, the classification unit 214 obtains the outlier data in the cluster according to the value of exemplar as the post-processing clustered by the improved AP method, and further clusters this outlier data as the exmplar candidate. do it. Further, the update unit 220 may use one of the two update methods described above, sets the initial value of the autocorrelation using the second update method, and further sets the first update method. It may be used to update its initial value.

The classification unit 214 repeats the classification process until the added value of r (i, k) + a (i, k) converges. Convergence means that, for example, even if reclassification is performed a predetermined number of times or more, the added value is within a predetermined range. Thereby, in this specific example, the classification accuracy can be improved by using the clustering in the improved AP method for processing the out-of-cluster data.

(2) Labeling The search unit 224 searches the sound data stored in the storage unit 240 for sound data similar to the representative sound (for example, the sound of the exemplar) of each category classified by the classification unit 214. For example, the search unit 224 searches for sound data having the highest degree of similarity to the representative sound.

The assigning unit 226 assigns a label including identification information related to the searched sound data to the representative sound. In addition to the identification information, the label may include a sound generation position, a sound generation time zone, and the like. Further, the adding unit 226 may give the same label as the representative sound to the sound data in the cluster. This eliminates the need to perform search processing on all data, and can reduce the processing load.

Further, as labeling, the user may listen to the representative sound, select a label suitable for the representative sound, and label the label. In this case, the giving unit 226 may give the label of the representative sound given by the user to the sound data of the same category as the representative sound.

(3) Learning processing The analysis unit 228 uses each labeled sound data as a teacher signal and uses the above-mentioned feature quantities (f', e, fa) of the environmental sound for each as training data for one day. Build an analytical model to identify environmental sounds. Here, f'is the frequency mean value of Pr _s . Further, f'may use the mean value of the basis vector, the vector f.

In this specific example, the analysis unit 228 builds an analysis model for identifying environmental sounds using ensemble learning. The learning unit 230 executes ensemble learning using the above-mentioned random subspace method as ensemble learning. For weak learners, a model is constructed using the k-nearest neighbor method. Here, for example, the MATLAB function (fitensemble) is used for model construction.

≪Experiment≫
Here, the experiments conducted by the inventors will be described. The inventors install the microphone array 20 shown in FIGS. 5 and 6 at the position shown in FIG. 4, and the microphone array 20 measures the environmental sound generated in the public space.

Five days' worth of sound data measured from July 7, 2014 (Monday) to July 11, 2014 (Friday) will be used to build the environmental sound model. First, the classification result by the classification unit 214 will be described. In this experiment, the second method is applied as the update method, and the parameter a in the equation (8) is set to 3.

The elements s (i, k) of the similarity matrix are calculated using the Itakura spectral distance (Di) and the COSH spectral distance (Dcosh) using the features of each environmental sound data.
The parameter Ci (i = 1,2,3,4) represents the weighting factor for each distance and is given a positive value.
FIG. 11 is a diagram showing an example of the result of clustering sound data using the improved AP method. In the example shown in FIG. 11, the number of clusters with respect to the number of data is shown. In the example shown in FIG. 11, the compression rate (= number of clusters / number of data) is 46.2%.

Although not shown, the AP method using the method of setting the median value of all s (i, k) to s (k, k) used in the conventional method has a compression ratio of 91.7%. In other words, since the number of clusters is small, there is a high possibility that outlier data is included in the cluster. When the first update method is used in the update of the autocorrelation, the parameters may be appropriately adjusted by a preliminary experiment or the like so that the compression rate matches the update method.

Next, verification of the accuracy of the analysis model constructed by the above processing will be described. The sound data used for the verification is the environmental sound measured from July 14, 2014 to July 18, 2014. An analysis model constructed using the sound data from July 7, 2014 (Monday) to July 11, 2014 (Friday) is applied to each sound data. At this time, the analysis model is applied for each day of the week, and the number of times an incorrect label is output (the number of sounds different from the representative sound classified into the cluster of the representative sound) is counted.

FIG. 12 is a diagram showing an example of the accuracy result (No. 1) of the improved AP method. As shown in FIG. 12, the average accuracy rate for each day of the week is 91%. The number of incorrect answers indicates a number in which a person listens to sound data classified into clusters and the sound data is different from the representative sound of the cluster.

Although not shown, the correct answer rate is about 78% when the analysis model when labeled using the representative sounds clustered by the conventional AP method is used. From this, it can be considered that the improved AP method appropriately generated categories in the clustering result and increased the classification accuracy of the categories.

Furthermore, we will consider the case where the analysis model is constructed using all the sound data from July 7th to July 11th, not every day of the week. This analysis model was applied to each environmental sound data from July 14 to July 18, and the inventors verified whether the classification accuracy would improve as the number of data used for model construction increased.

FIG. 13 is a diagram showing an example of the accuracy result (No. 2) of the improved AP method. As shown in FIG. 13, the average accuracy rate for each day of the week is 93%. In the example shown in FIG. 13, the number of incorrect answers is reduced and the average correct answer rate is improved on each day of the week. From this, it can be seen that the reliability of the model is improved when the analysis model is constructed using a large amount of data.

Next, using the analysis model described above, the generation rate of environmental sounds classified by the visualization unit 234 is shown.

FIG. 14 is a diagram showing an example of the generation rate of environmental sounds in one day. As shown in FIG. 14, in this experiment, the place where the microphone array 20 was installed has the station ST in the immediate vicinity (see FIG. 4), so that the train sound generation rate is the highest.

The second is the high rate of crowded noise that represents the zawatsuki of the plaza AR, and the third is the high rate of the maintenance sound of the plaza. In addition, there are rattling noises, wind noises, and event noises.

The giving unit 226 labels such a sound, and the visualization unit 234 can visualize the generation ratio thereof. From the graph shown in FIG. 14, it becomes possible to understand the atmosphere of the sound environment of the day. Furthermore, the analysis model generated using the improved AP method has a correct answer rate of over 90% as shown in FIGS. 12 and 13, so that the visualization of the sound environment has reached a reliable level. I can say.

As described above, the analysis model in this specific example can grasp the atmosphere of the sound environment from the visualization of the generation rate of the environmental sound. Therefore, as an application example of this analysis model, the inventors verified whether the atmosphere of the event could be grasped from the sound generated during the event regarding the event held in the plaza AR.

Regarding the creation of the analysis model, the event sounds (songs, music, etc.) and applause sounds were newly labeled using the sound data of the environmental sounds of the day generated on the day the event was held, and were created before. The model was built by adding a cluster with a new label to the analysis model.

FIG. 15 is a diagram showing an example of the environmental sound generation ratio in the event A from 10:00 to 13:00 on a certain event day. Here, all the examples shown below are the results obtained from the sound data not used for model construction. In the following figures, the environmental sounds include a hash sound, an event sound, an applause sound, and the like.

FIG. 16 is a diagram showing an example of the environmental sound generation ratio in the event A from 14:00 to 18:00 on a certain event day. Event A shown in FIG. 16 and event A shown in FIG. 15 are the same event. According to the graphs shown in FIGS. 15 and 16, it can be seen that the hashing sound, the event sound, and the clapping sound are generated at almost the same ratio from the middle to the latter half of the event.

This makes it possible to examine how the atmosphere and excitement of this event was by comparing it with the results of questionnaires from actual events, and by comparing it with this graph.

FIG. 17 is a diagram showing an example of the environmental sound generation ratio in the event B from 14:00 to 18:00 on a certain event day. In the example shown in FIG. 17, the rate of occurrence of event sounds is high, and it can be seen that this is a concert-type event centered on music and songs. In addition, although there are rattling sounds and applause sounds, the proportion of event sounds is overwhelmingly large, so it can be inferred that the event was lively with music and songs.

Further, an example in which the atmosphere is different from the above-mentioned environmental sound will be described with reference to FIGS. 18 and 19. FIG. 18 is a diagram showing an example of the environmental sound generation ratio in the event C from 10:00 to 13:00 on a certain event day.

FIG. 19 is a diagram showing an example of the environmental sound generation ratio in the event C from 14:00 to 18:00 on a certain event day. Event C shown in FIG. 18 and event C shown in FIG. 19 are the same event. According to the graphs shown in FIGS. 18 and 19, applause sounds were generated at the start of the event, and there was an atmosphere of excitement, but the atmosphere of excitement gradually increased toward the middle and the latter half of the event. It is possible to grasp the tendency that it seems to be decreasing.

From the above, the atmosphere of the event can be grasped by using the analysis model for identifying the environmental sound. Therefore, as a further application example of this analysis model, it can be expected to be used for event evaluation.

In this specific example, it is possible to provide a method for classifying environmental sound data and a method for constructing an analysis model for identifying environmental sounds from the classification result. In this specific example, the identification of the environmental sound can be realized with a probability of about 90% or more, and the place where the environmental sound is generated and the atmosphere of the sound environment can be understood.

In addition, as an application example of this specific example, it can be expected to be used for grasping the atmosphere of the event from the sound generated at the time of the event, and further for evaluating the excitement of the event. In addition, it is possible to verify whether or not the result inferred from the environmental sound generation rate at the time of the event matches the actual atmosphere by questionnaires to the event participants.

<Operation>
Next, the operation of the information processing system 1 in the first embodiment will be described. FIG. 20 is a flowchart showing an example of data analysis processing according to the first embodiment. In step S102 shown in FIG. 20, the classification unit 214 performs classification processing on a plurality of data. The details of the classification process will be described with reference to FIG.

In step S104, the search unit 224 and the granting unit 226 perform the labeling process. The details of the labeling process will be described with reference to FIG.

In step S106, the analysis unit 228 performs data analysis using the classified data. The details of the data analysis will be described with reference to FIG.

FIG. 21 is a flowchart showing an example of the classification process in the first embodiment. For example, the process shown in FIG. 21 is executed when necessary data is periodically collected. For example, in the case of sound data, the classification process may be started every day after the data acquisition process is completed.

In step S202 shown in FIG. 21, the acquisition unit 210 acquires a plurality of data received from the acquisition device 20. For example, in the case of a microphone in which the acquisition device 20 is installed outside, the acquisition unit 210 acquires the sound data of the environmental sound. Further, the plurality of data may be acquired from the storage unit 240 of the own device.

In step S204, the extraction unit 212 extracts the feature amount from each data. The feature amount to be extracted may be changed according to the data. For example, in the case of sound data, features related to the power spectrum in the frequency domain are extracted.

In step S206, the generation unit 216 uses the extracted features to generate a similarity matrix represented by the similarity between the data.

In step S208, the calculation unit 218 calculates the evaluation value used for the convergence test of the classification. For example, the calculation unit 218 calculates the first evaluation value r (i, k) and the second evaluation value a (i, k) when the improved AP method is used. Further, the calculation unit 218 calculates the added value of the first evaluation value and the second evaluation value as the end determination parameter of the classification process.

In step S210, the determination unit 222 determines whether or not the end determination parameters have converged. For example, the determination unit 222 determines whether or not the added value has converged. If the end determination parameters have converged (step S210-YES), the process proceeds to step S214, and if the end determination parameters have not converged (step S210-NO), the process proceeds to step S212.

In step S212, the update unit 220 detects the outlier data in the cluster and updates the autocorrelation of the detected outlier data to another value. The autocorrelation is included in the similarity matrix. When the update process is completed, the process returns to step S208, and the first evaluation value and the second evaluation value are updated using the updated autocorrelation.

In step S214, the classification unit 214 determines the cluster when the determination target parameter is completed as the final result. As a result, since clustering is performed in consideration of the outlier data as compared with the conventional AP method, the number of outlier data in the cluster can be reduced and the classification accuracy can be improved.

FIG. 22 is a flowchart showing an example of the tagging process according to the first embodiment. In step S302 shown in FIG. 22, the search unit 224 selects one cluster from each cluster and selects representative data in the cluster. The representative data is, for example, cluster-centered data.

In step S304, the search unit 224 searches the storage unit 240 for data corresponding to the selected representative data. For example, the search unit 224 searches for data that is most similar to the representative data.

In step S306, the assigning unit 226 includes the identification information associated with the searched data in the label and assigns it to the representative data. In addition to the identification information, the label may contain other information about the data.

In step S308, the granting unit 226 determines whether or not all the representative data have been labeled. One representative data is included in each cluster, for example. If all the representative data are labeled (step S308-YES), the process ends, and if all the representative data are not labeled (step S308-NO), the process returns to step S302, and the other Representative data is selected.

FIG. 23 is a flowchart showing an example of the analysis process according to the first embodiment. In step S402 shown in FIG. 23, the learning unit 230 and the construction unit 232 perform ensemble learning using each labeled data as a teacher signal and the feature amount of each teacher signal as training data, and construct an analysis model. ..

In step S404, the classification unit 214 applies this analysis model to classify the data to be analyzed. Further, when calculating the correct answer rate of the classification, the analysis unit 228 compares the label of the data assigned in advance with the label assigned after the classification, and counts the data having different labels. In addition, a person may analyze the classified data and count the number of misclassified data.

In step S406, the visualization unit 234 processes the analysis result so that it can be visualized on the display unit. For example, the visualization unit 234 graphs the analysis result and makes it easy to grasp.

As described above, an analysis model can be constructed for the entire acquired data, and further, the analysis result can be visualized so that it can be easily grasped at a glance.

As described above, in the first embodiment, in order to help the service industry suffering from processing a large amount of data, it is possible to improve the classification accuracy in the data classification first. Furthermore, it is possible to propose a data analysis method for executing data utilization. In addition, it is possible to present what kind of service the results obtained from the analysis can be utilized.

For example, according to the first embodiment, when analyzing sound data, not only the direction and position of the sound are estimated, but also the atmosphere of the sound environment is determined from the sound generation status. You can grasp the environmental pattern.

More specifically, it is possible to measure the sound environment at the time of the event and present an objective index by sound such as excitement and liveliness as to what kind of atmosphere the event was. In addition, in an apartment house, the sound environment around it can be measured and the sound environment can be presented to prospective tenants. Further, by detecting the outlier data in the cluster, this outlier data may be analyzed and some characteristics may be grasped.

Further, according to the first embodiment, representative data can be detected from a large amount of data and analysis can be performed using the representative data, so that data analysis does not take time. Regarding data model construction, if data is classified into categories by an appropriate classification method, it is possible to label the categories and build a model.

[Second Embodiment]
Next, the second embodiment will be described. The second embodiment relates to a technique for monitoring the sound environment as a color pattern. The data acquired in the second embodiment is sound data. In the second embodiment, one or a plurality of feature amounts are extracted from the sound data, and the extracted feature amounts are expressed by colors, so that the sound data such as environmental sound is expressed by a color pattern (hereinafter, also referred to as a sound pattern). Can be expressed.

For example, the information processing appeal device according to the second embodiment extracts three features of pitch, pitch, loudness, and continuation of the sound from the sound data of the environmental sound, assigns a predetermined color to each of them, and expresses a sound pattern. Further, by estimating the sound source position of the sound, it is possible to express the sound pattern within a predetermined region. This sound pattern is visible to the user by being displayed on the screen.

Since the system configuration and the hardware configuration in the second embodiment are the same as the system configuration and the hardware configuration in the first embodiment, the description thereof will be omitted. Further, the same reference numerals as those in FIGS. 1 and 2 are used for the system configuration and the hardware configuration in the second embodiment.

<Functional configuration>
FIG. 24 is a block diagram showing an example of the functional configuration of the information processing apparatus 10 according to the second embodiment. The information processing device 10 shown in FIG. 24 includes a transmission unit 202, a reception unit 204, a processing control unit 300, and a storage unit 310. Since the transmitting unit 202 and the receiving unit 204 are the same as those in the first embodiment, they are designated by the same reference numerals and the description thereof will be omitted. The storage unit 310 stores various data such as data used for classification, data used for feature amount extraction, and data used for model construction. For example, the storage unit 310 stores data used for color expression, color-expressed image data, and the like.

The processing control unit 300 includes an acquisition unit 210, an extraction unit 212, a classification unit 214, a sound feature extraction unit 302, an expression unit 304, and an estimation unit 306. Since the acquisition unit 210, the extraction unit 212, and the classification unit 214 are the same as those in the first embodiment, they are designated by the same reference numerals and the description thereof will be omitted.

The sound feature extraction unit 302 extracts the first feature amount of the sound data based on each category classified by the classification unit 214. For example, the sound feature extraction unit 302 has a frequency that is representative of each category (for example, the center of gravity of the category) for each category in which sound data within a predetermined period is classified for each feature by the classification unit 214 using the improved AP method. Frequency) is used to extract the first feature quantity.

As a more specific example, the sound feature extraction unit 302 classifies representative frequencies into three, large, medium, and small, and generates a histogram. Next, the sound feature extraction unit 302 gives a value of 0 or 1 for each of large, medium, and small depending on the frequency in the histogram. The sound feature extraction unit 302 uses seven scores from (0,0,1) to (1,1,1) of (large, medium, small) as the first feature amount.

The expression unit 304 expresses the first feature amount by the first color from the sound data. The first color is, for example, G (green). As a result, the characteristics of the sound data can be expressed by colors, and it is possible to make it easier to visually recognize a sound that is difficult to see.

Further, the sound feature extraction unit 302 may extract the second feature amount and the third feature amount from the sound data. For example, the sound feature extraction unit 302 may obtain a second feature amount indicating the magnitude of the sound data and a third feature amount indicating the continuity of the sound data. The sound feature extraction unit 302 generates a second feature amount indicating the magnitude of the acquired sound data and a third feature amount indicating the continuity of the sound data.

As a second feature quantity, the sound feature extraction unit 302 has, for example, a known loudness (reference ITU-R BS.17701-1 recommendation http://www.itu.int/rec/R-REC-BS.1770/). The strength of the sound is expressed by a score of 7 paragraphs based on the volume (unit: LKFS) obtained by the calculation of. The sound feature extraction unit 302 sets the score so that the larger the score, the louder the volume. Further, the sound feature extraction unit 302 may simply obtain the envelope of the waveform of the sound data in a predetermined time unit, and set the average value as the loudness of the sound. Further, the sound feature extraction unit 302 may add the power spectrum in the frequency axis direction and use the average value in the time axis direction as a substitute for the loudness.

As the third feature amount, the sound feature extraction unit 302 obtains a histogram from the mutual information amount based on, for example, the time of maximum power in the power spectrum of the sound data (reference Takuya Tsunoda, Masahiro Nakagawa, "mutual information amount and mutual". Estimating the time difference between myoelectricity and mechanomyogram using a correlation function, credential technique, MBE2015-60, pp.37-42, 2015.). In addition, the sound feature extraction unit 302 calculates the kurtosis of the obtained histogram. At this time, the sound feature extraction unit 302 represents a sudden sound as the kurtosis value is larger, and represents a sound with less fluctuation and higher sustainability as the kurtosis value is smaller. Therefore, the sound feature extraction unit 302 sets the degree corresponding to the kurtosis value as the sound continuity. For example, the continuity of a sound is represented by a score of 7 levels.

As described above, when the three feature amounts are extracted, the expression unit 304 expresses the second feature amount by the second color and the third feature amount by the third color. The second color is represented as R (red), and the third color is represented as B (blue). For example, each color is represented by a value between 0 and 255. This makes it possible to express sounds such as environmental sounds using colors. The color combination used is not limited to the three primary colors of light.

The estimation unit 306 estimates the sound source position within a predetermined region with respect to the sound data. For example, the estimation unit 306 divides a predetermined region into blocks, selects a frequency for each division, and uses the frequency by the MUSIC method (reference RO Schmidt, "Multiple Emitter Location and Signal Parameter Optimization," IEEE Trans. On Antennas and The sound source position can be estimated from Propagation, Vol. 34, No. 3, pp. 276-280, 1986.). Further, the estimation unit 306 may mainly use a beamformer method, a linear prediction method, a minimum variance method, or the like as a method for estimating the arrival direction of sound (reference: Juro Oga, Yoshio Yamazaki, Yutaka Kaneda, acoustic system). And digital processing, published by the Institute of Electronics, Information and Communication Engineers).

At this time, the expression unit 304 expresses the first feature amount in the first color and the second feature amount in the second color with respect to the sound data of the sound source position estimated by the estimation unit 306. The feature amount is expressed in the third color. As a result, the sound in the predetermined region can be expressed in RGB for each sound source position, and the sound pattern can be visually expressed using colors.

<Extraction of sound features>
Next, the extraction of sound features will be described. First, the sound feature extraction unit 302 obtains the power spectrum of the sound data in order to obtain the first sound feature amount. Here, the sound feature extraction unit 302 captures a power spectrum that can change with time within a predetermined period as one image.

FIG. 25 is a diagram showing an example of a power spectrum. In the example shown in FIG. 25, a power spectrum having a predetermined value or more is displayed. An appropriate threshold value for indicating that sound is being generated is set in the predetermined value. The sound feature extraction unit 302 performs edge detection on a diagram in which a power spectrum of a predetermined value or more is expressed.

FIG. 26 is a diagram showing an example of an edge-detected power spectrum. In the example shown in FIG. 26, in each center of gravity of the region surrounded by the edge detection, those having a short distance are classified into one category by the improved AP method by the classification unit 214.

Here, the reason for using the improved AP method is that the classification becomes more detailed and the frequency distribution can be grasped more accurately.

FIG. 27 is a diagram showing an example of each of the classified categories. In the example shown in FIG. 27, the categories are classified into seven categories CT1 to CT7. For example, the sound feature extraction unit 302 classifies the frequency (representative frequency) of the center of gravity in each category into large (4 kHz or more), medium (1 kHz or more and less than 4 kHz), and small (less than 1 kHz) regions. The threshold value of each region is an example and is not limited to this.

Next, the sound feature extraction unit 302 counts how many categories of each frequency domain exist on the time axis. For example, counting the number of categories on the horizontal axis in a large area is 2. The sound feature extraction unit 302 normalizes the count value by using the maximum value of the count number in each region. For example, the sound feature extraction unit 302 divides each count value by the maximum value. At this time, a value above a certain threshold value (for example, 0.4) is set to 1, and a value below the threshold value is set to 0. For example, when (large, medium, small) is (1, 0.2, 0.5), it becomes (1, 0, 1).

Further, the sound feature extraction unit 302 assigns a score (first feature amount) to each pattern from (0,0,1) to (1,1,1). For example, the following processing is performed for sound feature extraction.
Pattern: Score value (0,0,1): 1
(0,1,0): 2
(0,1,1): 3
(1,0,0): 4
(1,0,1): 5
(1,1,0): 6
(1,1,1): 7
As described above, the sound feature extraction unit 302 uses the numbers 1 to 7 as the first feature amount.

FIG. 28 is a diagram for explaining the calculation of loudness of sound. In the example shown in FIG. 28, the horizontal axis is time and the vertical axis is loudness. Loudness is used for the loudness, but any index may be used as long as the loudness can be calculated.

The sound feature extraction unit 302 assigns a score of, for example, 7 paragraphs according to the sound level. The sound feature extraction unit 302 sets a normal sound level (for example, a reference level) to 1, sets a threshold value stepwise according to the sound level of the sound environment, and assigns a 7-step score (second feature amount). To do so. The reference level and threshold can be changed as appropriate.

FIG. 29 is a diagram for explaining the calculation of the continuity of sound. In the example shown in FIG. 29, the kurtosis of sound is used to determine the degree of continuity. First, the sound feature extraction unit 302 calculates the correlation value with respect to the maximum value of the power spectrum (the portion surrounded by the circle shown in FIG. 29) using the mutual information amount.

Next, the sound feature extraction unit 302 generates a histogram of the obtained value using the hist of MATLAB. The sound feature extraction unit 302 calculates the kurtosis for grasping the shape of the histogram with respect to the created histogram.

Finally, the sound feature extraction unit 302 assigns seven scores as follows according to the calculated kurtosis value.
Condition: Score value
Kurtosis ≧ 4.5: 1
4.5> Kurtosis ≧ 3.5: 2
3.5> Kurtosis ≧ 2.5: 3
2.5> Kurtosis ≧ 1.5: 4
1.5> Kurtosis ≧ 1.0: 5
1.0> Kurtosis ≧ 0.5: 6
0.5> Kurtosis: 7

The expression unit 304 replaces the score value of 7 stages with a value of 0 to 255 for each feature amount and expresses the color. For example, assuming that (first feature amount: second feature amount: third feature amount) is (5: 2: 1), the expression unit 304 has (255 × 1: 2) for the color (G: R: B). It can be expressed as 255 × (2/5): 255 × (1/5)).

<Sound pattern experiment>
Here, an experiment on a sound pattern conducted by the inventors will be described. FIG. 30 is a diagram showing a predetermined region expressing a sound pattern in the experimental site. As shown in FIG. 30, the experimental site is the same as the experimental site shown in FIG. 4, and the predetermined area is the area AR10 in the vicinity of the microphone array 20. The sound data is data for a maximum of 2 seconds, and is an environmental sound having a certain amplitude. Further, the area AR10 is divided into each block area of 5 m × 5 m, and an RGB value is obtained for each block area.

FIG. 31 is a diagram showing a sound pattern in normal times. FIG. 32 is a diagram showing a sound pattern at the time of an event. As shown in FIGS. 31 and 32, when various sounds are generated (at the time of an event), the sound pattern becomes colorful (the difference in shade is large), and in normal times when there is not much change in sound, it is blue. The sound pattern is mainly yellowish green (the difference in shade is small). That is, it can be seen that the sound pattern can detect changes in the state of the sound environment. Therefore, it can be said that the sound pattern using the above-mentioned three feature quantities is an effective monitoring method that is sufficiently useful for detecting and grasping the state of the sound environment.

In the above-mentioned example, three feature quantities are expressed by a score of seven stages, and each value is replaced with an RGB value to draw a sound pattern. The reason for setting 7 levels is that when calculating the pitch of the sound as the first feature, a histogram is created in 3 stages of high, medium, and low, and the size of the histogram is converted to 0 or 1. Therefore, the values for (high, medium, low) are divided into seven cases from (0,0,1) to (1,1,1). Similar to the first feature amount, the other feature amounts use a score of 7 levels according to the size.

Here, regarding the loudness value of the sound and the kurtosis value of the continuity, the threshold value for classifying into seven stages varies depending on the sound data acquired in the sound environment. Therefore, the threshold value is divided into cases according to each sound environment. Should be decided.

In addition, by using the sound pattern as learning data and using the date when the sound pattern was measured as the teacher data, understanding of the sound environment using the sound pattern, for example, which day is similar, is different from usual. It is thought that it will be possible to build an identification model that can grasp whether or not it has a sound pattern. Here, for the discriminative model, for example, a naive Bayes classifier or discriminant analysis (Fisher discriminant analysis (see C.M. Bishop "Pattern Recognition and Machine Learning," Springer, 2006)) can be used.

In addition, by fusing other sensors such as a vibrometer and an image, the above-mentioned sound pattern and identification model can be used for monitoring the aging (process until collapse) of the building.

In addition, the system can search for similar sound patterns to a predetermined sound pattern by associating the sound patterns with each date and storing them in the database. At this time, a similar sound pattern may be searched for each block.

<Operation>
Next, the operation of the information processing system 1 in the second embodiment will be described. FIG. 33 is a flowchart showing an example of the sound pattern expression processing in the second embodiment. In step S502 shown in FIG. 33, the acquisition unit 210 acquires sound data.

In step S504, the estimation unit 306 estimates the sound source position of the acquired sound data at predetermined intervals.

In step S506, the sound feature extraction unit 302 extracts each feature amount of the sound data corresponding to the estimated sound source position. In order to extract the first feature amount, the classification process of step S102 shown in FIG. 20 is executed.

In step S508, the expression unit 304 replaces each of the extracted feature amounts with a value indicating each color, and expresses the sound by color. As a result, as described above, the sound environment can be visually expressed in color as a sound pattern and can be easily grasped.

In addition, each processing step included in the processing flow in the first embodiment and the second embodiment described above can be arbitrarily changed in order or executed in parallel as long as there is no contradiction in the processing contents. , Other steps may be added between each processing step. Further, the steps described as one step for convenience can be executed by being divided into a plurality of steps, while those described as one step can be grasped as one step for convenience.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and modifications can be made based on the technical idea of the present invention.

1 Information processing system 10 Information processing device 20 Acquisition device 102 Control unit 108 Storage unit 206 Processing control unit 210 Acquisition unit 212 Extraction unit 214 Classification unit 224 Search unit 226 Granting unit 228 Analysis unit 234 Visualization unit

Claims (23)

An acquisition unit that acquires multiple sound data representing environmental sounds,
An extraction unit that extracts the features of each sound data,
A classification unit that classifies the plurality of sound data into each category using the extracted features, and
An analysis unit that analyzes the sound data based on each category classified by the classification unit using the identification information of the sound data, the direction of the sound data, and / or the generation time of the sound data.
A visualization unit that visualizes the analysis results analyzed by the analysis unit,
Equipped with a,
The classification unit
By repeating the update of each of the first evaluation value and the second evaluation value based on the similarity matrix between each sound data generated using the extracted feature data, a final cluster is generated and classified into each category. Then, the autocorrelation of the similarity matrix is set using the similarity with respect to the outlier data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value, and the first evaluation value is The cluster-centered candidate data is a value indicating the appropriateness of the member data belonging to the cluster to be the cluster center, and the second evaluation value is the appropriateness of the member data belonging to the cluster of the candidate data. An information processing device that is a value indicating . A sound feature extraction unit that extracts the first feature amount, the second feature amount, and the third feature amount of the sound data based on each category, and a sound feature extraction unit.
The first feature amount is represented by the first color, the second feature amount is represented by the second color, and the third feature amount is represented by the third color. , The information processing apparatus according to claim 1. Further, an estimation unit for estimating the sound source position within a predetermined region is provided for the sound data.
The expression part
The information processing device according to claim 2, wherein the sound data of the sound source position is expressed by the first color, the second color, and the third color. The sound feature extraction unit
The first feature amount indicating the pitch of the sound data based on the center frequency of the category, the second feature amount indicating the magnitude of the sound data, and the third feature amount indicating the continuity of the sound data. The information processing apparatus according to claim 2, which is extracted. The classification unit
Claim that the similarity in the column vector is selected based on the standard deviation of the maximum value in each column vector of the similarity matrix, and the selected similarity is set as the initial value of the autocorrelation of the similarity matrix. The information processing apparatus according to 1 . The classification unit
The information processing apparatus according to any one of claims 1 to 5 , wherein the final cluster is generated by repeatedly updating the first evaluation value, the second evaluation value, and the autocorrelation value. .. The classification unit
A generator that generates the similarity matrix using the features, and
A calculation unit that calculates the first evaluation value and the second evaluation value,
An update unit that obtains the deviation data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value and updates the autocorrelation value of the deviation data to another value.
A determination unit that determines whether or not the first evaluation value and the value based on the second evaluation value newly calculated by the calculation unit using the updated value converge.
The information processing apparatus according to any one of claims 1 to 6 , comprising the above. The update part
The information processing apparatus according to claim 7 , wherein a value based on the similarity in the column vector of the similarity matrix with respect to the deviation data is set to the other value. The update part
The information processing apparatus according to claim 8 , wherein the similarity in the column vector is rearranged in a predetermined order, and the predetermined degree of similarity is set to the other value. The sound data is sound data collected by a microphone.
The information processing apparatus according to any one of claims 1 to 9 , wherein the feature amount is a feature amount relating to a power spectrum in the frequency domain. The feature quantity is a basis vector obtained by applying a non-negative matrix factor decomposition to the power spectrum, a dispersion matrix having the same dimensions of the power spectrum, and a vector obtained by performing a mel filter bank analysis on the power spectrum. The information processing according to claim 10 , further comprising at least one of the coefficient vector obtained by the non-negative matrix factor division and the first principal component vector of the power spectrum reconstructed using the basis vector. apparatus. The classification unit
The information processing apparatus according to claim 11 , wherein the similarity included in the similarity matrix is calculated by weighting the similarity of each vector included in the feature amount or the variance matrix. A storage unit that stores each sound data of a different type in association with identification information indicating the type.
A search unit that searches for sound data stored in the storage unit, which matches the sound data in the center of the cluster generated by the classification unit, and a search unit.
An assigning unit that assigns a label including identification information associated with the searched sound data to the sound data at the center of the cluster, and
The information processing apparatus according to any one of claims 1 to 12 , further comprising. The analysis unit
The information processing apparatus according to claim 13 , wherein the plurality of sound data are analyzed by using the identification information included in the label. The sound data acquired by the acquisition unit is the sound data acquired by the microphone array.
The information processing apparatus according to claim 14 , wherein the label further includes the direction of the sound data and / or the generation time of the sound data. The analysis unit
The information processing device according to claim 15 , wherein the ratio of each identification information in the plurality of sound data is calculated and the ratio is used as an analysis result. The visualization unit
The information processing apparatus according to claim 16 , wherein the analysis result is graphed using the direction of the sound data and / or the generation time of the sound data. The analysis unit
A learning unit that performs ensemble learning using the labeled sound data and the feature amount.
The information processing apparatus according to claim 15 , further comprising a construction unit that constructs a model related to the sound data by learning by the learning unit. The visualization unit
The information processing apparatus according to claim 18 , wherein the analysis result analyzed by using the model is processed so as to be visualized on the display unit. An acquisition unit that acquires multiple sound data representing environmental sounds,
An extraction unit that extracts the features of each sound data,
A classification unit that classifies the plurality of sound data into each category using the extracted features, and
An analysis unit that analyzes the sound data based on each category classified by the classification unit using the identification information of the sound data, the direction of the sound data, and / or the generation time of the sound data.
A visualization unit that visualizes the analysis results analyzed by the analysis unit,
Equipped with a,
The classification unit
By repeating the update of each of the first evaluation value and the second evaluation value based on the similarity matrix between each sound data generated using the extracted feature data, a final cluster is generated and classified into each category. Then, the autocorrelation of the similarity matrix is set using the similarity with respect to the outlier data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value, and the first evaluation value is The cluster-centered candidate data is a value indicating the appropriateness of the member data belonging to the cluster to be the cluster center, and the second evaluation value is the appropriateness of the member data belonging to the cluster of the candidate data. An information processing system that is a value indicating . The computer
Acquiring multiple sound data representing environmental sounds and
Extracting the features of each sound data and
Using the extracted features to classify the plurality of sound data into each category,
Analyzing the sound data based on each of the classified categories using the identification information of the sound data and the direction of the sound data and / or the generation time of the sound data.
Visualizing the analyzed analysis results and
The execution,
The above classification is
By repeating the update of each of the first evaluation value and the second evaluation value based on the similarity matrix between each sound data generated using the extracted feature data, a final cluster is generated and classified into each category. Then, the autocorrelation of the similarity matrix is set using the similarity with respect to the outlier data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value, and the first evaluation value is The cluster-centered candidate data is a value indicating the appropriateness of the member data belonging to the cluster to be the cluster center, and the second evaluation value is the appropriateness of the member data belonging to the cluster of the candidate data. An information processing method that is a value indicating . On the computer
Acquiring multiple sound data representing environmental sounds and
Extracting the features of each sound data and
Using the extracted features to classify the plurality of sound data into each category,
Analyzing the sound data based on each of the classified categories using the identification information of the sound data and the direction of the sound data and / or the generation time of the sound data.
Visualizing the analyzed analysis results and
To execute ,
The above classification is
By repeating the update of each of the first evaluation value and the second evaluation value based on the similarity matrix between each sound data generated using the extracted feature data, a final cluster is generated and classified into each category. Then, the autocorrelation of the similarity matrix is set using the similarity with respect to the outlier data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value, and the first evaluation value is The cluster-centered candidate data is a value indicating the appropriateness of the member data belonging to the cluster to be the cluster center, and the second evaluation value is the appropriateness of the member data belonging to the cluster of the candidate data. A program that is a value that indicates . On the computer
Acquiring multiple sound data representing environmental sounds and
Extracting the features of each sound data and
Using the extracted features to classify the plurality of sound data into each category,
Analyzing the sound data based on each of the classified categories using the identification information of the sound data and the direction of the sound data and / or the generation time of the sound data.
Visualizing the analyzed analysis results and
To execute ,
The above classification is
By repeating the update of each of the first evaluation value and the second evaluation value based on the similarity matrix between each sound data generated using the extracted feature data, a final cluster is generated and classified into each category. Then, the autocorrelation of the similarity matrix is set using the similarity with respect to the outlier data in the cluster generated in the calculation process of the first evaluation value and the second evaluation value, and the first evaluation value is The cluster-centered candidate data is a value indicating the appropriateness of the member data belonging to the cluster to be the cluster center, and the second evaluation value is the appropriateness of the member data belonging to the cluster of the candidate data. A computer-readable recording medium on which a program is recorded, which is a value indicating .