JP2017041206A - Learning device, search device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To search for a multi-modal signal even when a part of the modal is missing.SOLUTION: Feature data is extracted by a target feature extraction part 40 with respect to each of inputted target signals to be single modal or multi-modal, target quantization data after converting the feature data of a target signal into quantization data using a code is acquired by a target feature quantizing part 42 on the basis of the extracted feature data of the target signal and the created conversion table with respect to the respective target signals, and attributes associated with accumulated quantization data corresponding to the target quantization data are searched for by a search part 44 in the created database on the basis of the target quantization data of the acquired target signal with respect to the respective target signals.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning device, a search device, a method, and a program for searching for a multimodal signal.

  Conventionally, as a method of searching for a multimodal signal, there is a method of searching a location similar to a target signal that is time-series data from an accumulated signal that is time-series data.

Japanese Patent No. 4358229

  However, the conventional method has a problem in that when a plurality of modals are used and a part of the modals are missing, a multimodal signal cannot be searched.

  In the present invention, a learning device, a search device, a method, and a program, which are made to solve the above-described problems, are capable of searching for a multimodal signal even when some modals are missing. The purpose is to provide.

  In order to achieve the above object, a learning device according to a first aspect of the present invention extracts a learning feature extraction unit that extracts feature data for each input learning signal that is multimodal, and the learning feature extraction unit extracts the learning data. Based on the feature data of each of the learning signals, feature data is extracted for each of the learning unit that creates a conversion table from the feature data to a common code and the input single-modal or multi-modal stored signal. For each of the accumulated signals, the feature data of the accumulated signal is extracted based on the feature data of the accumulated signal extracted by the accumulated feature extractor and the conversion table created by the learning unit. An accumulation feature quantization unit that obtains accumulated quantized data converted into quantized data using the code, and each of the accumulated signals A database creating unit that creates the database by associating the accumulated quantized data of the accumulated signal acquired by the accumulated feature quantizing unit with the attribute of the accumulated signal and registering them in a database. ing.

  A learning method according to a second invention is a learning method in a learning device including a learning feature extraction unit, a learning unit, an accumulation feature extraction unit, an accumulation feature quantization unit, and a database creation unit, The learning feature extraction unit extracts feature data for each of the input learning signals that are multimodal, and the learning unit, based on each feature data of the learning signal extracted in the learning feature extraction unit, A conversion table from the feature data to a common code is created, and the storage feature extraction unit extracts feature data for each input single-modal or multi-modal storage signal, and the storage feature quantization unit For each of the accumulated signals, based on the feature data of the accumulated signal extracted by the accumulated feature extracting unit and the conversion table created by the learning unit. And acquiring the accumulated quantized data obtained by converting the feature data of the accumulated signal into the quantized data using the code, and the database creating unit obtains each of the accumulated signals by the accumulated feature quantizing unit. The accumulated quantized data of the accumulated signal and the attribute of the accumulated signal are registered in association with each other to create the database.

  According to the first and second aspects, the learning feature extraction unit extracts feature data for each of the input multimodal learning signals, and the learning unit extracts the feature data of each of the extracted learning signals. Based on this, a conversion table from the feature data to a common code is created, and the storage feature extraction unit extracts the feature data for each of the input single-modal or multi-modal storage signals, and the storage feature quantization unit For each of the accumulated signals, based on the extracted accumulated signal feature data and the created conversion table, the accumulated quantized data obtained by converting the accumulated signal feature data into quantized data using a code is obtained, For each of the accumulated signals, the database creation unit associates the accumulated quantized data of the acquired accumulated signal with the attribute of the accumulated signal. Registered in the over scan, to create a database.

  In this way, feature data is extracted for each of the input learning signals that are multimodal, a conversion table is created based on the feature data of each of the extracted learning signals, and the input single modal or multimodal For each of the accumulated signals, the feature data is extracted, and for each of the accumulated signals, the accumulated quantized data is acquired based on the extracted feature data of the accumulated signal and the created conversion table. For each accumulated signal, the accumulated quantized data of the accumulated signal and the attribute of the accumulated signal are registered in the database in association with each other, and by creating a database, even if some modals are missing, it is multimodal. A database can be constructed that can search for signals.

  Further, in the learning device according to the first invention, the accumulated feature quantization unit is configured such that the feature data of the accumulated signal lacks data corresponding to the modal included in the multimodal of the learned signal. Acquiring the accumulated quantized data based on the feature data in which the missing portion of the feature data of the accumulated signal is zeroed and the conversion table, or the feature data of the accumulated signal and Based on the conversion table, the accumulated quantized data is obtained by ignoring data corresponding to the missing portion of the feature data stored in the conversion table, or the accumulation Based on the feature data in which the missing value of the feature data of the signal is filled with the representative value of the feature data of the corresponding learning signal and the conversion table, the accumulated amount Data may be acquired.

  According to a third aspect of the present invention, there is provided a search device according to a third aspect of the present invention, wherein a target feature extraction unit that extracts feature data for each input single-modal or multi-modal target signal and the target feature extraction unit for each of the target signals The target quantized data obtained by converting the feature data of the target signal into quantized data using the code based on the extracted feature data of the target signal and the conversion table created in the learning device according to claim 1. From the database created in the learning device based on the target quantization data of the target signal acquired by the target feature quantization unit for each of the target signals, A search unit that searches for the attribute associated with the accumulated quantized data corresponding to the target quantized data. It is.

  A search method according to a fourth aspect of the present invention is a search method in a search device including an objective feature extraction unit, an objective feature quantization unit, and a search unit, wherein the objective feature extraction unit is an input single modal. Alternatively, feature data is extracted for each target signal that is multimodal, and the target feature quantizing unit extracts, for each of the target signals, feature data of the target signal extracted by the target feature extracting unit; Based on the conversion table created in the learning method of the invention, the target quantized data obtained by converting the feature data of the target signal into quantized data using the code is obtained, and the search unit includes the target signal For each of the above, from the database created in the learning device, based on the target quantization data of the target signal acquired by the target feature quantization unit, Searching the attribute associated with the storage quantized data corresponding to the serial object quantized data.

  According to the third and fourth inventions, the target feature extraction unit extracts feature data for each of the input single-modal or multi-modal target signals, and the target feature quantization unit extracts each of the target signals. The target quantization data is acquired based on the extracted feature data of the target signal and the conversion table created in the learning device according to the first invention, and is acquired for each of the target signals by the search unit. Based on the target quantized data of the target signal, an attribute associated with the accumulated quantized data corresponding to the target quantized data is searched from a database created in the learning device.

  In this way, feature data is extracted for each input single-modal or multi-modal target signal, and for each target signal, the extracted target signal feature data and the created conversion table are used. The target quantized data is acquired, and each of the target signals is associated with the accumulated quantized data corresponding to the target quantized data from the created database based on the target quantized data of the acquired target signal. By searching for attributes, it is possible to search for multimodal signals even if some modals are missing.

  Further, in the search device according to the third invention, the target feature quantization unit includes a case where data corresponding to a modal included in the multimodal of the learning signal is missing in the feature data of the target signal. Obtaining the target quantized data based on the feature data in which the missing part of the feature data of the target signal is zeroed and the conversion table, or the feature data of the target signal and Based on the conversion table, the target data is obtained by ignoring data corresponding to the missing portion of the feature data stored in the conversion table, or Based on the feature data in which the missing value of the feature data of the signal is filled with the representative value of the feature data of the corresponding learning signal and the conversion table, the target amount Data may be acquired.

  In the search device according to the third invention, the learning signal includes two or more sensor data or media data, the accumulated signal includes one or more sensor data or media data, and the target signal is a sensor One or more data or media data may be included.

  Moreover, the program of this invention is a program for functioning a computer as each part which comprises said learning apparatus or search apparatus.

  As described above, according to the learning device, method, and program of the present invention, feature data is extracted for each input multimodal learning signal, and based on each feature data of the extracted learning signal. Then, a conversion table is created, and feature data is extracted for each of the inputted single-modal or multi-modal accumulated signals, and for each accumulated signal, the extracted accumulated signal feature data, the created conversion table, Based on the above, the accumulated quantization data is acquired, and the database creation unit registers the accumulated quantization data of the acquired accumulated signal and the attribute of the accumulated signal in the database in association with each accumulated signal, and creates the database. To search for multimodal signals even if some modals are missing It can be constructed over the nest.

  In addition, according to the search device, method, and program of the present invention, feature data is extracted for each input target signal that is single-modal or multi-modal, and the characteristics of the extracted target signal for each target signal. The target quantization data is acquired based on the data and the created conversion table, and the target quantization is performed for each target signal from the database created based on the target quantization data of the acquired target signal. By searching for the attribute associated with the accumulated quantized data corresponding to the data, a multimodal signal can be searched even if some modals are missing.

It is a block diagram which shows the functional structure of the multimodal signal search apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart figure of the learning signal processing routine in the multimodal signal search device which concerns on the 1st Embodiment of this invention. It is a flowchart figure of the accumulation signal processing routine in the multimodal signal search device concerning a 1st embodiment of the present invention. It is a flowchart figure of the search process routine in the multimodal signal search device which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the functional structure of the multimodal signal search apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart figure of the accumulation | storage signal processing routine in the multimodal signal search apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart figure of the search process routine in the multimodal signal search apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the functional structure of the multimodal signal search apparatus which concerns on the 3rd Embodiment of this invention. It is a flowchart figure of the accumulation | storage signal processing routine in the multimodal signal search apparatus which concerns on the 3rd Embodiment of this invention. It is a flowchart figure of the search process routine in the multimodal signal search device which concerns on the 3rd Embodiment of this invention. It is a figure which shows an example of the data collection using a wearable modal. It is a figure which shows an example of the data collection using the modal installed outside. It is a figure which shows an example of an experimental result. It is a figure which shows an example of an experimental result.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Multimodal Signal Searching Device According to First Embodiment of the Present Invention>
First, the configuration of the multimodal signal search apparatus according to the first embodiment of the present invention will be described. As shown in FIG. 1, the multimodal signal search apparatus 100 according to the first embodiment of the present invention executes a CPU, a RAM, a learning signal processing routine, an accumulated signal processing routine, and a search processing routine, which will be described later. And a computer including a ROM storing various programs and various data. As shown in FIG. 1, this multimodal signal search device 100 functionally includes a learning signal acquisition unit 10, an accumulated signal acquisition unit 12, a target signal acquisition unit 14, a calculation unit 20, and an output unit 90. It is configured to include.

  The learning signal acquisition unit 10 acquires at least two or more multimodal signals used for learning (hereinafter referred to as learning signals) and outputs the acquired signals to the learning feature extraction unit 22. Here, the multimodal signal is, for example, audio signal data (or acoustic signal data) collected using a microphone, image signal data captured using a camera, a wearable or environment-installed sensor, specifically Includes acceleration sensors, gyro sensors, geomagnetic sensors, illuminance sensors, pressure sensors, proximity sensors, temperature sensors, humidity sensors, heart rate / electrocardiographs, barometric pressure sensors, GPS, and depth sensors, Sensor signal data such as acceleration, geomagnetism, illuminance, pressure, proximity, temperature, humidity, heart rate / electrocardiogram, barometric pressure, GPS data, and depth map. A sentence may also be used as one modal. In this case, character data is used as signal data. In the first embodiment, audio signal data and image signal data are used as multimodal learning signals. In addition, sensor signal data collected using a measuring device is sensor data, and character data is a kind of media data. The sensor signal data is not limited to the above example, and other sensor signal data may be used. The media data is not limited to the character data, and other media data may be used.

  The accumulated signal acquisition unit 12 acquires at least one single-modal or multi-modal signal (hereinafter referred to as an accumulated signal) accumulated in a database, which will be described later, and outputs the acquired signal to the accumulated feature extraction unit 30. In the first embodiment, the accumulated signal is the same as the modal acquired as the learning signal and the same number of modals, and a part of each of the modal acquired as the learning signal. Each missing modal may be acquired. In the first embodiment, it is assumed that attribute data corresponding to each accumulated signal is added to each accumulated signal. If the correspondence between attribute data and accumulated data can be clarified, the attribute data is not limited to being added, and attribute data may be separately input to the attribute assigning unit 34. Here, the attribute data is information that is effective for representing the data. For example, the attribute data represents a description, a tag, or the like regarding the environment in which the data is acquired or the contents of the data. For example, in the case of dance data, information relating to dance technique and composition, performer type, or the like is applicable. Further, the accumulated signal itself may be used as attribute data. For example, in the case of dance data, an image signal obtained by photographing and recording a dance, and acoustic signal data may be used as attribute data.

  For example, in the first embodiment, since the learning signal is multimodal including a modal of audio signal data and a modal of image signal data, the same modal of audio signal data as an accumulated signal, There are a case of acquiring a multimodal including a modal of image signal data and a case of acquiring one of a modal of audio signal data and a modal of image signal data.

  The target signal acquisition unit 14 acquires at least one single-modal or multi-modal signal (hereinafter referred to as a target signal) serving as a query, and outputs the acquired signal to the target feature extraction unit 40. As for the target signal, as in the case of the accumulated signal described above, the same kind and the same number of modals as the learning signal are acquired, and a part of each of the modals acquired as the learning signal. In some cases, each of the modals lacking is acquired.

  The calculation unit 20 includes a learning feature extraction unit 22, a learning unit 24, a conversion table storage unit 26, an accumulation feature extraction unit 30, an accumulation feature quantization unit 32, an attribute assignment unit 34, and a database creation unit 36. The database storage unit 38, the target feature extraction unit 40, the target feature quantization unit 42, and the search unit 44 are configured.

  For each of the learning signals input from the learning signal acquisition unit 10, the learning feature extraction unit 22 extracts feature data from the learning signal and outputs the feature data to the learning unit 24. Specifically, feature data is extracted for each modal included in the learning signal. Note that each piece of data included in the extracted feature data is a minimum unit necessary for processing in the learning unit 24 described later.

  Here, extracting feature data means that for audio signal data (or acoustic signal data), first, the signal data is resampled at a designated sampling frequency. For example, resampling is performed with a sampling frequency of 8000 Hz. At this time, as preprocessing, for example, high frequency emphasis may be performed by pre-emphasis with a coefficient of 0.76. Thereafter, a process of cutting out a signal with a constant window width is performed while shifting by a constant interval. For example, the window width is 1024 samples and the shift width is 100 samples. Then, a short-time frequency spectrum is obtained by performing discrete Fourier transform on the cut out individual signal data. Since the frequency spectrum of the audio signal often includes noise in the low frequency region, a part of the obtained frequency spectrum may be used. For example, 65th data to 512th data from the low frequency region are used. A time series vector is acquired by arranging the short-time frequency spectra obtained here in the time series direction. In the first embodiment, an example in which discrete Fourier transform is performed has been described, but discrete cosine transform may be used as a conversion method to a power spectrum. Further, as the feature data, other known methods such as a spectral envelope obtained from audio signal data, time change information of the fundamental frequency, and the like may be used.

  For image signal data, first, as preprocessing, a moving image is resampled at a specified frame rate. For example, resampling is performed with a frame rate of 15. At this time, in order to improve the processing speed of feature extraction, image size reduction processing may be executed. For example, the image is reduced to 48 pixels in the vertical direction and 64 pixels in the horizontal direction. Next, for each image in the moving image, the image area is divided into blocks at regular intervals, and an average value is calculated for each of RGB within each block, thereby obtaining a feature vector for each image. As a parameter, it is divided into 12 in the vertical direction and 16 in the horizontal direction. In the case of a moving image, since image signal data continues over time, the feature data extraction is applied to each image and connected to obtain a time series vector. In the first embodiment, an example in which RGB data in a block area is used has been described. In addition, Scale-Invariant Feature Transform (SIFT) (Non-Patent Document 1: David G. Lowe. Local recognition such as Object recognition from local scale-invariant features. In Proceedings of the International Conference on Computer Vision, 1999.) may be used as image feature data.

  For 9-axis sensor data obtained by acceleration sensors, gyro sensors, and geomagnetic sensors, or for heart rate / electrocardiographic data obtained using a heart rate / electrocardiograph, feature data is extracted for each axis. I do. First, resampling is performed at a constant sampling frequency as preprocessing. For example, resampling is performed with a sampling frequency of 200 Hz. In addition to the above processing, as a preprocessing, a filtering process such as smoothing may be performed to perform a noise removal process. Next, a process of cutting out a signal with a constant window width on the time axis is performed while shifting the signal from the front end to the end of the signal. As parameters, for example, the window width is 1 second and the shift width is 1 second. A short-time frequency spectrum is obtained by performing discrete cosine transform on each cut out signal data. A time series vector is acquired by arranging the short-time frequency spectra obtained here in the time series direction. In the first embodiment, an example using discrete cosine transform has been described, but discrete Fourier transform may be performed as a conversion method to a power spectrum. Further, the peak position information of the sensor signal data may be used as feature data. In the above example, feature data is extracted for each axis. However, in the case of a 3-axis sensor, the magnitude of the sensor value is obtained by taking the root of the square sum of the sensor data for each axis. The same processing as described above may be performed for the value.

  In addition, for depth data obtained from the depth sensor, feature data is obtained by obtaining a human skeleton model using depth data and performing frequency analysis on the trajectory data of each joint using a known method. To extract. For character data in a sentence, in the case of English, word information appearing in the sentence is acquired as feature data based on information such as spaces, periods, and commas that are character breaks. Also, for illuminance data, pressure data, proximity data, temperature data, humidity data, atmospheric pressure data, and GPS data obtained from illuminance sensor, pressure sensor, proximity sensor, temperature sensor, humidity sensor, atmospheric pressure sensor, and GPS Then, arbitrary feature data is extracted using a known method.

  Note that the feature data extraction method described above is not particularly limited, and other known methods may be used. In addition, since the scale of the feature data is different for each modal, the feature data may be centered or normalized as a post-processing of the feature data extraction to reduce the difference between the modals.

  The learning unit 24 converts the feature data into a common code (or number) in the modal combination acquired as the learning signal based on the feature data of each learning signal input from the learning feature extraction unit 22. A table is created and stored in the conversion table storage unit 26. In the first embodiment, a conversion table is created for combinations of modal audio signals and modal image signals included in each learning signal. In the first embodiment, a case has been described in which a conversion table is created for a combination of two modals: a modal of an audio signal and a modal of an image signal. However, the present invention is not limited to this. For example, the number of modals used for the combination of modals for creating the conversion table is not limited. This means that two modals may be combined like voice feature data and image feature data, and three modals may be combined like voice feature data, image feature data, and acceleration feature data. Therefore, it is possible to create a conversion table corresponding to any modal combination. Note that modal combinations corresponding to the conversion tables created in the learning unit 24 are defined in advance.

Specifically, the conversion table is created, for example, by obtaining a representative vector V _k by an LBG algorithm based on the K-means method, which is a known method, and assigning a number k to the representative vector. Therefore, the conversion table can be a table for converting a feature vector close to the representative vector V _k into a number k. Here, k = 1, 2,..., K, where K represents the number of representative vectors, for example, K = 100. Although there are a plurality of K-means methods, for example, the Elkan algorithm is used. Further, since the K-means method depends on the initial value, it is necessary to set the initial value. For this, for example, a random value is used. In K-means, an iterative process is performed from the initial value until convergence, and the upper limit of the number of iterations is 50, for example. In the first embodiment, if the dimension of the feature data of the voice modal is D _x and the dimension of the feature data of the image modal is D _y , the dimension of the representative vector V _k in the first embodiment is , D _x + D _y . Here, the priority order of each modal is defined in advance, and in the first embodiment, it is defined in advance that the elements of the image modal are arranged after the audio modal. Therefore, the dimension of the representative vector V _k is D _x + D _y .

The conversion table storage unit 26 stores a conversion table created by the learning unit 24. In the first embodiment, since K = 100, it is assumed that 100 combinations of representative vectors V _k and numbers k are stored.

  The accumulated feature extraction unit 30 extracts feature data from each accumulated signal input from the accumulated signal acquisition unit 12 and outputs the feature data to the accumulated feature quantization unit 32. Note that the method for extracting feature data from the accumulated signal in the accumulated feature extraction unit 30 is the same as that of the learning feature extraction unit 22 described above, and thus detailed description thereof is omitted.

The accumulated feature quantization unit 32 determines, for each accumulated signal, the accumulated signal based on the feature data of the accumulated signal extracted by the accumulated feature extracting unit 30 and the conversion table stored in the conversion table storage unit 26. Is converted into quantized data, and accumulated quantized data is generated based on each of the converted quantized data and output to the attribute assigning unit 34. Here, the quantization uses number k corresponding to the closest representative vectors of the representative vector V _k described above as quantized values (quantized data). If at least one modal among the modal combinations that are the objects of the conversion table is missing in the target accumulated signal, the feature data of the accumulated signal corresponding to the missing modal. Fills the value of zero with zero.

Specifically, data obtained by combining the minimum processing unit data of the modals from the head of the feature data of the modals included in the target accumulated signal is set as the process feature data. Further, the distance between the processing feature data and each representative vector V _k included in the conversion table is calculated, the representative vector V _k having the smallest distance is determined, and _k corresponding to the determined representative vector V k is converted. Obtained as quantized data from the table. This processing is repeated from the beginning of each modal feature data included in the accumulated signal to a range where processing is possible for each minimum processing unit. Then, based on each k value acquired in the iterative process, a histogram for the k value is created as accumulated quantized data of the accumulated signal. The unit of the histogram represents, for example, a probability distribution, and a value obtained by dividing each number of k by the total number of acquired k is used.

For example, a case where one processing feature data is converted into quantized data will be described in the case where the conversion table is created by combining two modals. When the combination of modals to be converted is two modals, if the dimension of the feature data of _one modal M ₁ is D ₁ and the dimension of the feature data of the other modal M ₂ is D ₂ , the conversion table The dimension of the representative vector V _k is D ₁ + D ₂ (predefined that modal M ₁ is followed by modal M ₂ ). When the modal included in the accumulated signal matches each of the modals to be converted, the dimension of the vector W _t at a certain time t in the accumulated feature data is D ₁ + D ₂ , and the representative vector V since matches the dimensions of _k, the distance between W _t and V _k calculated directly, by obtaining the k such that the value is the smallest, it is possible to acquire the quantized data. Here, when calculating the distance, for example, the L2 distance is used. In addition, any known distance evaluation scale such as L1 distance or Hamming distance may be used. When the modal is insufficient, for example, when the _first modal M ₁ is missing, the dimension of the t-th vector W _t ⁽²⁾ in the accumulated feature data is D ₂ dimensional, and the representative vector Compared to V _k , the first D ₁ dimension is missing. In the first embodiment, the missing portion is dealt with by filling in zeros. That is, by calculating the distance between the vector in which one vector of D _{1 is} arranged and W _t ⁽²⁾ and the representative vector V _k, and obtaining _k that minimizes the value, quantization is performed. Get the data. In the above example, the case where the first modal is lost has been described. However, the case where the second modal is lost can also be handled by performing the same process.

  For each of the accumulated signals, the attribute assigning unit 34 associates the accumulated quantized data of the accumulated signal input from the accumulated feature quantizing unit 32 with the attribute data of the accumulated signal and outputs the associated data to the database creating unit 36. .

  The database creation unit 36 registers a combination of accumulated quantized data and attribute data for each accumulated signal input from the attribute assigning unit 34 in a database stored in the database storage unit 38.

  The database storage unit 38 stores a database in which each combination of accumulated quantized data and attribute data is stored.

  The target feature extraction unit 40 extracts feature data from the target signal for each of the target signals input from the target signal acquisition unit 14 and outputs the feature data to the target feature quantization unit 42. Note that the method of extracting feature data from the target signal in the target feature extraction unit 40 is the same as that of the learning feature extraction unit 22 described above, and thus detailed description thereof is omitted.

  The target feature quantization unit 42, for each target signal, based on the feature data of the target signal extracted by the target feature extraction unit 40 and the conversion table stored in the conversion table storage unit 26. The feature data for each minimum processing unit included in the signal is converted into quantized data, target quantized data is generated based on each of the converted quantized data, and is output to the search unit 44. The method for converting the feature data of the target signal into the target quantized data is the same processing as the method for converting the feature data of the stored signal into the stored quantized data in the stored feature quantizing unit 32 described above. The detailed explanation is omitted.

  The search unit 44 determines, for each target signal, the attribute of the target signal based on the target quantization data of the target signal acquired by the target feature quantization unit 42 and the database stored in the database storage unit 38. And the search result is output from the output unit 90.

  Specifically, for each target signal, the degree of coincidence between the histogram that is the target quantized data of the target signal and each histogram that is the accumulated quantized data included in the database is calculated. When the predetermined threshold is exceeded, it is determined that both histograms match, and attribute data corresponding to accumulated quantized data determined to match is acquired from the database, and the attribute data is used as attribute data of the target signal. Output from the output unit 90. Here, the degree of coincidence is evaluated by, for example, the L1 distance. This coincidence evaluation method is not limited to a specific distance evaluation scale, and any known distance evaluation scale such as an L2 distance or a Hamming distance may be used. Further, with respect to the accumulated quantized data and the assigned attribute, the discriminant function is learned in advance using logistic regression, a support vector machine or the like, and the attribute data corresponding to the target quantized data is used using the learned evaluation function. May be output. Further, when it is determined that the plurality of accumulated quantized data match, the attribute data corresponding to the accumulated quantized data having the highest degree of coincidence (for example, the shortest distance) is output from the output unit 90. Alternatively, the output unit 90 may output the result of rearranging the corresponding attribute data in ascending order of distance. Further, the output unit 90 may output the result of rearranging the attribute data corresponding to the calculated matching degree order without using the threshold value.

<Operation of Multimodal Signal Searching Device According to First Embodiment of the Present Invention>
Next, the operation of the multimodal signal search apparatus 100 according to the first embodiment of the present invention will be described. In the multimodal signal search device 100, when each learning signal is acquired by the learning signal acquisition unit 10, the multimodal signal search device 100 executes a learning signal processing routine shown in FIG. Further, when the multimodal signal search device 100 receives the accumulated signal by the accumulated signal acquisition unit 12, the multimodal signal search device 100 executes the accumulated signal processing routine shown in FIG. Further, when the multimodal signal search apparatus 100 receives the target signal by the target signal acquisition unit 14, the multimodal signal search apparatus 100 executes a search processing routine shown in FIG.

  First, the learning signal processing shown in FIG. 2 will be described.

  First, in step S100 of the learning signal processing routine shown in FIG. 2, a learning signal to be processed is determined from each of the received learning signals.

  Next, in step S102, feature data is extracted for the learning signal to be processed.

  Next, in step S104, it is determined whether or not the processing in step S102 has been completed for all received learning signals. If it is determined that the processing in step S102 has been completed for all accepted learning signals, the learning signal processing proceeds to step S106. On the other hand, if it is determined that the processing of step S102 has not been completed for all received learning signals, the learning signal processing proceeds to step S100, changes the learning signal to be processed, and steps S102 to S102. The process up to step S104 is repeated.

  Next, in step S106, a conversion table is created based on the feature data of each of the accepted learning signals acquired in step S102, stored in the conversion table storage unit 26, and the learning signal processing routine ends.

  Next, the accumulated signal processing routine shown in FIG. 3 will be described.

  First, in step S120 of the accumulated signal processing routine shown in FIG. 3, the conversion table stored in the conversion table storage unit 26 is read.

  Next, in step S122, an accumulated signal to be processed is determined from each of the accepted accumulated signals.

  Next, in step S124, feature data is extracted from the accumulated signal to be processed in the same manner as in step S102 described above.

  Next, in step S126, for the accumulated signal to be processed, the minimum unit to be processed is determined from the feature data acquired in step S124.

  Next, in step S128, it is determined whether or not the accumulated signal to be processed includes all the modals to be converted in the conversion table acquired in step S106 described above. When the accumulated signal includes all the modals to be processed, the accumulated signal processing moves to step S132. On the other hand, when the accumulated signal does not include all the target modals (partially missing), the accumulated signal processing proceeds to step S130.

  Next, in step S130, zero is filled in the element of the part corresponding to the missing part of the feature data of the minimum unit to be processed acquired in step S124.

  Next, in step S132, the minimum unit to be processed based on the feature data of the minimum unit to be processed acquired in step S124 or padded with zeros in step S130 and the conversion table acquired in step S120. The value of k which is the quantized data corresponding to is determined.

  Next, in step S134, it is determined whether or not the processing from step S128 to step S132 has been completed for all the minimum units of the accumulated signal to be processed. If it is determined that the processing from step S128 to step S132 has been completed for all the minimum units, the accumulated signal processing proceeds to step S136. On the other hand, if it is determined that the processing from step S128 to step S132 has not been completed for all the minimum units, the accumulated signal processing proceeds to step S126, changes the minimum unit to be processed, and step S128. The process up to step S134 is repeated.

  Next, in step S136, for the accumulated signal to be processed, accumulated quantized data is generated based on each value of k for each minimum unit included in the accumulated signal acquired in step S132.

  Next, in step S138, for the accumulated signal to be processed, the accumulated quantized data acquired in step S136 is associated with the attribute data added to the accumulated signal.

  Next, in step S140, the combination of the accumulated quantized data and the attribute data acquired in step S138 is stored in the database stored in the database storage unit 38.

  Next, in step S142, it is determined whether or not the processing from step S124 to step S140 has been completed for all received accumulated signals. If it is determined that the processing from step S124 to step S140 has been completed for all the accumulated signals, the accumulated signal processing routine ends. On the other hand, if it is determined that the processing from step S124 to step S140 has not been completed for all the accumulated signals, the accumulated signal processing routine proceeds to step S122, changes the accumulated signal to be processed, The processing from step S124 to step S142 is repeated.

  Next, the search processing routine shown in FIG. 4 will be described.

  First, in step S150 of the search processing routine shown in FIG. 4, the conversion table stored in the conversion table storage unit 26 is read.

  Next, in step S152, the database stored in the database storage unit 38 is read.

  Next, in step S154, a target signal to be processed is determined from each of the received target signals.

  Next, in step S156, feature data is extracted for the target signal to be processed in the same manner as in step S102 described above.

  Next, in step S158, for the target signal to be processed, the minimum unit to be processed is determined from the feature data acquired in step S156.

  Next, in step S160, it is determined whether or not the target signal to be processed includes all modals that are targets of the conversion table acquired in step S106 described above. When the target signal includes all the modals to be processed, the search process proceeds to step S164. On the other hand, when the target signal does not include all of the target modals (partially missing), the search process proceeds to step S162.

  Next, in step S162, zeros are filled in the elements of the part corresponding to the missing part of the feature data of the minimum unit to be processed acquired in step S156.

  Next, in step S164, the processing is performed based on the minimum unit feature data acquired in step S156 or zero-filled in step S162 and the conversion table acquired in step S150 described above. The value of k that is quantized data corresponding to the minimum unit is determined.

  Next, in step S166, it is determined whether or not the processing from step S160 to step S164 has been completed for all the minimum units of the target signal to be processed. If it is determined that the processes from step S160 to step S162 have been completed for all the minimum units, the search process proceeds to step S168. On the other hand, if it is determined that the processes from step S160 to step S164 have not been completed for all the minimum units, the search process proceeds to step S158, changes the minimum unit to be processed, and steps S160 to S160. The processing up to step S166 is repeated.

  Next, in step S168, for the target signal to be processed, target quantized data is generated based on each value of k for each minimum unit included in the target signal acquired in step S164.

  Next, in step S170, attribute data corresponding to the target signal is obtained based on the target quantization data of the target signal acquired in step S168 and the database acquired in step S152 for the target signal to be processed. Explore.

  Next, in step S172, for the target signal to be processed, the attribute data acquired in step S170 is output from the output unit 90 as a search result.

  Next, in step S174, it is determined whether or not the processing from step S156 to step S172 has been completed for all received target signals. When it is determined that the processing from step S156 to step S172 has been completed for all target signals, the search processing routine is completed. On the other hand, if it is determined that the processing from step S156 to step S172 has not been completed for all target signals, the search processing routine proceeds to step S154, changes the target signal to be processed, The processing from S156 to step S174 is repeated.

  As described above, according to the multimodal signal search apparatus according to the first embodiment of the present invention, feature data is extracted for each of the input single modal or multimodal target signals, and the target signal For each target signal, the target quantization data is acquired based on the extracted target signal feature data and the created conversion table, and for each target signal, based on the acquired target quantization data of the target signal, By searching the created database for an attribute associated with the accumulated quantized data corresponding to the target quantized data, a multimodal signal can be searched even if some modals are missing.

  In addition, single-modal or multi-modal signals can be quantized using a common code conversion table for all combinations of target modals, enabling search even if some modals are missing. , Corresponding attribute data can be acquired.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  Next, a multimodal signal search apparatus according to the second embodiment will be described.

  In the second embodiment, when the accumulated signal or a part of the target signal is deficient, the accumulated quantized data and the target quantized data are generated by ignoring the deficient part. Different from the first embodiment. In addition, about the structure and effect | action similar to the multimodal signal search apparatus which concerns on 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

<Configuration of Multimodal Signal Searching Device According to Second Embodiment>
Next, the configuration of the multimodal signal search apparatus according to the second embodiment of the present invention will be described. As shown in FIG. 5, the multimodal signal search apparatus 200 according to the second embodiment of the present invention executes a CPU, a RAM, a learning signal processing routine, an accumulated signal processing routine, and a search processing routine, which will be described later. And a computer including a ROM storing various programs and various data. The multimodal signal search apparatus 200 functionally includes a learning signal acquisition unit 10, an accumulation signal acquisition unit 12, a target signal acquisition unit 14, a calculation unit 220, and an output unit 90 as shown in FIG. It is configured to include.

    The calculation unit 220 includes a learning feature extraction unit 22, a learning unit 24, a conversion table storage unit 26, an accumulation feature extraction unit 30, an accumulation feature quantization unit 232, an attribute assignment unit 34, and a database creation unit 36. The database storage unit 38, the target feature extraction unit 40, the target feature quantization unit 242, and the search unit 44 are included.

  For each of the accumulated signals, the accumulated feature quantization unit 232 performs the accumulation based on the feature data of the accumulated signal extracted by the accumulated feature extraction unit 30 and the conversion table stored in the conversion table storage unit 26. The feature data for each minimum processing unit included in the signal is converted into quantized data, accumulated quantized data is generated based on each of the converted quantized data, and is output to the attribute assigning unit 34. If at least one or more modal combinations are missing from the modal combination that is the target of the conversion table in the target accumulated signal, the portion of the feature data of the accumulated signal corresponding to the missing modal Is ignored.

For example, a case where one processing feature data is converted into quantized data will be described in the case where the conversion table is created by combining two modals. When the combination of modals to be converted is two modals, if the dimension of the feature data of _one modal M ₁ is D ₁ and the dimension of the feature data of the other modal M ₂ is D ₂ , the conversion table The dimension of the representative vector V _k is D ₁ + D ₂ (predefined that modal M ₁ is followed by modal M ₂ ). When the modal included in the accumulated signal matches each of the modals to be converted, the dimension of the t-th vector W _t in the accumulated feature data is D ₁ + D ₂ , and the representative vector V _k Therefore, quantized data can be acquired by calculating the distance between W _t and V _k as it is and obtaining k that minimizes the value. If there is a lack of modalities, for example, if the _first modal M ₁ is missing, the dimension of the vector W _t ⁽²⁾ at a certain time t in the accumulated feature data is D ₂ dimensional and is representative. Compared to the vector V _k , the first D ₁ dimension is missing. In the second embodiment, the missing portion is dealt with by ignoring it. That is, the quantized data is acquired by calculating Wt ⁽²⁾ and the distance from the D ₁ +1 dimensional dimension of V _k to the D ₁ + D ₂ dimensional dimension and obtaining k that minimizes the value. . In the above example, the case where the first modal is lost has been described. However, the case where the second modal is lost can also be handled by performing the same process.

  For each target signal, the target feature quantization unit 242 determines the target signal based on the target signal feature data extracted by the target feature extraction unit 40 and the conversion table stored in the conversion table storage unit 26. The feature data for each minimum processing unit included in the signal is converted into quantized data, target quantized data is generated based on each of the converted quantized data, and is output to the search unit 44. The method for converting the feature data of the target signal into the target quantized data is the same as the method for converting the feature data of the stored signal into the stored quantized data in the stored feature quantizing unit 232 described above. The detailed explanation is omitted.

  Note that the other configurations of the multimodal signal search apparatus according to the second embodiment are the same as the configurations of the multimodal signal search apparatus according to the first embodiment, and thus the description thereof is omitted.

<Operation of Multimodal Signal Searching Device According to Second Embodiment of the Present Invention>
Next, the operation of the multimodal signal search apparatus 200 according to the second embodiment of the present invention will be described. In the multimodal signal search device 200, when each of the learning signals is acquired by the learning signal acquisition unit 10, the learning signal processing routine shown in FIG. 2 is executed by the multimodal signal search device. Further, when the multimodal signal search device 200 receives the accumulated signal by the accumulated signal acquisition unit 12, the multimodal signal search device 200 executes the accumulated signal processing routine shown in FIG. Further, when the multimodal signal search apparatus 200 receives the target signal by the target signal acquisition unit 14, the multimodal signal search apparatus 200 executes a search processing routine shown in FIG. Note that the learning signal processing routine according to the second embodiment is the same as the learning signal processing routine according to the first embodiment, and a description thereof will be omitted.

  First, the accumulated signal processing routine shown in FIG. 6 will be described.

  In step S200 of the accumulated signal processing routine shown in FIG. 6, the feature corresponding to the missing modal based on the feature data of the minimum unit to be processed acquired in step S124 and the conversion table acquired in step S120. The value of k which is quantized data is determined ignoring the data portion.

  In step S202, based on the minimum unit feature data to be processed acquired in step S124 and the conversion table acquired in step S120, the value of k, which is quantized data corresponding to the minimum unit to be processed, is set. decide.

  Note that other processes of the accumulation signal processing routine according to the second embodiment are the same as those of the accumulation signal processing routine according to the first embodiment, and a description thereof will be omitted.

  Next, the search processing routine shown in FIG. 7 will be described.

  In step S220 of the search processing routine shown in FIG. 7, the feature data corresponding to the missing modal based on the feature data of the minimum unit to be processed acquired in step S156 and the conversion table acquired in step S150. The value of k, which is quantized data, is determined by ignoring the part.

  In step S222, based on the feature data of the minimum unit to be processed acquired in step S156 and the conversion table acquired in step S150, the value of k that is quantized data corresponding to the minimum unit to be processed is determined. decide.

  Note that other processes of the search process routine according to the second embodiment are the same as those of the search process routine according to the first embodiment, and thus the description thereof is omitted.

  As described above, according to the multimodal signal search device according to the second embodiment of the present invention, feature data is extracted for each of the input single modal or multimodal target signals, and the target signal For each, based on the extracted feature data of the target signal and the created conversion table, if part of the target signal is missing, the target quantized data is acquired so as to ignore that part. For each target signal, by searching an attribute associated with the accumulated quantized data corresponding to the target quantized data from the created database based on the target quantized data of the acquired target signal Even if some modals are missing, a multimodal signal can be searched.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  Next, a multimodal signal search apparatus according to the third embodiment will be described.

  In the third embodiment, when a defect occurs in a part of the accumulated signal or the target signal, the accumulated quantized data is filled with the representative value of the feature data of the corresponding learning signal in the missing part. And the point which produces | generates target quantization data differs from 1st, 2nd embodiment. In addition, about the structure and effect | action similar to the multimodal signal search apparatus which concerns on 1st, 2nd embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

<Configuration of Multimodal Signal Searching Device According to Third Embodiment>
Next, the configuration of the multimodal signal search apparatus according to the third embodiment of the present invention will be described. As shown in FIG. 8, a multimodal signal search apparatus 300 according to the third embodiment of the present invention executes a CPU, a RAM, a learning signal processing routine, an accumulation signal processing routine, and a search processing routine, which will be described later. And a computer including a ROM storing various programs and various data. Functionally, the multimodal signal search apparatus 300 includes a learning signal acquisition unit 10, an accumulated signal acquisition unit 12, a target signal acquisition unit 14, a calculation unit 320, and an output unit 90 as shown in FIG. It is configured to include.

    The calculation unit 320 includes a learning feature extraction unit 22, a learning unit 24, a conversion table storage unit 26, an accumulation feature extraction unit 30, an accumulation feature quantization unit 332, an attribute assignment unit 34, and a database creation unit 36. The database storage unit 38, the target feature extraction unit 40, the target feature quantization unit 342, and the search unit 44 are configured.

  The accumulation feature quantization unit 332 performs the accumulation for each accumulation signal based on the feature data of the accumulation signal extracted by the accumulation feature extraction unit 30 and the conversion table stored in the conversion table storage unit 26. The feature data for each minimum processing unit included in the signal is converted into quantized data, accumulated quantized data is generated based on each of the converted quantized data, and is output to the attribute assigning unit 34. If at least one modal among the modal combinations that are targets of the conversion table is missing in the target accumulated signal, the feature data representative of the learning signal corresponding to the missing modal Fill in the value. The representative value is one of the basic statistics and represents the entire distribution as a single number, and includes, for example, an average value, a central location, a mode value, a minimum value, and a maximum value.

For example, a case where one processing feature data is converted into quantized data will be described in the case where the conversion table is created by combining two modals. When the combination of modals to be converted is two modals, if the dimension of the feature data of _one modal M ₁ is D ₁ and the dimension of the feature data of the other modal M ₂ is D ₂ , the conversion table The dimension of the representative vector V _k is D ₁ + D ₂ (predefined that modal M ₁ is followed by modal M ₂ ). When the modal included in the accumulated signal matches each of the modals to be converted, the dimension of the vector W _t at a certain time t in the accumulated feature data is D ₁ + D ₂ , and the representative vector V since matches the dimensions of _k, the distance between W _t and V _k calculated directly, by obtaining the k such that the value is the smallest, it is possible to acquire the quantized data. When the modal is insufficient, for example, when the _first modal M ₁ is missing, the dimension of the t-th vector W _t ⁽²⁾ in the accumulated feature data is D ₂ dimensional, and the representative vector Compared to V _k , the first D ₁ dimension is missing. In the third embodiment, the missing portion is dealt with by filling the representative value of the feature data of the corresponding learning signal data. The representative value is obtained for each dimension of the feature data of the learning signal, and a D _one- dimensional vector is obtained by connecting the representative values obtained in each dimension. Quantized data is acquired by calculating the distance between the vector connecting the D _one- dimensional vector and Wt ⁽²⁾ and the representative vector V _k and obtaining k that minimizes the value. In the above example, the case where the first modal is lost has been described. However, the case where the second modal is lost can also be handled by performing the same process.

  The target feature quantization unit 342 performs, for each target signal, based on the feature data of the target signal extracted by the target feature extraction unit 40 and the conversion table stored in the conversion table storage unit 26. The feature data for each minimum processing unit included in the signal is converted into quantized data, target quantized data is generated based on each of the converted quantized data, and is output to the search unit 44. Note that the method of converting the feature data of the target signal into the target quantized data is the same processing as the method of converting the feature data of the stored signal into the stored quantized data in the stored feature quantizing unit 332 described above. The detailed explanation is omitted.

  In addition, about the other structure of the multimodal signal search apparatus which concerns on 3rd Embodiment, since it is the same as that of the structure of the multimodal signal search apparatus which concerns on 1st Embodiment, description is abbreviate | omitted.

<Operation of Multimodal Signal Searching Device According to Third Embodiment of the Present Invention>
Next, the operation of the multimodal signal search apparatus 300 according to the third embodiment of the present invention will be described. In the multimodal signal search device 300, when each of the learning signals is acquired by the learning signal acquisition unit 10, the learning signal processing routine shown in FIG. 2 is executed by the multimodal signal search device. Further, when the accumulated signal acquisition unit 12 receives the accumulated signal, the multimodal signal search apparatus 300 executes an accumulated signal processing routine shown in FIG. Further, when the multimodal signal search device 300 receives the target signal by the target signal acquisition unit 14, the multimodal signal search device 300 executes a search processing routine shown in FIG. Note that the learning signal processing routine according to the third embodiment is the same as the learning signal processing routine according to the first embodiment, and thus description thereof is omitted.

  First, the accumulated signal processing routine shown in FIG. 9 will be described.

  In step S300 of the accumulated signal processing routine shown in FIG. 9, the characteristic data of the learning signal corresponding to the element of the portion corresponding to the missing portion of the feature data of the minimum unit to be processed acquired in step S124. Fill in the value.

  Note that other processes of the accumulation signal processing routine according to the third embodiment are the same as those of the accumulation signal processing routine according to the first embodiment, and thus the description thereof is omitted.

  Next, the search processing routine shown in FIG. 10 will be described.

  In step S320 of the search processing routine shown in FIG. 10, the characteristic value of the learning signal corresponding to the element corresponding to the missing portion of the feature data of the minimum unit to be processed acquired in step S156. Fill.

  Note that other processes of the search process routine according to the third embodiment are the same as those of the search process routine according to the first embodiment, and thus description thereof is omitted.

  As described above, according to the multimodal signal search device according to the third embodiment of the present invention, feature data is extracted for each input single-modal or multimodal target signal, and the target signal For each, if a part of the target signal is missing based on the extracted feature data of the target signal and the created conversion table, the representative value of the feature data of the learning signal corresponding to the part is displayed. Acquire target quantized data to fill, and for each target signal, from the created database based on the acquired target quantized data of the target signal, corresponding to the accumulated quantized data corresponding to the target quantized data By searching for attached attributes, it is possible to search for multimodal signals even if some modals are missing.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the first, second, and third embodiments, the case where the learning process, the accumulation process, and the search process are performed in one apparatus has been described, but the present invention is not limited to this. For example, the learning process and the storage process may be performed in the learning device, and the search process may be configured as a search device different from the learning device.

  In the first, second, and third embodiments, when a part of the accumulated signal or the target signal is missing, a process of filling 0, a process of ignoring, or a learning signal for each embodiment Although one of the processes for filling the representative value of the feature data has been described, the present invention is not limited to this. For example, if the stored signal or part of the target signal is missing, any appropriate process of filling the zero value, ignoring process, and filling the representative value of the feature data of the learning signal is arbitrarily selected for each process. You may choose.

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium or provided via a network. It is also possible to do.

<Experimental example>
Actually, an example in which the processing in the multimodal signal search apparatus according to the first, second, and third embodiments is applied to actual data will be described below. First, in the experiment, we collected data for hip-hop dance. There are two types of data collection. The first is data collection using a wearable modal as shown in FIG. 11, and the other is data collection using an external modal as shown in FIG. . In the example of FIG. 11, a combination of acquired data or the like and a part or the like is shown. As shown in FIG. 11 and FIG. 12, in this experiment, data is collected in various modalities, and the invention technique can be applied to all combinations. However, in the verification, image signal data, audio signal data, Acceleration sensor data was targeted. The image signal data was collected using a fixed camera installed in front of the performer, and the audio signal data was collected using a microphone attached to the camera, and the acceleration of the dance music flowed through the speakers was accelerated. Sensor data collected using wearable devices attached to the top, torso, both wrists, and both ankles of the performer.

  There are eight types of modals, one type of image signal data, one type of audio signal data, and six types of acceleration sensor data, and one or two of these signal data are combined. The data was regarded as single-modal data or multi-modal data, and was used as an input for stored signals, learning signals, and target signals. The dance is composed of four parts in total, and information on which part it belongs to is used as attribute data. In other words, in this experiment, accumulated quantized data obtained based on a single-modal or multi-modal signal and part attribute data are registered as a set in the database, and when the target signal is given, the database is Based on this, it is possible to obtain as output which attribute the target signal has, that is, which part it belongs to. Note that the data division method used in the experiment is described in order to create a conversion table by using data of one part among four parts as a learning signal. The remaining three parts were used as an accumulation signal and a target signal, and the degree of coincidence was evaluated. In other words, the evaluation target of the experiment is whether or not an appropriate part can be applied among the three parts, and is a problem that hits with a probability of 1/3 when randomly predicted. In the data collection, the performer performed the same dance twice, but the first performance was used as the accumulation signal and the second performance as the target signal.

In the experiment, comparative verification was performed by changing the combination of modals used for the target signal, accumulated signal, and learning signal. In this experiment, multimodal data is assumed to be a combination of two modals. Hereinafter, for convenience of description, the _first modal is modal M ₁ and the second modal is modal M ₂ . In this case, eight types of modals are used. The number of combinations of the two modal data is ₈ P ₂ = 56 in consideration of which modal acquisition time is used as a reference.

  The nine patterns compared are as follows.

(1) A pattern that uses only modal M ₁ data for all of the target signal, accumulated signal, and learning signal.
(2) target signal, the stored signal, either the learning signal using only data of the modal M ₂ pattern.
(3) A pattern using data in which the modal M ₁ and the modal M ₂ are combined for any of the target signal, the accumulation signal, and the learning signal.
(4) The learning signal uses data combining modal M ₁ and modal M ₂ , and the target signal and accumulated signal use only modal M ₁ . In addition, when quantization is performed in the target feature quantization part and the storage feature quantization unit, a modal shortage occurs, but the value of the shortage is ignored.
(5) A pattern in which the learning signal uses data combining modal M ₁ and modal M ₂ , and the target signal and the accumulation signal use only modal M ₂ . In addition, when quantization is performed in the target feature quantization part and the storage feature quantization unit, a modal shortage occurs, but the value of the shortage is ignored.
(6) The learning signal uses data combining modal M ₁ and modal M ₂ , and the target signal and accumulated signal use only modal M ₁ . In addition, when the target feature quantization part and the storage feature quantization unit perform quantization, a modal shortage occurs, but zero is filled in for the shortage. In this experiment, since centering is performed as a post-processing of feature data extraction, filling zero corresponds to filling an average value which is one of representative values of feature data.
(7) The learning signal uses data combining modal M ₁ and modal M ₂ , and the target signal and accumulated signal use only modal M ₂ . In addition, when the target feature quantization part and the storage feature quantization unit perform quantization, a modal shortage occurs, but zero is filled in for the shortage. In this experiment, since centering is performed as a post-processing of feature data extraction, filling zero corresponds to filling an average value which is one of representative values of feature data.
(8) The learning signal uses a combination of modal M ₁ and modal M ₂ , the target signal uses only modal M ₁ , and the accumulated signal uses only modal M ₂ . In this case, when quantization is performed by the target feature quantization part and the accumulation feature quantization unit, a modal shortage occurs, but the value of the shortage is ignored.
(9) The learning signal uses data obtained by combining modal M ₁ and modal M ₂ , the target signal uses only modal M ₂ , and the accumulation signal uses only modal M ₁ . In this case, when quantization is performed by the target feature quantization part and the accumulation feature quantization unit, a modal shortage occurs, but the value of the shortage is ignored.

  Among the above nine patterns, (8) and (9) are patterns in which the modal of the target signal and the accumulated signal are completely different, and search is performed in a cross modal manner. The results are summarized in FIG. This is a value obtained by averaging the accuracy of 56 modal combinations. When the average is taken, the patterns (1), (2), (4), (5), (6), (7), (8), and (9) are evaluated for the same combination. The K-means used when creating the code table is dependent on the initial value, and there is randomness, so they do not necessarily match.

  The problem that is the subject of this experiment is the problem of hitting the corresponding part of the three parts, and when the answer is selected at random, the accuracy is 1/3 = 0.33. When this is compared with the results of (1) to (9) in FIG. 13, both values are over 0.33, and the effectiveness of the inventive technique can be understood.

FIG. 14 shows the results when the wearable device at the top of the head is used as the modal M ₁ and the wearable device at the torso is used as the modal M ₂ . In FIG. 14, the upper part represents the results of patterns (1) to (3) from the left, the middle part represents the results of patterns (4) to (6) from the left, and the lower part represents the results from pattern (7) from the left. The result of (9) is represented. In each confusion matrix, the vertical axis represents the part to which the target signal belongs, and the horizontal axis represents the part to which the accumulated signal belongs. In all of these result examples, the accuracy is 100%, which shows the effectiveness of the inventive technique. 14 represents the distance, the smaller the value, the higher the matching degree.

DESCRIPTION OF SYMBOLS 10 Learning signal acquisition part 12 Accumulated signal acquisition part 14 Objective signal acquisition part 20 Operation part 22 Learning feature extraction part 24 Learning part 26 Conversion table memory | storage part 30 Accumulated feature extraction part 32 Accumulated feature quantization part 34 Attribute provision part 36 Database preparation part 38 Database storage unit 40 Target feature extraction unit 42 Target feature quantization unit 44 Search unit 90 Output unit 100 Multimodal signal search device 200 Multimodal signal search device 220 Operation unit 232 Accumulated feature quantization unit 242 Target feature quantization unit 300 Multi Modal signal search device 320 Operation unit 332 Accumulated feature quantization unit 342 Target feature quantization unit

Claims (8)

A learning feature extraction unit that extracts feature data for each of the input multimodal learning signals;
A learning unit that creates a conversion table from the feature data to a common code based on the feature data of each of the learning signals extracted by the learning feature extraction unit;
A storage feature extraction unit that extracts feature data for each of the input single-modal or multi-modal storage signals;
For each of the accumulated signals, the feature data of the accumulated signal is quantized using the code based on the feature data of the accumulated signal extracted by the accumulated feature extraction unit and the conversion table created by the learning unit. An accumulation feature quantization unit for obtaining accumulated quantization data converted into data;
For each of the accumulated signals, a database creating unit for creating the database by registering the accumulated quantized data of the accumulated signal acquired by the accumulated feature quantizing unit and the attribute of the accumulated signal in a database in association with each other,
Including a learning device. A target feature extraction unit that extracts feature data for each of the input single-modal or multi-modal target signals;
The feature data of the target signal is encoded with respect to each of the target signals based on feature data of the target signal extracted by the target feature extraction unit and a conversion table created in the learning device according to claim 1. A target feature quantization unit for acquiring target quantized data converted into quantized data using
For each of the target signals, the accumulated quantum corresponding to the target quantized data from the database created in the learning device based on the target quantized data of the target signal acquired by the target feature quantizing unit. A search unit for searching for the attribute associated with the digitized data;
A search device including:   The accumulated feature quantization unit, when the feature data of the accumulated signal lacks data corresponding to the modal included in the multimodal of the learning signal, the missing feature data of the accumulated signal. The accumulated quantized data is acquired based on the feature data in which zeros are embedded in the portion and the conversion table, or the conversion table is acquired based on the feature data of the stored signal and the conversion table. Ignoring data corresponding to the missing portion of the feature data stored in the memory, obtaining the accumulated quantized data, or in the missing portion of the feature data of the accumulated signal 2. The learning device according to claim 1, wherein the accumulated quantized data is acquired based on feature data in which a representative value of feature data of the corresponding learning signal is filled and the conversion table. .   The target feature quantization unit, when the feature data of the target signal lacks data corresponding to the modal included in the multimodal of the learning signal, the feature data of the target signal. The target quantized data is acquired based on the feature data in which zeros are embedded in the portion and the conversion table, or the conversion table is acquired based on the feature data of the target signal and the conversion table. Ignoring data corresponding to the missing portion of the feature data stored in the target data, obtaining the target quantized data, or in the missing portion of the feature data of the target signal 3. The search device according to claim 2, wherein the target quantized data is acquired based on feature data in which representative values of feature data of the corresponding learning signal are filled and the conversion table. . The learning signal includes two or more sensor data or media data,
The accumulated signal includes one or more sensor data or media data,
The search device according to claim 2 or 4, wherein the target signal includes one or more sensor data or media data. A learning method in a learning device including a learning feature extraction unit, a learning unit, an accumulation feature extraction unit, an accumulation feature quantization unit, and a database creation unit,
The learning feature extraction unit extracts feature data for each of the input multimodal learning signals,
The learning unit creates a conversion table from the feature data to a common code based on the feature data of each of the learning signals extracted by the learning feature extraction unit,
The accumulation feature extraction unit extracts feature data for each of the inputted single-modal or multi-modal accumulation signals,
The accumulated feature quantization unit, for each of the accumulated signals, based on the accumulated signal feature data extracted by the accumulated feature extraction unit and the conversion table created by the learning unit, To obtain accumulated quantized data converted into quantized data using the code,
The database creation unit creates, for each of the accumulated signals, the accumulated quantization data of the accumulated signal acquired by the accumulated feature quantization unit and the attribute of the accumulated signal in association with each other, and creates the database. How to learn. A search method in a search device, including a target feature extraction unit, a target feature quantization unit, and a search unit,
The target feature extraction unit extracts feature data for each of the input single-modal or multi-modal target signals,
The target feature quantization unit, for each of the target signals, based on feature data of the target signal extracted by the target feature extraction unit and a conversion table created in the learning method according to claim 6, Obtaining target quantized data obtained by converting characteristic data of the target signal into quantized data using the code,
For each of the target signals, the search unit converts the target quantized data from the database created in the learning device based on the target quantized data of the target signal acquired by the target feature quantizing unit. A search method for searching for the attribute associated with the corresponding accumulated quantized data.   A program for causing a computer to function as each part of the learning device according to claim 1 or 3, or the search device according to claim 2, claim 4, or claim 5.