JP7200597B2 - Information processing method and information processing system - Google Patents

Description

The present invention relates to technology for processing various types of information.

For example, in various facilities such as transportation facilities and commercial facilities, there has been conventionally proposed a technique for providing a terminal device with information representing the content of the guidance voice of the facility. For example, Patent Literature 1 discloses a configuration in which identification information according to an operation by an administrator of a transportation system is transmitted to a terminal device by acoustic communication. The terminal device displays guidance information corresponding to the identification information received by acoustic communication.

JP 2016-157412 A

It is envisioned that the above-described technology will be applied to situations in which information about the performance is provided to the audience in parallel with the performance of a play or Noh play. However, there is a problem that, for example, the operator of the demonstration is overloaded with the work of sequentially instructing the identification information manually in parallel with the progress of the demonstration. In the above explanation, it is assumed that information is provided to the audience in parallel with the progress of the demonstration. A problem is expected. In consideration of the above circumstances, it is an object of the present invention to reduce the work load of sequentially specifying identification information in parallel with the progress of a demonstration.

In order to solve the above problems, an information provision method according to a preferred aspect of the present invention compares an observed sound collected by a sound collecting device with a reference sound expected to be pronounced in the demonstration, estimating a performance position, which is a position on the time axis of the observed sound with respect to the reference sound, and referring to registered data representing temporal correspondence between the reference sound and the plurality of identification information, thereby obtaining the plurality of identification information; identification information temporally corresponding to the performance position.

An information processing system according to a preferred aspect of the present invention compares an observation sound collected from a performance by a sound collecting device with a reference sound expected to be pronounced in the performance, and determines the time of the observation sound with respect to the reference sound. A position estimating unit for estimating a performance position, which is a position on an axis, and registration data representing a temporal correspondence between the reference sound and a plurality of pieces of identification information, thereby determining the performance position among the plurality of pieces of identification information. and an information specifying unit that specifies identification information temporally corresponding to .

1 is a block diagram illustrating the configuration of an information providing system according to a first embodiment; FIG. 1 is a block diagram illustrating the configuration of an information processing system; FIG. 1 is a block diagram illustrating a functional configuration of an information processing system; FIG. 10 is a flowchart illustrating a specific procedure of processing for estimating a performance position by a position estimating unit; It is a block diagram which illustrates the structure of a terminal device. 4 is a flowchart illustrating specific procedures of operations of the information processing system and the terminal device; FIG. 7 is a block diagram illustrating the functional configuration of an information processing system according to a second embodiment; FIG. 10 is a flowchart illustrating a specific procedure of processing for estimating a performance position by a position estimating unit in the second embodiment; FIG. 11 is a block diagram illustrating the configuration of a management system according to a third embodiment; FIG. FIG. 12 is a block diagram illustrating a functional configuration of an information processing system according to a fourth embodiment; FIG. FIG. 16 is a flowchart illustrating a specific procedure of processing for generating registration data in the fourth embodiment; FIG. FIG. 14 is a flowchart illustrating a specific procedure of processing for generating registration data in the fifth embodiment; FIG.

<First Embodiment>
FIG. 1 is a block diagram illustrating the configuration of an information providing system 100 according to the first embodiment of the invention. The information providing system 100 is a computer system for providing information (hereinafter referred to as “related information”) C regarding a performance by a performer P to a user (that is, an audience) A viewing the performance. A performer P performs a performance such as drama or Kyogen at a facility such as a stage. Specifically, the performer P sequentially pronounces each of a plurality of lines written in a script prepared in advance. User A watches the demonstration while carrying the terminal device 30 . The terminal device 30 is, for example, a portable information terminal such as a mobile phone, a smart phone, a tablet terminal, or a personal computer. A guidance display terminal installed in the facility may be used as the terminal device 30 . Note that although a plurality of users A actually view the performance by the performer P, the following description focuses on the terminal device 30 of one user A for the sake of convenience.

As illustrated in FIG. 1 , the information providing system 100 includes an information processing system 10 and an information distribution system 20 . The information processing system 10 is a computer system that sequentially transmits the identification information D in parallel with the performance by the performer P. FIG. The identification information D is a code for identifying the related information C. FIG. The related information C of the first embodiment is a character string (that is, subtitles) representing the content of the lines pronounced by the performer P. In parallel with the performance by the performer P, the identification information D of the related information C corresponding to the lines pronounced by the performer P are sequentially transmitted from the information processing system 10 to the terminal device 30 . The identification information D can also be rephrased as information for identifying the lines pronounced by the performer P.

The information distribution system 20 can communicate with the terminal device 30 via a communication network 40 including a mobile communication network or the Internet. The information distribution system 20 transmits related information C corresponding to the identification information D received by the terminal device 30 from the information processing system 10 to the terminal device 30 . As can be understood from the above description, in parallel with the performance by the performer P, the terminal device 30 is sequentially provided with the related information C representing the content of the lines uttered by the performer P.

<Information processing system 10>
FIG. 2 is a block diagram illustrating the configuration of the information processing system 10. As shown in FIG. As illustrated in FIG. 2, the information processing system 10 of the first embodiment includes a sound collecting device 13, a control device 11, a storage device 12, and a sound emitting device . The information processing system 10 may be realized by a single device, or may be realized by a plurality of devices configured separately from each other.

The sound pickup device 13 is, for example, a microphone that picks up sound on the stage. The sound pickup device 13 of the first embodiment picks up a sound (hereinafter referred to as an "observation sound") pronounced by the performer P on the stage, and generates an acoustic signal X representing the waveform of the observation sound. That is, the observed sound is the sound of the performance collected by the sound collection device 13 . For the sake of convenience, an A/D converter that converts the acoustic signal X generated by the sound collecting device 13 from analog to digital is omitted. The sound emitting device 14 is a speaker that reproduces sound according to the acoustic signal Y generated by the control device 11 . A D/A converter for converting the acoustic signal Y from digital to analog is omitted from the drawing for the sake of convenience.

The control device 11 is composed of a processing circuit such as a CPU (Central Processing Unit), for example, and controls each element of the information processing system 10 in an integrated manner. The storage device 12 stores programs executed by the control device 11 and various data used by the control device 11 . For example, a known recording medium such as a semiconductor recording medium and a magnetic recording medium, or a combination of multiple types of recording media may be arbitrarily adopted as the storage device 12 . A portable recording medium that can be attached to and detached from the information processing system 10 or an external recording medium (for example, an online storage) with which the information processing system 10 can communicate may be used as the storage device 12 . That is, storage device 12 may be omitted from information processing system 10 .

FIG. 3 is a block diagram illustrating the functional configuration of the information processing system 10. As shown in FIG. As illustrated in FIG. 3, the storage device 12 of the first embodiment stores a time series of feature amounts Fref of sounds scheduled to be pronounced by the performer P (hereinafter referred to as "reference sounds"). A reference sound is, for example, a sound produced by pronouncing each line written in a script. The feature quantity Fref is a physical quantity representing acoustic features of the reference sound. For example, information such as MFCC (Mel-Frequency Cepstrum Coefficients) extracted from the reference sound is stored in the storage device 12 as the feature amount Fref.

As exemplified in FIG. 3, the feature quantity Fref is extracted in advance by analysis processing for the acoustic signal V representing the reference sound and stored in the storage device 12 . Specifically, the feature amount Fref is extracted for each of a plurality of analysis periods obtained by dividing the acoustic signal V on the time axis. The acoustic signal V of the reference sound is generated by speech synthesis of a character string to be pronounced in the demonstration by the performer P (for example, a registered character string W such as lines written in a script, which will be described later). A known technique is arbitrarily adopted for speech synthesis of the acoustic signal V. FIG. For example, unit-connected speech synthesis that generates an acoustic signal V by connecting multiple speech units, or statistical model-type speech synthesis that generates an acoustic signal V using a statistical model such as HMM (Hidden Markov Model). Speech synthesis is used to generate the acoustic signal V. FIG. According to the above configuration, there is an advantage that the reference sound (acoustic signal V) can be easily prepared from the character string that is expected to be pronounced in the demonstration.

The storage device 12 of the first embodiment stores registration data Q. FIG. The registered data Q is data representing the temporal correspondence between the reference sound and the plurality of pieces of identification information D. FIG. As exemplified in FIG. 3, the registration data Q specifies identification information D (D1, D2, . . . ) for each of a plurality of unit periods U (U1, U2, . . Each unit period U is, for example, an utterance period of each line in the reference sound. In the registered data Q, the time of the start point and the end point of each unit period U (for example, the time based on the beginning of the reference sound) is registered. A unit period U in which each line is pronounced is associated with identification information D of related information C corresponding to the line. The analysis period during which the aforementioned feature amount Fref is extracted is set to a sufficiently short time length compared to the unit period U.

As illustrated in FIG. 3, the control device 11 of the first embodiment performs a plurality of functions (position estimation unit 51, information identification unit 52 and signal processing unit 53) by executing a program stored in the storage device 12. Realize A part of the functions of the control device 11 may be realized by a dedicated electronic circuit. Also, the functions of the control device 11 may be installed in a plurality of devices.

The position estimating unit 51 compares the observation sound collected from the performance by the sound collection device 13 with the reference sound expected to be pronounced in the performance, and obtains the position on the time axis of the observation sound with respect to the reference sound (hereinafter referred to as “demonstration sound”). Estimate T (referred to as "position"). The performance position T is the time point on the time axis at which the performer P is currently performing, and is expressed, for example, by the elapsed time from the start point of the reference sound (that is, the start of the performance).

The position estimation unit 51 of the first embodiment compares the time series of the feature amount Fx extracted from the observed sound with the time series of the feature amount Fref of the reference sound stored in the storage device 12 to determine the performance position T. presume. The feature quantity Fx of the observed sound is the same physical quantity as the feature quantity Fref of the reference sound. In the first embodiment, MFCC is exemplified as the feature amount Fx.

FIG. 4 is a flowchart illustrating a specific procedure of processing for specifying the performance position T by the position estimation unit 51. As shown in FIG. The processing of FIG. 4 is executed at a predetermined cycle. 4, the position estimating unit 51 analyzes the sound signal X supplied from the sound collecting device 13 to obtain a time series of the feature amount Fx for a predetermined period including the current time in the sound signal X. Extract (Sa1). A known analysis technique such as short-time Fourier transform is arbitrarily used to extract the feature quantity Fx.

The position estimator 51 estimates the performance position T by comparing the time series of the feature amount Fx extracted from the acoustic signal X with the time series of the feature amount Fref of the reference sound stored in the storage device 12 ( Sa2). A known technique is arbitrarily adopted for estimating the performance position T. FIG. For example, the position estimating unit 51 performs arithmetic processing for calculating the similarity (e.g., distance or correlation) between the feature quantity Fx and the feature quantity Fref, and dynamic processing for analyzing the temporal correspondence between the observed sound and the reference sound. The performance position T is estimated by searching for a period corresponding to the time series of the feature amount Fx of the observed sound among the reference sounds by time warping (DTW). A performance position T is estimated for each predetermined period of the observed sound.

The information specifying unit 52 of FIG. 3 specifies the identification information D temporally corresponding to the performance position T among the plurality of identification information D by referring to the registration data Q stored in the storage device 12 . Specifically, among the plurality of unit periods U registered in the registration data Q, the unit period U including the performance position T (that is, the unit period U in which the performance position T is located between the time of the start point and the time of the end point ), and acquires the identification information D corresponding to the unit period U from the registered data Q. Identification of the identification information D by the information identification unit 52 is repeated each time the position estimation unit 51 estimates the performance position T. FIG. As can be understood from the above description, the information identifying unit 52 sequentially identifies the identification information D corresponding to the lines uttered by the performer P.

The signal processing unit 53 generates the acoustic signal Y including the identification information D specified by the information specifying unit 52 . As illustrated in FIG. 3 , the signal processing section 53 of the first embodiment includes a modulation processing section 531 and a synthesis processing section 532 .

The modulation processing unit 531 generates a modulated signal M containing the identification information D specified by the information specifying unit 52 as an acoustic component by predetermined modulation processing. Modulated signal M is generated by frequency-modulating a carrier wave of a predetermined frequency with identification information D, for example. Modulated signal M may be generated by sequentially performing spread modulation of identification information D using a spread code and frequency conversion using a carrier wave of a predetermined frequency. The frequency band of the sound represented by the modulated signal M is the frequency band in which the sound can be emitted by the sound emitting device 14 and collected by the terminal device 30, and the frequency band of the sound that the user A listens to in a normal environment. (for example, 18 kHz or more and 20 kHz or less). However, the frequency band of the sound represented by the modulated signal M is arbitrary.

The synthesizing unit 532 generates an acoustic signal Y by synthesizing the acoustic signal X generated by the sound collecting device 13 and the modulated signal M generated by the modulation processing unit 531 . The synthesizing unit 532 generates the acoustic signal Y by, for example, adding the acoustic signal X and the modulated signal M (for example, weighted addition). Note that the modulated signal M may be supplied to the sound emitting device 14 . That is, the composition processing section 532 can be omitted.

By supplying the sound signal Y generated by the signal processing unit 53 (synthesis processing unit 532) according to the above configuration and procedure to the sound emitting device 14, the sound represented by the sound signal Y is reproduced for the user A. . Specifically, the observed sound represented by the acoustic signal X and the acoustic component of the identification information D represented by the modulated signal M are reproduced from the sound emitting device 14 . As can be understood from the above description, the sound emitting device 14 functions as an acoustic device that emits the observation sound of the performance by the performer P to the audience seat where the user A is located, and also functions as a sound wave as an air vibration. It also functions as a transmitter that transmits the identification information D by acoustic communication using the transmission medium. That is, as described above, in parallel with the performance by the performer P, the information processing system 10 sequentially transmits the identification information D of the related information C corresponding to the lines pronounced by the performer P to the terminal device 30. do. In conjunction with the progress of the demonstration by the performer P, the identification information D transmitted to the terminal device 30 is updated sequentially.

<Terminal device 30>
FIG. 5 is a block diagram illustrating the configuration of the terminal device 30. As shown in FIG. As illustrated in FIG. 5, the terminal device 30 includes a control device 31, a storage device 32, a communication device 33, a sound pickup device 34, and a playback device 35. FIG.

The sound pickup device 34 is a microphone that picks up ambient sounds. Specifically, the sound pickup device 34 picks up the sound reproduced by the sound emitting device 14 of the information processing system 10 and generates the sound signal Z representing the waveform of the sound. The acoustic signal Z includes an acoustic component representing the identification information D. FIG. Therefore, the sound pickup device 34 is used for voice communication or sound recording during video shooting, and also functions as a receiver that receives the identification information D through acoustic communication using sound waves as air vibrations as a transmission medium. For the sake of convenience, the illustration of an A/D converter that converts the acoustic signal Z generated by the sound collecting device 34 from analog to digital is omitted. Further, instead of the sound collecting device 34 configured integrally with the terminal device 30, a separate sound collecting device 34 may be connected to the terminal device 30 by wire or wirelessly.

The storage device 32 stores programs executed by the control device 31 and various data used by the control device 31 . For example, a known recording medium such as a semiconductor recording medium and a magnetic recording medium, or a combination of multiple types of recording media can be arbitrarily adopted as the storage device 32 . The communication device 33 communicates with the information distribution system 20 under the control of the control device 31 .

The reproduction device 35 is an output device that reproduces the related information C. FIG. The playback device 35 of the first embodiment includes a display device (for example, a liquid crystal display panel) that displays related information C. FIG. That is, the character string of the lines spoken by the performer P is displayed as the related information C on the playback device 35 . Therefore, a hearing-impaired person who has difficulty hearing the voice pronounced by the performer P can grasp the contents of the demonstration by the performer P. Note that a sound emitting device that emits a sound representing the content of the related information C may be used as the reproducing device 35 .

The control device 31 is composed of a processing circuit such as a CPU, for example, and controls each element of the terminal device 30 in an integrated manner. As illustrated in FIG. 5, the control device 31 of the first embodiment implements a plurality of functions (an information extraction unit 311 and an operation control unit 312) by executing programs stored in the storage device 32. FIG. A part of the functions of the control device 31 may be realized by a dedicated electronic circuit. Also, the functions of the control device 31 may be installed in a plurality of devices.

The information extraction unit 311 extracts the identification information D from the acoustic signal Z generated by the sound collection device 34 . Specifically, the information extraction unit 311 performs, for example, filter processing for emphasizing the frequency band containing the acoustic component of the identification information D in the acoustic signal Z, and demodulation processing corresponding to the modulation processing performed by the modulation processing unit 531. The identification information D is extracted from the acoustic signal Z.

The operation control unit 312 acquires the relevant information C from the information distribution system 20 and causes the reproduction device 35 to reproduce the relevant information C. FIG. Specifically, the operation control unit 312 generates an information request R including the identification information D extracted by the information extraction unit 311 and transmits the information request R from the communication device 33 to the information distribution system 20 . The information request R is transmitted each time the identification information D extracted from the acoustic signal Z changes (that is, each line uttered by the performer P). Further, the operation control unit 312 receives the related information C transmitted from the information distribution system 20 in response to the information request R by the communication device 33 and causes the playback device 35 to reproduce the related information C. FIG. As understood from the above description, the terminal device 30 reproduces the related information C corresponding to the identification information D received from the information processing system 10 .

<Information distribution system 20>
The information distribution system 20 of FIG. 1 is a server device (for example, a web server) that stores a plurality of pieces of related information C. As shown in FIG. Each of a plurality of pieces of related information C (C1, C2, . . . ) is associated with identification information D (D1, D2, . . . ) of the related information C and stored. The information distribution system 20 transmits related information C corresponding to the identification information D included in the information request R from the terminal device 30 to the terminal device 30 .

<Action>
FIG. 6 is a flow chart illustrating the procedure of operations of the information processing system 10 and the terminal device 30 . In parallel with the demonstration by the performer P, the process of FIG. 6 is repeated at predetermined intervals, for example.

The position estimating unit 51 of the information processing system 10 estimates the performance position T by comparing the observed sound of the performance collected by the sound collecting device 13 with the reference sound prepared in advance (Sb1). Specifically, as described above with reference to FIG. 4, the position estimation unit 51 estimates the performance position T by matching the feature amount Fx of the observed sound and the feature amount Fref of the reference sound.

The information specifying unit 52 refers to the registration data Q representing the temporal correspondence between the reference sound and the plurality of pieces of identification information D to obtain the identification information D temporally corresponding to the performance position T estimated by the position estimating unit 51. is specified (Sb2). The signal processing unit 53 generates the acoustic signal Y including the identification information D specified by the information specifying unit 52 (Sb3). The signal processing unit 53 transmits the identification information D by acoustic communication by supplying the acoustic signal Y to the sound emitting device 14 (Sb4).

The information extraction unit 311 of the terminal device 30 acquires the acoustic signal Z generated by the sound pickup device 34 by picking up the reproduced sound by the sound emitting device 14 (Sb5). The information extractor 311 extracts the identification information D from the acoustic signal Z (Sb6). The operation control unit 312 transmits the information request R including the identification information D extracted by the information extraction unit 311 from the communication device 33 to the information distribution system 20 (Sb7). The information delivery system 20 transmits the related information C corresponding to the identification information D included in the information request R received from the terminal device 30 to the terminal device 30 that made the request.

The operation control unit 312 of the terminal device 30 receives the related information C transmitted from the information distribution system 20 by the communication device 33 (Sb8), and causes the playback device 35 to play back the related information C (Sb9). That is, at appropriate points in time parallel to the performance by the performer P, subtitles of each line pronounced in the performance are sequentially displayed.

As described above, in the first embodiment, the performance position T of the observation sound with respect to the reference sound is estimated by matching the observation sound and the reference sound. , the identification information D corresponding to the performance position T can be specified. Therefore, it is possible to specify the identification information D at an appropriate point in time during the demonstration without having to sequentially indicate the identification information D at an appropriate point in time during the demonstration. According to the first embodiment, there is also the advantage that the performance position T can be appropriately estimated by matching the feature amount Fx of the observed sound and the feature amount Fref of the reference sound.

<Second embodiment>
A second embodiment of the present invention will be described. It should be noted that, in each of the following illustrations, the reference numerals used in the description of the first embodiment are used for the elements whose functions are the same as those of the first embodiment, and detailed description of each will be omitted as appropriate.

FIG. 7 is a block diagram illustrating the functional configuration of the information processing system 10 according to the second embodiment. The storage device 12 of the second embodiment stores registration data Q similar to that of the first embodiment. In addition, the storage device 12 stores a character string (hereinafter referred to as “reference character string”) Sref representing the content of the reference sound, instead of the feature amount Fref of the reference sound exemplified in the first embodiment. A reference character string Sref is estimated in advance by speech recognition for an acoustic signal V representing a reference sound and stored in the storage device 12 . As in the first embodiment, the acoustic signal V is generated by voice synthesis of a character string to be pronounced in the demonstration by the performer P (that is, the lines written in the script).

A known technique is arbitrarily adopted for speech recognition of the acoustic signal V. FIG. The reference character string Sref of the second embodiment is estimated by speech recognition using a recognition model. The recognition model includes, for example, an acoustic model such as an HMM, and a language model indicating linguistic constraints.

As in the first embodiment, the position estimation unit 51 estimates the performance position T by matching the observed sound picked up by the sound pickup device 13 with reference sounds prepared in advance. A position estimating unit 51 of the second embodiment compares a character string Sx estimated by speech recognition for an observed sound (hereinafter referred to as an "observed character string") with a reference character string Sref stored in the storage device 12. , the performance position T by the performer P is estimated.

FIG. 8 is a flowchart illustrating a specific procedure of processing for specifying the performance position T by the position estimation unit 51 of the second embodiment. The processing of FIG. 8 is repeated at a predetermined cycle. 8, the position estimating unit 51 estimates an observed character string Sx by speech recognition of the acoustic signal X supplied from the sound collecting device 13 (Sc1). Speech recognition for the acoustic signal X is performed in the same way as speech recognition for the acoustic signal V of the reference sound. Specifically, a common recognition model (acoustic model and language model) is used in speech recognition for the acoustic signal X of the observed sound and speech recognition for the acoustic signal V of the reference sound.

The position estimation unit 51 estimates the performance position T by comparing the observed character string Sx estimated from the acoustic signal X and the reference character string Sref stored in the storage device 12 (Sc2). A known technique is arbitrarily adopted for estimating the performance position T. FIG. Specifically, the position estimation unit 51 calculates the similarity between each of a plurality of sections obtained by dividing the reference character string Sref on the time axis and the observed character string Sx, and calculates the similarity with the observed character string Sx. The performance position T is estimated according to the section in which is the maximum. Edit distance, for example, is preferably used as an index of similarity between the reference character string Sref and the observed character string Sx. The performance position T is estimated for each predetermined period of the observed sound by the processing illustrated above.

Identification of the identification information D according to the demonstration position T and transmission of the identification information D are the same as in the first embodiment. Also, the operations of the terminal device 30 and the information distribution system 20 are the same as in the first embodiment. Therefore, the same effects as in the first embodiment are realized in the second embodiment as well.

By the way, in a performance of traditional entertainment such as Kyogen or Kabuki, for example, the performer P speaks special words such as archaic language. Also, in a performance performed in a specific area, the performer P speaks special words including phrases peculiar to the area. When the observed sound is a speech uttered with special words such as the above examples, accurate recognition is difficult with speech recognition using a recognition model for ordinary words such as modern words. Therefore, in the configuration in which the observed character string Sx estimated by speech recognition of the observed sound and the dialogue written in the script are collated, the two character strings diverge, and as a result, the performance position T can be estimated with high accuracy. Can not. In contrast to the above configuration, in the second embodiment, an observed character string Sx estimated by speech recognition using a common recognition model (for example, a recognition model for ordinary words such as modern language) and a reference character string Sref is matched. Therefore, regardless of whether the result of speech recognition is correct or not, the tendency of speech recognition itself is the same between the observed character string Sx and the reference character string Sref. Therefore, according to the second embodiment, there is an advantage that the performance position T can be estimated with high accuracy even when the observed sound is a speech in which a special word such as an ancient word is pronounced. If the accuracy of speech recognition does not pose a particular problem, the observed character string Sx estimated by speech recognition of the observed sound may be compared with the dialogue written in the script.

Note that, in the first embodiment, the sound signal V of the reference sound is generated by speech synthesis for a character string (for example, a line written in a script) to be pronounced by the performer P, and extracted from the sound signal V of the reference sound. The obtained feature amount Fref and the observed sound feature amount Fx are collated. The above configuration also achieves the same effect as the second embodiment, that is, the performance position T can be estimated with high accuracy even when the observed sound is a voice pronouncing a special word such as an ancient word.

The matching between the feature quantity Fx and the feature quantity Fref in the first embodiment and the matching between the observed character string Sx and the reference character string Sref in the second embodiment are specific examples of processing for comparing the observed sound and the reference sound. corresponds to Note that the collation between the observed sound and the reference sound is not limited to the above examples. For example, the performance position T may be estimated by mutually collating the acoustic signal X of the observed sound and the acoustic signal V of the reference sound.

<Third Embodiment>
FIG. 9 is a block diagram illustrating the configuration of a management system 200 according to the third embodiment. A management system 200 according to the third embodiment is a computer system for controlling the presentation of a performance by a performer P, and includes an information processing system 10 and a stage system 60 . The stage system 60 performs various productions in parallel with the performance by the performer P.

The information processing system 10 has the same configuration as that of the first embodiment, and in parallel with the performance by the performer P, sequentially transmits the identification information D by acoustic communication. Specifically, the information processing system 10 estimates the performance position T by matching the observed sound and the reference sound, and specifies and transmits the identification information D corresponding to the performance position T. FIG. Note that the configuration of the second embodiment may be applied to the information processing system 10 of the third embodiment.

While the identification information D in the first embodiment is a code for identifying the related information C, the identification information D in the third embodiment is a code for identifying the content of the performance by the stage system 60. FIG. In other words, the information processing system 10 of the third embodiment sequentially instructs the stage system 60 about the contents of the presentation in parallel with the performance by the performer P.

As illustrated in FIG. 9 , the stage system 60 includes a management device 61 and a production device 62 . The production device 62 is a device for performing production on the stage, and includes a sound device 621 and a lighting device 622 . The sound device 621 is equipment for reproducing various sounds such as music or sound effects. The lighting device 622 is equipment for lighting the stage. The lighting characteristics (lighting/lighting out, light intensity, emission color) of the lighting device 622 are variably controlled.

The management device 61 controls the effect device 62 according to the identification information D received from the information processing system 10 through acoustic communication. For example, a mobile phone, a smart phone, a tablet terminal, or a portable information terminal such as a personal computer is used as the management device 61 . The management device 61 stores a performance action for each of the plurality of pieces of identification information D. FIG. The performance operation is, for example, reproduction of various sounds by the sound device 621 or control of lighting by the lighting device 622 . The management device 61 extracts the identification information D from the sound reproduced by the information processing system 10 and instructs the production device 62 to perform the production operation corresponding to the identification information D. FIG. Therefore, in parallel with the performance by the performer P, the performance on the stage by the stage system 60 is sequentially controlled. As can be understood from the above description, the stage system 60 executes a performance action corresponding to the identification information D received from the information processing system 10. FIG.

In the third embodiment, similarly to the first embodiment, since the performance position T of the observation sound with respect to the reference sound is estimated by matching the observation sound and the reference sound, the temporal correspondence with the reference sound is registered. The identification information D corresponding to the performance position T can be specified among the plurality of identification information D. FIG. Therefore, it is possible to specify the identification information D at an appropriate point in time during the demonstration without having to sequentially indicate the identification information D at an appropriate point in time during the demonstration. Further, according to the third embodiment, it is possible to perform the production related to the performance by the performer P at an appropriate time in parallel with the performance.

<Fourth Embodiment>
The information processing system 10 of the fourth embodiment has a function of generating registration data Q. FIG. FIG. 10 is a block diagram illustrating a functional configuration for generating registration data Q by the control device 11 in the fourth embodiment. As illustrated in FIG. 10, the storage device 12 of the fourth embodiment stores registered character strings W. FIG. The registered character string W is a character string that the performer P is expected to pronounce. For example, the time series of each line written in the script of the demonstration by the performer P is stored as the registered character string W in the storage device 12 in advance. The registered character string W is divided into a plurality of sections (hereinafter referred to as “registered sections”) σ corresponding to different lines. Any one registration section σ is composed of a series of character strings (for example, a single or a plurality of sentences) that serve as an utterance unit, for example. As described above, the registered character string W includes a plurality of registered sections σ.

The performer P performs, for example, at the stage of preparation (for example, rehearsal) for a formal performance. At the stage of preparation, the performer P pronounces the registered character string W. The sound pickup device 13 of the fourth embodiment picks up the reference sound uttered by the performer P and generates an acoustic signal V representing the waveform of the reference sound.

As illustrated in FIG. 10 , the control device 11 of the fourth embodiment functions as an analysis processing section 55 by executing programs stored in the storage device 12 . The analysis processing unit 55 generates registered data Q by matching the registered character string W stored in the storage device 12 with the reference sound picked up by the sound pickup device 13 . The registration data Q specifies identification information D (D1, D2, . . . ) for each of a plurality of unit periods U (U1, U2, . Each unit period U is a period during which each registered section σ of the registered character string W is pronounced among the reference sounds. That is, the analysis processing unit 55 compares the registered character string W with the reference sound, and divides the reference sound into a plurality of unit periods U corresponding to different registered intervals σ on the time axis.

FIG. 11 is a flowchart illustrating a specific procedure of the operation of generating the registration data Q by the analysis processing unit 55 in the fourth embodiment. Triggered by an instruction to generate registration data Q, the process of FIG. 11 is started. 11, the analysis processing unit 55 analyzes the acoustic signal V supplied from the sound collection device 13 to extract the time series of the feature quantity Fref of the reference sound (Sd1).

The analysis processing unit 55 generates an acoustic signal (hereinafter referred to as a “synthesized signal”) representing the waveform of the synthesized voice uttering the registered character string W by speech synthesis for the registered character string W stored in the storage device 12 (Sd2 ). For generating the synthesized signal, for example, the above-described segment connection type or statistical model type speech synthesis is preferably used. The analysis processing unit 55 extracts the time series of the feature quantity Fsyn of the synthesized speech by analyzing the synthesized signal generated by speech synthesis (Sd3). For example, a time series of the feature amount Fsyn is extracted for each registered section σ of the registered character string W. FIG. The feature quantity Fsyn of the synthesized sound is a physical quantity (for example, MFCC) of the same kind as the feature quantity Fref of the reference sound. Note that the order of the extraction of the feature amount Fref of the reference sound (Sd1) and the extraction of the feature amount Fsyn of the synthesized sound (Sd2, Sd3) may be reversed. As can be understood from the above description, the control device 11 of the fourth embodiment functions as an element (feature extractor) that extracts the feature quantity Fsyn of the synthetic voice generated by speech synthesis for the registered character string W. FIG. Note that a representative value (for example, an average value) of feature amounts of a plurality of reference sounds recorded at different times or locations may be used as the feature amount Fsyn to generate registration data Q. FIG. Also, a representative value (for example, an average value) of a plurality of feature amounts extracted from the synthesized speech for the registered section σ may be used as the feature amount Fsyn.

The analysis processing unit 55 divides the reference sound into a plurality of unit periods U by comparing the time series of the feature amount Fref of the reference sound and the time series of the feature amount Fsyn of the synthesized sound (Sd4). For example, for each of a plurality of registered sections σ of the registered character string W, the period of the reference sound from which the feature amount Fref similar to the time series of the feature amount Fsyn in the registered section σ is extracted corresponds to the registered section σ. It is defined as a unit period U. That is, the reference sound is segmented on the time axis into unit periods U for each registration section σ (for example, each line). Specifically, the analysis processing unit 55 identifies the time of the start point and the end point of each unit period U. FIG. As can be understood from the above description, the control device 11 of the fourth embodiment compares the feature quantity Fref of the reference sound with the feature quantity Fsyn of the synthesized sound, thereby matching each registered section σ of the reference sound. It functions as an element (period definition section) that defines the unit period U.

After dividing the reference sound into a plurality of unit periods U in the above procedure, the analysis processing unit 55 generates registered data Q in which the identification information D is associated with each of the plurality of unit periods U (Sd5). Specifically, the analysis processing unit 55 assigns identification information D that does not overlap each other to each of a plurality of unit periods U of the reference sound, and associates the times of the start and end points of the unit periods U with the identification information D. registration data Q is generated.

The analysis processing unit 55 stores the registration data Q generated by the above procedure and the time series of the feature amount Fref extracted from the reference sound (Sd1) in the storage device 12 (Sd6). As exemplified in the first embodiment, the time series of the feature amount Fref is used for estimating the performance position T by the position estimating unit 51, and the registration data Q is used for specifying the identification information D by the information specifying unit 52. .

According to the fourth embodiment, the registration data Q used for sequentially transmitting each of the plurality of identification information D to the terminal device 30 in parallel with the demonstration by the performer P can be easily obtained from the registration character string W. It has the advantage of being able to generate The fourth embodiment is similarly applied to the generation of the registration data Q used in the second or third embodiment as well as the registration data Q used in the first embodiment. In the example of FIG. 11, the feature amount Fref of the reference sound is stored in the storage device 12 along with the registration data Q (Sd6). It is stored in the storage device 12 as a reference character string Sref.

<Fifth Embodiment>
5th Embodiment is a form for producing|generating the registration data Q similarly to 4th Embodiment. The control device 11 of the information processing system 10 according to the fifth embodiment executes the program stored in the storage device 12 to match the registered character string W with the reference sound to obtain the registered data, as in the example of FIG. It functions as an analysis processing unit 55 that generates Q.

FIG. 12 is a flowchart illustrating a specific procedure of the operation of generating the registration data Q by the analysis processing unit 55 of the fifth embodiment. Triggered by an instruction to generate registration data Q, the process of FIG. 12 is started. When the processing of FIG. 12 is started, the analysis processing unit 55 estimates the reference character string Sref by speech recognition for the acoustic signal V of the reference sound supplied from the sound collection device 13 (Se1). The reference string Sref is estimated by known speech recognition using a recognition model including an acoustic model and a language model.

The analysis processing unit 55 generates a synthesized signal representing the waveform of the synthesized voice uttering the registered character string W by speech synthesis for the registered character string W stored in the storage device 12 (Se2). For generating the synthesized signal, for example, the above-described segment connection type or statistical model type speech synthesis is preferably used.

The analysis processing unit 55 generates a character string (hereinafter referred to as “synthesized character string”) Wsyn representing the utterance content of the synthesized sound through speech recognition of the synthesized signal after speech synthesis (Se3). A synthesized character string Wsyn is estimated for each registered section σ of the registered character string W. FIG. A common recognition model is used for the speech recognition of the synthesized sound (Se3) and the speech recognition of the reference sound (Se1). As can be understood from the above description, the control device 11 of the fifth embodiment uses the voice recognition unit (speech recognition unit) for estimating the synthesized character string Wsyn by recognizing the synthetic sound generated by speech synthesis of the registered character string W. Function. Since the synthesized character string Wsyn is the result of speech recognition of the synthesized sound generated from the registered character string W, it ideally matches or resembles the registered character string W. However, the synthetic character string Wsyn and the registered character string W do not completely match due to factors such as misrecognition in speech recognition. The difference between the synthesized character string Wsyn and the registered character string W is particularly noticeable when the performer P speaks a special word such as an archaic language. The order of generating the reference character string Sref (Se1) and generating the composite character string Wsyn (Se2, Se3) may be reversed.

The analysis processing unit 55 divides the reference sound into a plurality of unit periods U by mutually collating the reference character string Sref and the synthesized character string Wsyn (Se4). For example, for each of a plurality of registered sections σ of the registered character string W, the period of the reference sound in which the reference character string Sref similar to the synthesized character string Wsyn in the registered section σ is estimated is the unit corresponding to the registered section σ. Defined as period U. That is, as in the fourth embodiment, the reference sound is divided on the time axis into unit periods U for each registration section σ (for example, each line). As can be understood from the above description, the control device 11 according to the fifth embodiment compares the reference character string Sref and the synthesized character string Wsyn to determine the unit period U corresponding to each registered section σ of the reference sound. It functions as a defining element (period defining part).

When the reference sound is divided into a plurality of unit periods U by the above procedure, the analysis processing unit 55 generates registered data Q corresponding to each of the plurality of unit periods U with the identification information D by the same procedure as in the fourth embodiment. (Se5). Then, the analysis processing unit 55 stores the registration data Q generated by the above procedure and the time series of the feature amount Fref extracted from the reference sound by separate processing in the storage device 12 (Se6).

When the reference sound is the uttered speech of a special word such as an ancient language, accurate recognition is difficult with speech recognition using a recognition model of a normal word such as a modern language. The performer P pronounces the registered character string W, but due to the above circumstances, the reference character string Sref estimated by speech recognition of the reference sound and the registered character string W expected to be pronounced in the demonstration may diverge. have a nature. Therefore, in the configuration in which each unit period U is divided by matching the reference character string Sref and the registered character string W, it is difficult to divide the reference sound into unit periods U for each registered section σ with high accuracy. In contrast to the above configuration, in the fifth embodiment, the reference character string Sref and the synthesized character string Wsyn are estimated by speech recognition using a common recognition model (for example, a recognition model for ordinary words such as modern languages). is matched. Therefore, even when the reference sound is an utterance of a special word such as an archaic language, it is possible to segment the reference sound into unit periods U for each registration period σ with high accuracy.

The fifth embodiment is similarly applied to the generation of the registration data Q used in the second or third embodiment as well as the registration data Q used in the first embodiment. In the example of FIG. 12, the feature amount Fref of the reference sound is stored in the storage device 12 together with the registered data Q (Se6), but in the second embodiment, the reference character string Sref estimated from the reference sound (Se1) is stored in the storage device. 12.

In the fourth and fifth embodiments, it is assumed that the information processing system 10 that transmits the identification information D to the terminal device 30 generates the registration data Q, but the information processing system 10 that transmits the identification information D is Registration data Q may be generated by a separate information processing system.

<Modification>
Specific modified aspects added to the above-exemplified aspects will be exemplified below. A plurality of aspects arbitrarily selected from the following examples may be combined as appropriate within a mutually consistent range.

(1) In each of the above embodiments, the acoustic signal V of the reference sound is generated by speech synthesis, but the method of generating the acoustic signal V is not limited to the above examples. For example, the acoustic signal V of the reference sound may be generated from a plurality of recorded sounds recorded in past demonstrations. For example, the sound signal V of the reference sound is generated by synthesizing a plurality of sound signals representing sounds recorded in different demonstrations of a common program. The time series of the feature amount Fref (reference character string Sref in the second embodiment) extracted from the acoustic signal V of the reference sound generated by the above procedure, and the registered data Q generated from the acoustic signal V, It is stored in the storage device 12 of the information processing system 10 .

Alternatively, a representative value (for example, an average) of feature amounts extracted from a plurality of recorded sounds may be calculated as the feature amount Fref of the reference sound. According to the configuration for generating the reference sound (for example, the acoustic signal V or the feature amount Fref) from a plurality of recorded sounds recorded in the past demonstrations as in the above examples, the reference sounds reflecting the tendency of the recorded sounds in the past demonstration It is possible to generate sounds. The "past performance" includes not only formal performances on stage but also preparatory performances for formal performances (for example, performances at rehearsals).

(2) In the first embodiment, the terminal device 30 reproduces the related information C received from the information distribution system 20, but the related information C stored in the storage device 32 of the terminal device 30 may be reproduced. For example, the storage device 32 of the terminal device 30 associates each of the plurality of related information C with the identification information D and stores them. The operation control unit 312 acquires the related information C corresponding to the identification information D extracted by the information extraction unit 311 from the storage device 32 and causes the reproduction device 35 to reproduce it. According to the above configuration, there is an advantage that the terminal device 30 can acquire the related information C without requiring communication via the communication network 40 . On the other hand, according to the configuration in which the terminal device 30 receives the related information C from the information distribution system 20 as in the first embodiment, there is an advantage that it is not necessary to hold a plurality of related information C in the storage device 32 of the terminal device 30. There is In each of the above-described embodiments, the information processing system 10 transmits the related information C to the terminal device 30 each time the information request R transmitted from the terminal device 30 is received. , the related information C may be transmitted (push distribution) from the information processing system 10 without requiring the information request R to be received.

(3) In the first and second embodiments, the character string of the lines pronounced by the performer P is provided to the terminal device 30 as the related information C, but the contents of the related information C are not limited to the above examples. For example, the terminal device 30 may be provided with a character string obtained by translating a line pronounced by the performer P into a specific language as the related information C. FIG. According to the above configuration, for example, even a foreigner who cannot understand the language of the lines pronounced by the performer P can understand the contents of the lines by visually recognizing the related information C reproduced by the terminal device 30 . Note that the terminal device 30 may be provided with a character string as the related information C, which is obtained by converting an archaic language or a unique phrase pronounced by the performer P into a standard expression (for example, modern language or standard language). Also, the commentary on the performance by the performer P or the information (for example, URL) representing the location of the site explaining the performance may be provided to the terminal device 30 as the related information C. FIG.

(4) The result of matching between the observed sound and the reference sound by the position estimation unit 51 (hereinafter referred to as “matching result”) may be displayed on the display device provided in the information processing system 10 . The matching result is, for example, a change in the performance position T over time. The collation result is displayed in real time in parallel with the collation of the observed sound and the reference sound, for example. The administrator of the information processing system 10 determines whether the matching result is appropriate by checking the display on the display device. Manually modify the results (eg, the demonstration position T).

(5) In each of the above embodiments, MFCC was exemplified as the feature quantity Fx and the feature quantity Fref, but the types of the feature quantity Fx and the feature quantity Fref are not limited to the above examples. For example, frequency characteristics such as a frequency spectrum and a spectrogram may be used to estimate the performance position T as the feature amount Fx and the feature amount Fref.

(6) In each of the above embodiments, the identification information D was transmitted to the terminal device 30 by acoustic communication, but the communication method for transmitting the identification information D is not limited to acoustic communication. For example, the identification information D may be transmitted to the terminal device 30 by wireless communication using electromagnetic waves such as radio waves or infrared rays as a transmission medium. Communication using electromagnetic waves as a transmission medium and acoustic communication exemplified in each of the above-described forms are comprehensively expressed as short-range wireless communication without intervening communication network 40 such as a mobile communication network.

(7) The functions of the information processing system 10 according to each of the above-described embodiments are realized by cooperation between processing circuits such as the control device 11 and programs, as illustrated in each embodiment. The program according to each of the forms described above can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disc) such as a CD-ROM is a good example. Also included are recording media in the form of It should be noted that the non-transitory recording medium includes any recording medium other than transitory, propagating signals, and does not exclude volatile recording media. Alternatively, the program may be provided to the computer in the form of distribution via a communication network.

<Appendix>
For example, the following configuration can be grasped from the form illustrated above.

An information processing method according to a preferred aspect (first aspect) of the present invention compares an observed sound picked up by a sound pickup device with a reference sound expected to be pronounced in the By estimating the performance position, which is the position of the observed sound on the time axis, and referring to registered data representing the temporal correspondence between the reference sound and the plurality of pieces of identification information, the performance position among the plurality of pieces of identification information. Identify identification information that temporally corresponds to the location. According to the above aspect, the performance position of the observation sound relative to the reference sound is estimated by collating the observation sound collected by the sound collecting device with the reference sound prepared in advance. Among the plurality of pieces of identification information for which such correspondence is registered, the identification information temporally corresponding to the performance position can be specified. Therefore, it is possible to specify the identification information at an appropriate point in time during the performance without having to sequentially indicate the identification information at appropriate points in parallel with the performance. The "observation sound" is typically speech (language sound such as speech produced by pronouncing a line), but also includes performance sound of a musical instrument.

In a preferred example of the first aspect (second aspect), the reference sound is generated by speech synthesis for a character string that is expected to be pronounced in the demonstration. According to the above aspect, there is an advantage that the reference sound used for matching with the observed sound can be easily prepared from a character string (for example, a script) that is expected to be pronounced in the demonstration.

In a preferred example of the first aspect (third aspect), the reference sounds are generated from a plurality of recorded sounds recorded in past demonstrations. According to the above aspect, it is possible to generate the reference sound that reflects the tendency of the recorded sound pronounced in the actual demonstration.

In a preferred example (fourth aspect) of any one of the first to third aspects (fourth aspect), the estimation of the performance position includes: an observed character string estimated by speech recognition of the observed sound; and an observed character string estimated by speech recognition of the reference sound. A common recognition model is used in the speech recognition for the observed sound and the speech recognition for the reference sound. For example, if the observed sound is an utterance of a special word such as an ancient language, accurate recognition is difficult with speech recognition using a recognition model (for example, an acoustic model and a language model) of a normal word such as a modern language. Therefore, in the configuration in which the observed character string estimated by speech recognition for the observed sound and the character string expected to be pronounced in the demonstration are matched, the two diverge, and as a result, the performance position cannot be estimated with high accuracy. Can not. According to the above-described aspect of collating the observed character string estimated by speech recognition using a common recognition model (for example, the recognition model of ordinary words such as modern language) and the reference character string, the observed sound is It is possible to estimate the position of the performance with high accuracy even if the speech is a special word.

In a preferred example (fifth aspect) of any one of the first to third aspects, in the estimation of the performance position, the feature amount extracted from the observed sound and the feature amount of the reference sound are collated to perform the Estimate the location of the demonstration. According to the above aspect, it is possible to appropriately estimate the performance position by matching the feature amount of the observed sound and the feature amount of the reference sound. A feature quantity suitable for estimating the performance position is, for example, MFCC.

In a preferred example (sixth aspect) of any one of the first to fifth aspects (sixth aspect), the specified Send identifying information. According to the above aspect, it is possible to reproduce each of the plurality of related information regarding the demonstration on the terminal device at an appropriate point in time in parallel with the demonstration.

In the preferred example (seventh mode) of any one of the first mode to the fifth mode, the identified identification information is transmitted to a stage system that executes a performance action corresponding to the received identification information. According to the above aspect, it is possible to execute the production related to the demonstration at an appropriate point in time in parallel with the demonstration. The production is, for example, sound reproduction, lighting control, or the like.

An information providing system for executing the information processing method according to any one of the first to seventh aspects, or a program for causing a computer to execute the information processing method according to any one of the first to seventh aspects of the present invention. A preferred embodiment of is realized.

<Other aspects>
According to the present invention, as exemplified below as Aspect A and Aspect B, a unit period corresponding to each of the registered intervals among the reference sounds is determined by matching a registered character string including a plurality of registered intervals with a reference sound. Also specified as a method of definition.

Aspect A of the present invention extracts a feature amount of a synthesized sound generated by speech synthesis for a registered character string, and compares the feature amount of the reference sound with the feature amount of the synthesized sound, thereby obtaining each registered sound among the reference sounds. A computer-implemented information processing method that defines a unit period corresponding to an interval.

Aspect B of the present invention estimates a synthesized character string by speech recognition of a synthesized sound generated by speech synthesis of a registered character string, and compares the reference character string estimated by speech recognition of a reference sound with the synthesized character string. This is a computer-implemented information processing method that defines a unit period corresponding to each registration interval of the reference sound, and a common recognition model is used for speech recognition of the reference sound and speech recognition of the synthesized sound. is used.

DESCRIPTION OF SYMBOLS 100... Information provision system, 200... Management system, 10... Information processing system, 11... Control device, 12... Storage device, 13... Sound collection device, 14... Sound emission device, 20... Information distribution system, 30... Terminal device, DESCRIPTION OF SYMBOLS 31... Control apparatus, 311... Information extraction part, 312... Operation control part, 32... Storage device, 33... Communication apparatus, 34... Sound collection apparatus, 35... Reproducing apparatus, 40... Communication network, 51... Position estimation part, 52 Information identification unit 53 Signal processing unit 531 Modulation processing unit 532 Synthesis processing unit 55 Analysis processing unit 60 Stage system 61 Management device 62 Production device 621 Acoustic device 622 … the lighting system.

Claims (7)

The position of the observed sound with respect to the reference sound on the time axis by matching the feature amount extracted from the observed sound collected by the sound collection device with the feature amount of the reference sound expected to be pronounced in the demonstration. Estimate the demonstration position where
Identifying identification information temporally corresponding to the performance position from among the plurality of identification information by referring to registration data representing the temporal correspondence between the reference sound and the plurality of identification information Implemented by a computer Information processing methods. 2. The information processing method according to claim 1, wherein said reference sound is a sound generated by speech synthesis for a character string to be pronounced in said demonstration. 2. The information processing method according to claim 1, wherein the reference sound is a sound generated from a plurality of recorded sounds recorded in past demonstrations. 4. The information processing method according to any one of claims 1 to 3 , wherein the specified identification information is transmitted to a terminal device that reproduces related information corresponding to the received identification information among the plurality of related information related to the demonstration. 4. The information processing method according to any one of claims 1 to 3 , wherein the identified identification information is transmitted to a stage system that executes a performance action corresponding to the received identification information. moreover,
reproducing the observed sound and the acoustic component of the identified identification information by a sound emitting device
The information processing method according to claim 1. The position of the observed sound with respect to the reference sound on the time axis by matching the feature amount extracted from the observed sound collected by the sound collection device with the feature amount of the reference sound expected to be pronounced in the demonstration. a position estimating unit that estimates a performance position that is
an information specifying unit that specifies, from among the plurality of identification information, identification information temporally corresponding to the performance position by referring to registration data representing temporal correspondence between the reference sound and the plurality of identification information; information processing system.

Images