JP2018091998A - Information processing system and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the work load for providing information concerning plural character strings pronounced in time series.SOLUTION: An information processing system 20 comprises: a time analysis unit 62 which generates time information Tn showing the time point at which a character string is pronounced on the time axis, for each of plural character strings, by analyzing character string information B which represents the plural character strings and an acoustic signal X which represents the sound pronounced for the plural character strings sequentially; and an information correspondence unit 64 which associates, for each of the plural character strings, delivery information which shows related information concerning the character string, with the time information Tn which the time analysis unit 62 generates for the character string.SELECTED DRAWING: Figure 9

Description

  The present invention relates to a technique for providing information to a terminal device.

  There has been proposed a technique for providing information related to performances of various entertainments such as plays and performances to user terminal devices in parallel with the progress of the performances. For example, Patent Document 1 discloses a configuration in which time codes are sequentially transmitted to a user's portable device in parallel with the performance of the box office, and caption information such as captions is displayed on the portable device at the time specified from the time code. Is disclosed.

JP 2009-213180 A

  However, in order to display information for explanation such as subtitles on the portable device under the technique of Patent Document 1, it is necessary to determine in advance the correspondence between the information for explanation and the time code. There is a problem that the workload of determining the relationship is heavy. In view of the above circumstances, an object of the present invention is to reduce the work load for providing a user with information on a plurality of character strings that are pronounced in time series.

In order to solve the above problems, an information processing system according to a preferred aspect of the present invention includes: character string information representing a plurality of character strings; and an acoustic signal representing a sound produced by sequentially producing the plurality of character strings. By analyzing, for each of the plurality of character strings, a time analysis unit that generates time information indicating a time point on the time axis at which the character string is pronounced, and for each of the plurality of character strings, the character string The information corresponding part which matches the delivery information which shows the relevant information relevant to, and the time information which the said time analysis part produced | generated about the said character string is comprised.
Further, in the information processing method according to a preferred aspect of the present invention, the computer system analyzes character string information representing a plurality of character strings and an acoustic signal representing a sound obtained by sequentially sounding the plurality of character strings. Then, for each of the plurality of character strings, time information indicating a point in time on which the character string is pronounced is generated, and for each of the plurality of character strings, related information related to the character string is indicated. The distribution information is associated with the time information generated for the character string.

1 is a configuration diagram of an information providing system according to a first embodiment of the present invention. It is a block diagram of an information delivery system. It is explanatory drawing of an acoustic signal and a modulation signal. It is a schematic diagram of a reference table. It is a flowchart of operation | movement of an information delivery system. It is a block diagram of a terminal device. It is a schematic diagram of a reference table. It is a flowchart of operation | movement of a terminal device. It is a block diagram of an information processing system. It is a schematic diagram of character string information. It is a flowchart of operation | movement of an information processing system. It is a block diagram of the information processing system in 2nd Embodiment.

<First Embodiment>
FIG. 1 is a configuration diagram of an information providing system 100 according to the first embodiment of the present invention. The information providing system 100 is a computer system for providing a user with information related to various types of entertainment (for example, theater, performance, or movie) in parallel with the progress. As illustrated in FIG. 1, the information providing system 100 according to the first embodiment includes an information distribution system 10 and an information processing system 20. The information distribution system 10 is installed in a facility such as a theater or a hall where various performances are held. A user who comes to the facility carries the terminal device 30. The terminal device 30 is a portable information terminal such as a mobile phone or a smartphone. Note that a plurality of terminal devices 30 actually exist in the facility, but in the following description, attention is paid to any one terminal device 30 for convenience.

  In the first embodiment, a case where a show (for example, a character show) in which a plurality of predetermined character strings (hereinafter referred to as “pronunciation character strings”) are chronologically pronounced in a predetermined order is held in the facility. Suppose. That is, the pronunciation character string of the first embodiment is, for example, dialogue, lyrics, or explanation (narration) that is pronounced at any time in parallel with the progress of the show. The user views the show with the terminal device 30 carried. The terminal device 30 sequentially displays information related to each of a plurality of pronunciation character strings that are pronounced in time series (hereinafter referred to as “related information”) in parallel with the progress of the show. In 1st Embodiment, the case where the translated sentence of a pronunciation character string is displayed on the terminal device 30 as related information is illustrated. For example, each pronunciation string is pronounced in Japanese, while an English translation corresponding to the pronunciation string is displayed as related information. According to the above configuration, by confirming the related information displayed sequentially by the terminal device 30 at the same time as the appreciation of the show, for example, foreigners who have difficulty understanding the language of the pronunciation character string can show the contents of the show. There is an advantage that can be grasped.

<Information distribution system 10>
FIG. 2 is a configuration diagram of the information distribution system 10. As illustrated in FIG. 2, the information distribution system 10 is realized by a computer system including a control device 12, a storage device 14, and a sound emission device 16. The control device 12 is a processing circuit including, for example, a CPU (Central Processing Unit), and comprehensively controls the entire information distribution system 10. The storage device 14 is configured by a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of a plurality of types of recording media, and a program executed by the control device 12 and various data used by the control device 12. Remember. The storage device 14 of the first embodiment stores the acoustic signal X and the reference table Qa.

  The acoustic signal X is a signal in the time domain that represents a sound produced by sequentially producing a plurality of pronunciation character strings. FIG. 3 is an explanatory diagram of the acoustic signal X. As illustrated in FIG. 3, the acoustic signal X of the first embodiment includes a pronunciation period An (A1, A2, A3,...) For each of a plurality of different pronunciation character strings Ln (L1, L2, L3,...). (N is a natural number). The pronunciation period An corresponding to any one pronunciation character string Ln is a period on the time axis in which the pronunciation character string Ln is pronounced. The time length of each pronunciation period An is a variable length according to the length of the pronunciation character string Ln. The acoustic signal X is generated in advance by recording a sound produced by a speaker such as a voice actor or by a known speech synthesis process. By reproducing the sound represented by the sound signal X in parallel with the show where the performers such as anime characters perform, an effect is realized in which each performer actually pronounces the sound.

  FIG. 4 is an explanatory diagram of the reference table Qa. As illustrated in FIG. 4, the reference table Qa includes, for each of a plurality of pronunciation character strings Ln, distribution information Dn (D1, D2, D3,...) Indicating related information Cn of the pronunciation character string Ln, and an acoustic signal. This is a data table in which time information Tn (T1, T2, T3,...) Indicating the time point on the time axis at which the pronunciation character string Ln is pronounced in X is associated. In the first embodiment, the distribution information Dn corresponding to any one piece of related information Cn is identification information for identifying the related information Cn. Since the phonetic character string Ln and the related information Cn correspond to each other, the distribution information Dn of the first embodiment can be rephrased as information for identifying the phonetic character string Ln. Further, the time information Tn corresponding to any one pronunciation character string Ln designates the time point (for example, the start point of the pronunciation period An) at which the pronunciation character string Ln is pronounced in the acoustic signal X. For example, the elapsed time up to the time when the pronunciation character string Ln is started is specified by the time information Tn with a predetermined time (hereinafter referred to as “reference time”) as a reference. Assuming the case where the reproduction of the sound represented by the acoustic signal X is started simultaneously with the start of the show, the reference time point is the starting point of the acoustic signal X (that is, the start time of the show). A configuration in which the time information Tn represents the time interval at each time point on the time axis at which the pronunciation character string Ln is pronounced may be employed.

  The control device 12 in FIG. 2 executes a program stored in the storage device 14 to thereby display a plurality of functions (information management unit 42 and signal processing unit 42) for sequentially displaying a plurality of related information Cn on the terminal device 30. 44). A configuration in which the function of the control device 12 is distributed to a plurality of devices or a configuration in which a dedicated electronic circuit realizes a part of the function of the control device 12 may be employed.

  The information management unit 42 sequentially selects each of a plurality of distribution information Dn corresponding to different pronunciation character strings Ln in parallel with the progress of the show. The reference table Qa is used for selecting each distribution information Dn. Specifically, when the time point specified by any one piece of time information Tn in the reference table Qa arrives, the information management unit 42 selects the distribution information Dn corresponding to the time information Tn in the reference table Qa. . For example, assuming the example of FIG. 4, the distribution information D1 is selected when the time indicated by the time information T1 arrives, and the distribution information D2 is selected when the time indicated by the time information T2 arrives. That is, every time the time point designated by the time information Tn (for example, the start point of the sound generation period An in the acoustic signal X) arrives, the distribution information Dn corresponding to the time information Tn is selected.

  The signal processing unit 44 generates an acoustic signal Z including the distribution information Dn selected by the information management unit 42 as an acoustic component. Specifically, the signal processing unit 44 of the first embodiment includes a modulation processing unit 441 and a signal synthesis unit 442. The modulation processing unit 441 generates a modulation signal Y representing an acoustic component indicating the distribution information Dn selected by the information management unit 42. Specifically, the modulation processing unit 441 performs modulation by modulation processing such as frequency modulation that modulates a carrier wave such as a sine wave of a predetermined frequency with the distribution information Dn or spread modulation of the distribution information Dn using a spread code. A signal Y is generated. The distribution information Dn is contained in the modulation signal Y as an acoustic component in a predetermined frequency band. Specifically, the frequency band of the acoustic component of the distribution information Dn is included in a range (for example, 18 kHz or more and 20 kHz or less) that exceeds the frequency band of sounds such as voices or musical sounds that the user listens in a normal environment.

  As illustrated in FIG. 3, the acoustic component of any one piece of distribution information Dn is contained in a unit period Un corresponding to the time point specified by the time information Tn corresponding to the distribution information Dn in the modulation signal Y. The In FIG. 3, a period of a predetermined length starting from the time point specified by the time information Tn is exemplified as the unit period Un. As described above, the time information Tn designates the point in time when the pronunciation character string Ln is pronounced in the acoustic signal X. Accordingly, a sound generation period An in which an arbitrary one of the sound character strings Ln of the sound signal X is sounded, and a unit period Un including a sound component of the distribution information Dn corresponding to the sound character string Ln of the modulation signal Y, and Overlap each other on the time axis. In addition, it is also possible to include the acoustic component of the distribution information Dn multiple times within one unit period Un. It is also possible to generate the distribution component Dn as an acoustic component from the time point before the time point indicated by the time information Tn (the start point of the sound generation period An).

  2 synthesizes the acoustic signal X stored in the storage device 14 and the modulation signal Y generated by the modulation processing unit 441 to generate the acoustic signal Z. Specifically, the signal synthesis unit 442 generates the acoustic signal Z by adding the acoustic signal X and the modulation signal Y in the time domain. The acoustic signal Z generated by the signal synthesis unit 442 is supplied to the sound emitting device 16. Note that a D / A converter that converts the acoustic signal Z from digital to analog and an amplifier that amplifies the acoustic signal Z are omitted for the sake of convenience.

  The sound emitting device 16 in FIG. 2 is, for example, a speaker device installed in a facility, and reproduces the sound represented by the acoustic signal Z generated by the signal processing unit 44 (signal combining unit 442) in the facility. Therefore, the uttered sound of the pronunciation character string Ln is reproduced in each pronunciation period An of the acoustic signal X, and the acoustic component of the distribution information Dn is reproduced in each unit period Un of the modulation signal Y. That is, the acoustic component of the distribution information Dn is reproduced together with the utterance sound of the pronunciation character string Ln represented by the acoustic signal X. As understood from the above description, the sound emitting device 16 of the first embodiment functions as an acoustic device that reproduces the uttered sound of the phonetic character string Ln, and in acoustic communication using sound waves as air vibration as a transmission medium. It also functions as a transmitter that transmits the distribution information Dn to the surroundings. That is, a show is constructed by sequentially reproducing the utterances of each of the plurality of pronunciation character strings Ln from the sound emitting device 16 in parallel with the performance of each performer, while the sound that pronounces each of the pronunciation character strings Ln. Is reproduced, the distribution information Dn corresponding to the pronunciation character string Ln is transmitted to the terminal device 30 by acoustic communication.

  FIG. 5 is a flowchart illustrating the operation of the information distribution system 10. For example, in response to an instruction from the show operator, the processing of FIG. The processing of FIG. 5 is executed in parallel with the operation of sequentially acquiring the acoustic signals X stored in the storage device 14 from the top and supplying them to the signal processing unit 44.

  The information management unit 42 waits until the time point specified by the unselected earliest time information Tn among the plurality of time information Tn in the reference table Qa arrives (Sa1: NO). When the time point specified by the time information Tn comes (Sa1: YES), the information management unit 42 selects the distribution information Dn corresponding to the time information Tn in the reference table Qa (Sa2).

  The modulation processing unit 441 generates a modulation signal Y representing the acoustic component of the distribution information Dn selected by the information management unit 42 (Sa3). The signal synthesis unit 442 generates an acoustic signal Z by synthesizing the modulation signal Y generated by the modulation processing unit 441 with the acoustic signal X, and supplies the acoustic signal Z to the sound emitting device 16 (Sa4). That is, the distribution information Dn is transmitted from the sound emitting device 16 by acoustic communication along with the reproduction of the uttered sound of the pronunciation character string Ln. Until transmission of all the distribution information Dn registered in the reference table Qa is completed (Sa5: NO), selection of the distribution information Dn by the information management unit 42 (Sa1, Sa2), reproduction of the pronunciation character string Ln, and the distribution Transmission of information Dn (Sa3, Sa4) is repeated sequentially for each pronunciation character string Ln. When transmission of all the distribution information Dn is completed (Sa5: YES), the processing in FIG.

<Terminal device 30>
FIG. 6 is a configuration diagram of the terminal device 30. As illustrated in FIG. 6, the terminal device 30 includes a control device 32, a storage device 34, a sound collection device 36, and a display device 38. The control device 32 is a processing circuit including, for example, a CPU, and comprehensively controls the entire terminal device 30. The storage device 34 is configured by a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of a plurality of types of recording media, and includes a program executed by the control device 32 and various data used by the control device 32. Remember.

  The storage device 34 of the first embodiment stores a reference table Qb illustrated in FIG. For example, the reference table Qb received by the terminal device 30 from the information distribution server (not shown) via a mobile communication network or a communication network such as the Internet is stored in the storage device 34. As illustrated in FIG. 7, the reference table Qb includes distribution information Dn (D1, D2, D3,...) And related information Cn (C1, C2, C3,. Is a data table corresponding to. The related information Cn in the first embodiment is text data representing a translated sentence of the pronunciation character string Ln as described above.

  The sound collection device 36 (microphone) in FIG. 6 is an acoustic device that collects ambient sounds and generates an acoustic signal V. Specifically, the sound collection device 36 generates an acoustic signal V that represents a sound reproduced by the sound emitting device 16 of the information distribution system 10 (that is, a mixed sound of the uttered sound of the pronunciation character string Ln and the acoustic component of the distribution information Dn). Generate. The A / D converter that converts the acoustic signal V generated by the sound collection device 36 from analog to digital and the amplifier (microphone amplifier) that amplifies the acoustic signal V are omitted for convenience. The display device 38 (for example, a liquid crystal display panel) displays the related information Cn under the control of the control device 32. Note that one or both of the sound collection device 36 and the display device 38 may be configured separately from the terminal device 30 and connected to the terminal device 30.

  As illustrated in FIG. 6, the control device 32 executes a program stored in the storage device 34 to provide a plurality of functions (information extraction unit 52 and reproduction control unit) for providing related information Cn to the user. 54) is realized. A configuration in which the function of the control device 32 is distributed to a plurality of devices, or a configuration in which a dedicated electronic circuit realizes a part of the function of the control device 32 may be employed.

  The information extraction unit 52 sequentially extracts the distribution information Dn from the acoustic signal V generated by the sound collection device 36. Specifically, the information extraction unit 52 emphasizes the frequency band including the acoustic component of the distribution information Dn in the acoustic signal V by, for example, a band pass filter, and performs a demodulation process corresponding to the modulation process at the time of generating the modulated signal Y. The distribution information Dn is extracted by executing the sound signal V after the enhancement. Each time the sound emitting device 16 reproduces the uttered sound of each phonetic character string Ln, the distribution information Dn corresponding to the phonetic character string Ln is extracted. As understood from the above description, the sound collection device 36 of the first embodiment is used for voice recording between the terminal devices 30 and voice recording at the time of moving image shooting, and also receives distribution information Dn by acoustic communication. Used for That is, as illustrated in FIG. 6, the sound collection device 36 and the information extraction unit 52 receive the distribution information Dn that is sequentially transmitted in parallel with the reproduction of the uttered sound of each phonetic character string Ln by acoustic communication. It functions as the communication unit 56.

  The reproduction control unit 54 causes the display device 38 to sequentially display the related information Cn of each of the plurality of pronunciation character strings Ln (that is, the translated sentence of the pronunciation character string Ln). Specifically, each time the information extraction unit 52 extracts the distribution information Dn, the reproduction control unit 54 selects the related information Cn corresponding to the distribution information Dn among the plurality of related information Cn registered in the reference table Qb. Select and display on the display device 38. Accordingly, each time the sound emitting device 16 reproduces the uttered sound of the plurality of pronunciation character strings Ln, the related information Cn corresponding to the pronunciation character string Ln is sequentially displayed on the display device 38. That is, the related information Cn of the pronunciation character string Ln is sequentially displayed in parallel with the reproduction of each pronunciation character string Ln synchronized with the performance of each performer.

  FIG. 8 is a flowchart illustrating the operation of the terminal device 30. For example, the process of FIG. 8 is started in response to an instruction from the user. When the process of FIG. 8 is started, the information extraction unit 52 determines whether or not the distribution information Dn can be extracted from the acoustic signal V generated by the sound collection device 36 (Sb1). When the information extracting unit 52 extracts the distribution information Dn (Sb1: YES), the reproduction control unit 54 searches the related information Cn corresponding to the distribution information Dn from the reference table Qb (Sb2), and displays the related information Cn. It is displayed on the device 38 (Sb3). Until the termination is instructed by the user (Sb4: NO), the retrieval (Sb2) and display (Sb3) of the related information Cn are executed each time the distribution information Dn is extracted by the information extraction unit 52. When the end is instructed by the user (Sb4: YES), the process of FIG. 8 ends.

<Information processing system 20>
The information processing system 20 in FIG. 1 is a computer system that generates the aforementioned reference table Qa (FIG. 4) that the information distribution system 10 refers to for transmission of distribution information Dn. FIG. 9 is a configuration diagram of the information processing system 20. As illustrated in FIG. 9, the information processing system 20 of the first embodiment is realized by a computer system including a control device 22 and a storage device 24. The control device 22 is a processing circuit including a CPU, for example, and comprehensively controls the entire information processing system 20. The storage device 24 is configured by a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of a plurality of types of recording media, and a program executed by the control device 22 and various data used by the control device 22. Remember. The storage device 24 of the first embodiment stores the acoustic signal X and the character string information B. As described above, the acoustic signal X is a time-domain signal representing a sound in which a plurality of pronunciation character strings Ln are sequentially generated.

  The character string information B is data representing a plurality of pronunciation character strings Ln that are pronounced in a show. For example, as illustrated in FIG. 10, a time series of a plurality of pronunciation character strings Ln that a single or a plurality of performers sequentially generate is designated by the character string information B. For example, text data (script data) representing a script in which each performer's dialogue or show commentary (narration) is arranged in time series is suitably used as the character string information B.

  The information processing system 20 generates the reference table Qa using the acoustic signal X and the character string information B stored in the storage device 24. The reference table Qa generated by the information processing system 20 is transferred to the storage device 14 of the information distribution system 10 and used for distributing the distribution information Dn to the terminal device 30 as described above. The reference table Qa is transferred to the information distribution system 10 via a communication network such as the Internet. It is also possible to transfer the reference table Qa from the information processing system 20 to the information distribution system 10 using a portable recording medium such as a semiconductor recording medium.

  The control device 22 executes a program stored in the storage device 24 to thereby generate a plurality of functions (a time analysis unit 62 and an information correspondence unit 64) for generating the reference table Qa from the acoustic signal X and the character string information B. Is realized. A configuration in which the function of the control device 22 is distributed to a plurality of devices or a configuration in which a dedicated electronic circuit realizes a part of the function of the control device 22 may be employed.

  The time analysis unit 62 analyzes the character string information B and the acoustic signal X stored in the storage device 24, so that the pronunciation character string Ln is pronounced for each of the plurality of pronunciation character strings Ln. The time information Tn indicating the time point is generated. As illustrated in FIG. 9, the time analysis unit 62 according to the first embodiment includes a character string specifying unit 621 and a matching processing unit 622.

  The character string specifying unit 621 acquires a character string (hereinafter referred to as a “recognized character string”) R that represents the pronunciation of the voice represented by the acoustic signal X. The character string specifying unit 621 of the first embodiment specifies the recognized character string R by voice recognition for the acoustic signal X. For speech recognition of the acoustic signal X, for example, a known technique such as recognition processing using an acoustic model such as a Hidden Markov Model (HMM) and a language model indicating linguistic restrictions is arbitrarily adopted. obtain. It is also possible to cause the voice recognition server with which the information processing system 20 can communicate to execute voice recognition of the acoustic signal X. For example, the character string specifying unit 621 transmits the acoustic signal X to the voice recognition server, and acquires the recognized character string R specified by the voice recognition by the voice recognition server from the voice recognition server. That is, the character string specifying unit 621 includes not only an element that itself generates the recognized character string R but also an element that acquires the recognized character string R generated by another device such as a voice recognition server.

  The collation processing unit 622 collates a plurality of pronunciation character strings Ln represented by the character string information B stored in the storage device 24 with the recognized character string R specified by the character string specifying unit 621, thereby generating a plurality of pronunciations. For each character string Ln, time information Tn indicating the time point on the time axis at which the pronunciation character string Ln is generated is generated. Since the acoustic signal X is a signal representing a sound produced by sequentially producing a plurality of pronunciation character strings Ln, the recognized character string R specified from the acoustic signal X is similar to each of the pronunciation character strings Ln represented by the character string information B. Includes parts that (ideally match). The matching processing unit 622 searches for a character string similar to each phonetic character string Ln in the recognized character string R, and in the acoustic signal X, a period on the time axis corresponding to the character string (that is, the character string is pronounced). Time information Tn indicating the time of the start point of the (period). As described above, the time information Tn designates an elapsed time with the start point of the acoustic signal X as a reference time, for example.

  For collation between the recognized character string R and each phonetic character string Ln, for example, an edit distance (Levenstein distance) for evaluating the similarity between two types of character strings is preferably used. That is, the collation processing unit 622 calculates the edit distance between the two for each of a plurality of cases in which the relative position of the pronunciation character string Ln with respect to the entire recognized character string R is changed. The position of the pronunciation character string Ln with respect to the recognized character string R is sequentially changed by, for example, a predetermined number of characters (for example, one or more characters). An overlapping portion of the recognized character string R and the pronunciation character string Ln when the edit distance is minimized is searched for as a character string similar to each pronunciation character string Ln in the recognized character string R. As understood from the above description, for each of the plurality of pronunciation character strings Ln indicated by the character string information B, time information Tn indicating the point in time when the pronunciation character string Ln is pronounced in the acoustic signal X is generated.

  For each of the plurality of phonetic character strings Ln, the information corresponding unit 64 in FIG. 9 includes distribution information Dn indicating related information Cn related to the phonetic character string Ln, and a time analysis unit 62 (matching process) for the phonetic character string Ln. The reference table Qa is generated by associating the time information Tn generated by the unit 622). The distribution information Dn is appropriately set so as not to overlap each other in the reference table Qa. The reference table Qa generated by the information corresponding unit 64 is stored in the storage device 24 and then transferred to the storage device 14 of the information distribution system 10.

  FIG. 11 is a flowchart illustrating the operation of the information processing system 20. For example, the process of FIG. 11 is started in response to an instruction from the administrator of the information processing system 20. When the processing of FIG. 11 is started, the time analysis unit 62 generates time information Tn of each phonetic character string Ln by analyzing the character string information B and the acoustic signal X as exemplified below (Sc1˜). Sc3). First, the character string specifying unit 621 specifies the recognized character string R by voice recognition for the acoustic signal X (Sc1). The matching processing unit 622 selects one phonetic character string Ln (for example, the unselected earliest phonetic character string Ln) from the time series of the plurality of phonetic character strings Ln indicated by the character string information B (Sc2). The matching processing unit 622 collates the recognized character string R of the acoustic signal X with the selected phonetic character string Ln, thereby obtaining time information Tn indicating the time point when the phonetic character string Ln is pronounced in the acoustic signal X. Generate (Sc3).

  For each of all the pronunciation character strings Ln included in the character string information B, the selection of the pronunciation character string Ln (Sc2) and the generation of the time information Tn (Sc3) are repeated (Sc4: NO). When the time information Tn is generated for all the pronunciation character strings Ln (Sc4: YES), the information correspondence unit 64 associates the distribution information Dn with the time information Tn for each of the plurality of pronunciation character strings Ln. Is stored in the storage device 24 (Sc5).

  As described above, in the first embodiment, the time information Tn of each phonetic character string Ln is generated by analyzing the character string information B and the acoustic signal X, and the distribution information Dn and time are generated for each phonetic character string Ln. A reference table Qa that associates the information Tn is generated. Therefore, it is possible to reduce the work load for providing the user of the terminal device 30 with the related information Cn of the pronunciation character string Ln that is chronologically pronounced.

Second Embodiment
A second embodiment of the present invention will be described. In addition, about the element which an effect | action or function is the same as that of 1st Embodiment in each form illustrated below, the code | symbol used by description of 1st Embodiment is diverted, and each detailed description is abbreviate | omitted suitably.

  FIG. 12 is a configuration diagram of the information processing system 20 in the second embodiment. In the second embodiment, the configuration of the time analysis unit 62 of the information processing system 20 is different from that of the first embodiment. The configurations of the information distribution system 10 and the terminal device 30 and the configurations of elements other than the time analysis unit 62 in the information processing system 20 are the same as those in the first embodiment.

  Similar to the first embodiment, the time analysis unit 62 analyzes the character string information B and the acoustic signal X stored in the storage device 24 to thereby generate the pronunciation character string for each of the plurality of pronunciation character strings Ln. Time information Tn indicating the time point on the time axis at which Ln is sounded is generated. As illustrated in FIG. 12, the time analysis unit 62 according to the second embodiment includes an estimation processing unit 624, a feature extraction unit 625, and a matching processing unit 626.

  The estimation processing unit 624 estimates the time series of the acoustic feature quantity Fa of speech observed when each of the plurality of pronunciation character strings Ln indicated by the character string information B is pronounced. A time series of the feature amount Fa is estimated for each of the plurality of pronunciation character strings Ln. The feature amount Fa is an acoustic characteristic value that tends to clearly reflect a difference in phonemes (ie, pronunciation content). For example, MFCC (Mel-Frequency Cepstrum Coefficients) is assumed as a suitable example of the feature amount Fa. The estimation processing unit 624 estimates the time series of the feature value Fa using a statistical mathematical model (hereinafter referred to as “feature value generation model”) expressing a statistical tendency of the relationship between the pronunciation character string Ln and the time series of the feature value Fa. Is used). The feature quantity generation model is expressed by, for example, a hidden Markov model (HMM), and is generated in advance by machine learning using a large number of learning data including an arbitrary character string and a feature quantity of a voice that pronounces the character string. . Therefore, when the phonetic character string Ln is assigned to the feature quantity generation model, the likelihood (specifically, the phonetic character string) with respect to the phonetic character string Ln under the tendency of the feature quantity with respect to the character string in a large number of learning data. The time series of the feature value Fa is estimated so that the posterior probability (under the condition that Ln is observed) is maximized.

  It is also possible to prepare a feature quantity generation model in advance for each performer of the show, and estimate the time series of the feature quantity Fa by selectively using one of a plurality of feature quantity generation models for each performer. It is. For example, assuming a configuration in which a performer who pronounces each phonetic character string Ln is specified by the character string information B, the estimation processing unit 624 generates the pronunciation of the phonetic character string Ln for any one phonetic character string Ln. A time series of the feature value Fa is generated using the feature value generation model of the performer designated by the character string information B as the performer. It is also possible to selectively use a plurality of feature quantity generation models prepared for each performer attribute (for example, adult / child, male / female).

  The feature extraction unit 625 extracts a time series of the feature amount Fb of the acoustic signal X. The feature amount Fb is the same acoustic characteristic value (for example, MFCC) as the feature amount Fa. A known analysis technique is arbitrarily employed for extracting the feature value Fb of the acoustic signal X. Note that the types of the feature value Fa and the feature value Fb are arbitrary. For example, it is also possible to use a chroma vector (PCP: Pitch Class Profile) including a plurality of elements corresponding to different scales (for example, each of 12 semitones of equal temperament) as the feature value Fa and the feature value Fb together with the MFCC. is there. The element corresponding to any one scale sound in the chroma vector is set to a numerical value obtained by adding or averaging the intensities of the band components corresponding to the scale sound over a plurality of octaves.

  The collation processing unit 626 collates the time series of the feature amount Fa estimated by the estimation processing unit 624 for each phonetic character string Ln with the time series of the feature amount Fb extracted from the acoustic signal X by the feature extraction unit 625. Thus, for each of the plurality of pronunciation character strings Ln designated by the character string information B, time information Tn indicating the time point on the time axis at which the pronunciation character string Ln is pronounced is generated. As described above, the acoustic signal X is a signal representing a sound in which a plurality of pronunciation character strings Ln are sequentially pronounced. Therefore, the time series of the feature amount Fa estimated for any one pronunciation character string Ln is similar to the time series of the feature quantity Fb of the portion of the acoustic signal X where the pronunciation character string Ln is pronounced (ideal Tend to match). Considering the above tendency, the matching processing unit 626 of the second embodiment searches for a portion similar to the time series of the feature quantity Fa of the pronunciation character string Ln in the time series of the feature quantity Fb of the acoustic signal X, Time information Tn indicating the time of the start point of the portion of the acoustic signal X is generated for the pronunciation character string Ln.

<Modification>
Each aspect illustrated above can be variously modified. Specific modifications are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range that does not contradict each other.

(1) As described above, for example, script data for writing a show script can be suitably used as the character string information B. However, the script data can include not only the time series of the plurality of pronunciation character strings Ln but also information that is not actually pronounced in the show (hereinafter referred to as “non-speech information”). For example, a show scene, a commentary on a character, or a cautionary note on performance is assumed as non-pronunciation information. Since non-pronunciation information is not required for collation between the character string information B and the sound signal X by the time analysis unit 62, data obtained by excluding non-pronunciation information from the script data is regarded as character string information B (622 or 626). Is preferable to contrast with the acoustic signal X.

(2) In the above-described embodiments, the translated sentence of the pronunciation character string Ln is exemplified as the related information Cn. However, the content of the related information Cn is not limited to the above examples. For example, the pronunciation character string Ln itself can be displayed on the display device 38 of the terminal device 30 as the related information Cn. According to the above configuration, for example, there is an advantage that a hearing impaired person who is difficult to listen to the pronunciation character string Ln pronounced in the show can grasp the contents of the show.

  Further, in addition to information that directly represents the content (line or lyrics) of the pronunciation character string Ln, information that explains the show (for example, information on the script or supplementary explanation of the show) is used as related information Cn. It is also possible to display on the display device 38 of the device 30. For example, related information Cn including information indicating the phonetic character string Ln itself or a translation thereof and information indicating a character who pronounces the phonetic character string Ln (for example, the name or icon of the character) may be displayed. Further, when the pronunciation character string Ln is the lyrics of a song (when a song having the pronunciation character string as a lyrics is sung in a show), music information (for example, a score or a chord) related to the song is used as related information Cn. The configuration presented in (1) is also suitable. The related information Cn is not limited to text data representing a character string. For example, it is possible to use sound or images representing various types of information as the related information Cn.

  For supplementary related information Cn such as music information or show commentary, collation between the pronunciation character string Ln and the recognized character string R (first embodiment), or the time series of the feature value Fa and the feature value Fb In the collation with the series (second embodiment), it is difficult to specify the time information Tn. Therefore, in the reference table Qa, the supplementary related information Cn illustrated above corresponds to the time point indicated by the time information Tn-1 corresponding to the immediately preceding related information Cn-1 and the immediately following related information Cn + 1. Time information Tn indicating a time point between the time point indicated by the time information Tn + 1 to be generated is generated. For example, the time information Tn indicating the midpoint between the time point indicated by the time information Tn-1 and the time point indicated by the time information Tn + 1 is associated with the distribution information Dn of the related information Cn. A configuration in which the time information Tn indicates a time point when the administrator of the information processing system 20 indicates between the time point indicated by the time information Tn-1 and the time point indicated by the time information Tn + 1 is also preferable.

(3) As described above, the phonetic character string Ln can include not only dialogue but also lyrics. That is, the case where the acoustic signal X represents the singing voice sounded in parallel with the musical accompaniment sound is assumed. Assuming the case where the acoustic signal X represents a singing voice, in the specification of the recognized character string R by the character string specifying unit 621 of the first embodiment, a configuration in which the acoustic signal X is separated into an accompaniment sound and a singing voice is preferable. is there. The recognition character string R is specified by voice recognition for the separated singing voice. For the separation of the accompaniment sound and the singing sound, for example, a well-known sound source separation using a difference in position where the sound source is localized is preferably used. According to the above configuration, since voice recognition is performed on the singing voice from which the accompaniment sound is excluded from the acoustic signal X, there is an advantage that the recognized character string R can be specified with high accuracy.

(4) When the related information Cn is displayed on the display device 38 of the terminal device 30, it is possible to adjust the related information Cn so that the related information Cn is contained in one screen of the display device 38. For example, when the number of characters represented by the related information Cn is large, a configuration in which the related information Cn is displayed on one screen of the display device 38 by reducing the size of the character string is preferable. For example, assuming that the pronunciation character string Ln is lyrics, the related information Cn corresponding to the pronunciation character string Ln is obtained for each phrase of the song (for example, each section into which the song is divided according to musical meaning). It can also be displayed.

(5) In each of the above embodiments, the related information Cn is displayed on the display device 38 of the terminal device 30, but the method of presenting the related information Cn to the user is not limited to display. For example, it is also possible to present the related information Cn to the user by emitting sound representing the related information Cn from the sound emitting device 16 such as a speaker or an earphone. Moreover, in each above-mentioned form, although portable information terminals, such as a mobile telephone or a smart phone, were illustrated as the terminal device 30, the specific form of the terminal device 30 is not limited to the above illustration. For example, a display device for guidance such as an electric bulletin board installed in a facility of a railway operator (for example, a station premises) or an electronic signboard (for example, digital signage) installed in a commercial facility can be suitably used as the terminal device 30. .

(6) The scene in which the information providing system 100 (particularly the information distribution system 10) is used is not limited to the shows exemplified in the above embodiments. For example, the information provided in the above-mentioned forms is also provided in a scene where sounds for guiding various facilities such as transportation facilities (for example, trains, buses), exhibition facilities (for example, museums, art museums), or tourist facilities are chronologically pronounced. System 100 may be utilized.

(7) In the above-described embodiments, the configuration in which the distribution information Dn is included in real time (that is, the modulation signal Y is synthesized) in parallel with the reproduction of the acoustic signal X is exemplified. It is also possible to generate Z in advance and hold it in the storage device 14. For the generation of the acoustic signal Z, the reference table Qa generated by the information processing system 20 is used as illustrated in the above-described embodiments. That is, the acoustic signal Z is generated by synthesizing the modulation signal Y of the distribution information Dn corresponding to the time information Tn at the time indicated by the time information Tn of the reference table Qa in the acoustic signal X. In the configuration in which the acoustic signal Z is generated in advance, the information management unit 42 and the signal processing unit 44 are omitted from the information distribution system 10, and the control device 12 sends the acoustic signal Z stored in the storage device 14 to the sound emitting device 16. Supply.

(8) In each of the above embodiments, the identification information for identifying the related information Cn is exemplified as the distribution information Dn. However, the related information Cn itself may be transmitted from the information distribution system 10 to the terminal device 30 as the distribution information Dn. Is possible. In the configuration in which the related information Cn is distributed as the distribution information Dn, the terminal device 30 does not need to hold the reference table Qb. As understood from the above examples, the distribution information Dn is comprehensively expressed as information indicating the related information Cn, and includes the related information Cn itself in addition to the identification information for identifying the related information Cn.

(9) In each of the embodiments described above, the distribution information Dn is transmitted to the terminal device 30 by acoustic communication using sound waves as a transmission medium. However, the communication method for transmitting the distribution information Dn to the terminal device 30 is not limited to acoustic communication. . For example, the distribution information Dn can be transmitted to the terminal device 30 by wireless communication using electromagnetic waves such as radio waves and infrared rays in synchronization with the sound emission of the pronunciation character string Ln by the sound emission device 16. For the transmission of the distribution information Dn, short-range wireless communication that does not involve a communication network such as a mobile communication network is suitable. Acoustic communication using sound waves as a transmission medium and wireless communication using electromagnetic waves as a transmission medium are short-range wireless communication. This is an example. In addition, according to the acoustic communication exemplified in each of the above-described embodiments, the sound emitting device 16 for emitting the sound of the pronunciation character string Ln can be used for transmission of the distribution information Dn and, for example, by the installation of a sound insulation wall There is an advantage that the communication range can be easily controlled.

(10) In each of the above-described embodiments, the case where the reference table Qb including the related information Cn is stored in the storage device 34 of the terminal device 30 is exemplified. However, the terminal device is connected via a mobile communication network or a communication network such as the Internet. It is also possible to hold the reference table Qb in an information distribution server capable of communicating with 30. Specifically, the reproduction control unit 54 of the terminal device 30 transmits an information request specifying the distribution information Dn extracted by the information extraction unit 52 to the information distribution server. The information distribution server searches the reference table Qb for related information Cn corresponding to the distribution information Dn specified in the information request and transmits it to the requesting terminal device 30. The reproduction control unit 54 of the terminal device 30 causes the display device 38 to display the related information Cn received from the information distribution server. However, according to the configuration in which the reference table Qb is held in the storage device 34 of the terminal device 30 as in each of the above-described embodiments, there is an advantage that the related information Cn can be displayed without requiring communication through the communication network.

(11) The first embodiment exemplifies a configuration in which the collation processing unit 622 collates each phonetic character string Ln and the recognized character string R. In the second embodiment, the time series of the feature amount Fa and the feature amount Fb The collation processing unit 626 collates the time series, but various devices can be applied to the collation processing by the collation processing unit (622 or 626), for example, as exemplified below.

  For example, it is possible to divide each show scene (for example, each of a plurality of scenes such as the first curtain, the second curtain, the utterance scene, and the singing scene) and execute the collation process. A configuration in which time information Tn is generated for any one phonetic character string Ln based on the result of collation processing related to the previous phonetic character string Ln may be employed. It is also possible to generate time information Tn of each phonetic character string Ln by referring to a moving image in which a show is photographed in collation processing. When the pronunciation character string Ln is the lyrics of the music, the time information Tn of each phonetic character string Ln is obtained by referring to the pitch or the pronunciation period (or rhythm) of each note specified by the musical score information of the music. The structure to generate is also suitable.

  It is also possible to execute the collation process after restricting the condition of the time information Tn. For example, when the provisional time information Tn for each phonetic character string Ln is generated by the collation processing by the collation processing unit, the administrator of the information processing system 20 knows about the show (for example, information that can be grasped from the script). ) Specifies one or more pieces of time information Tn that can be determined to be appropriate. For example, a flag set to an initial value indicating that the suitability of the time information Tn is unknown is added to each of the plurality of time information Tn, and the flag of the time information Tn designated by the administrator is appropriate. It is changed to a numerical value that means there is. Specifically, when the time at which the first pronunciation character string Ln is pronounced for each scene in the show is known in advance, the administrator can specify appropriate time information Tn based on that knowledge. .

  For example, when the user designates that the time information Tn1 and the time information Tn2 (n1 ≠ n2) are appropriate, the collation processing unit sets each time information Tm (n1 <m <n2) within the range from the numerical value n1 to the numerical value n2. ) Is performed under the constraint that the time point indicated by the time information Tm is located between the time point of the time information Tn1 and the time information Tn2. According to the above configuration, it is possible to improve the accuracy of the analysis of the time information Tn by reflecting the manual instruction (prior knowledge about the show) of the manager.

  Further, a restriction on the utterance or singing speed of the pronunciation character string Ln (for example, the tempo of singing or the utterance speed) is added to the feature value generation model for estimating the time series of the feature value Fa from the pronunciation character string Ln. Is also possible. For example, a semi-hidden Markov model (semi-HMM) that reflects the singing speed (tempo) specified in the music or the utterance speed specified in the script (for example, “pronounced early”) By using it as a generation model, it is possible to estimate the time series of the feature value Fa under restrictions on the speed of singing or speaking. Although the semi-hidden Markov model has been illustrated in the above examples, the same action can be realized even with the hidden Markov model if the transition probability between the states is appropriately set.

(12) As described above, a configuration in which collation processing is performed for each show scene can be assumed. The switching of the scene can be detected, for example, by analyzing a moving image generated by shooting the stage. Specifically, the presence or absence of dark transition is determined by referring to a change in brightness (for example, a cumulative value of brightness within one screen) calculated from a moving image. For example, when the amount of decrease in lightness exceeds a predetermined threshold, it is determined that a stage darkening has occurred. The collation process can be executed with the point in time when the stage darkening occurs as the point of scene switching. However, if the scene does not switch even if the stage is dark (for example, if the stage is dark in one scene), or if the scene is switched without the stage being dark (for example, approximately the same brightness) It is also possible to assume that the scene changes while maintaining Considering the above circumstances, for example, a well-known machine learning is used to express a probability model that expresses the probability that darkness will occur at the time of scene switching and a probability model that expresses the probability that darkness will occur at times other than scene switching. It is preferable to use the probability model for determining whether or not scenes are switched using both probability models.

(13) As illustrated in the above embodiments, the information processing system 20 is realized by cooperation of the control device 22 and a program. The program according to a preferred aspect of the present invention analyzes a plurality of character string information B representing a plurality of pronunciation character strings Ln and an acoustic signal X representing a sound obtained by sequentially sounding the plurality of pronunciation character strings Ln. For each of the phonetic character strings Ln, the time analysis unit 62 for generating time information Tn indicating the time point on the time axis at which the phonetic character string Ln is played, and for each of the plurality of phonetic character strings Ln, The computer is caused to function as the information corresponding unit 64 that associates the distribution information Dn indicating the related information Cn related to the character string Ln with the time information Tn generated by the time analysis unit 62 for the pronunciation character string Ln.

  The programs exemplified above can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium This type of recording medium can be included. Note that the non-transitory recording medium includes any recording medium except for a transient propagation signal (transitory, propagating signal), and does not exclude a volatile recording medium. It is also possible to provide a program to a computer in the form of distribution via a communication network.

(14) From the forms exemplified above, for example, the following aspects are grasped.
<Aspect 1>
An information processing system according to a preferred aspect (aspect 1) of the present invention analyzes character string information representing a plurality of character strings and an acoustic signal representing sound that is produced by sequentially producing the plurality of character strings, For each of the plurality of character strings, a time analysis unit that generates time information indicating a point in time on which the character string is pronounced, and for each of the plurality of character strings, related information related to the character string The information corresponding | compatible part which matches the delivery information which shows, and the time information which the said time analysis part produced | generated about the said character string is comprised. In the above aspect, the time information of each character string is generated by analyzing the character string information and the sound signal, and the distribution information and the time information are associated with each character string. Therefore, it is possible to reduce the work load for providing each piece of related information of a plurality of character strings pronounced in time series.
<Aspect 2>
In a preferred example of aspect 1 (aspect 2), the time analysis unit includes a character string specifying unit that obtains a recognized character string representing the pronunciation content of the sound represented by the acoustic signal, and the plurality of characters represented by the character string information. Collation that generates time information indicating the time point on the time axis at which the character string is pronounced for each of the plurality of character strings by collating the string with the recognized character string acquired by the character string specifying unit And a processing unit.
<Aspect 3>
In a preferred example of aspect 1 (aspect 3), the time analysis unit includes: an estimation processing unit that estimates a feature amount of a sound that pronounces the plurality of character strings; and a feature extraction unit that extracts a feature amount of the acoustic signal; By comparing the feature amount estimated by the estimation processing unit with the feature amount extracted by the feature extraction unit, for each of the plurality of character strings, a time point on the time axis at which the character string is pronounced is indicated. And a verification processing unit that generates time information.
<Aspect 4>
An information processing method according to a preferred aspect (aspect 4) of the present invention is such that a computer system analyzes character string information representing a plurality of character strings and an acoustic signal representing a sound produced by sequentially producing the plurality of character strings. For each of the plurality of character strings, time information indicating a point in time on which the character string is pronounced is generated, and for each of the plurality of character strings, related information related to the character string is generated. Is associated with the time information generated for the character string.

DESCRIPTION OF SYMBOLS 100 ... Information provision system, 10 ... Information distribution system, 12, 22, 32 ... Control device, 14, 24, 34 ... Storage device, 16 ... Sound emitting device, 20 ... Information processing system, 30 ... Terminal device, 36 ... Collection Sound device, 38 ... display device, 42 ... information management unit, 44 ... signal processing unit, 441 ... modulation processing unit, 442 ... signal synthesis unit, 52 ... information extraction unit, 54 ... reproduction control unit, 56 ... acoustic communication unit, 62 ... time analysis unit, 621 ... character string specifying unit, 622 ... collation processing unit, 624 ... estimation processing unit, 625 ... feature extraction unit, 626 ... collation processing unit, 64 ... information correspondence unit.

Claims (4)

By analyzing character string information representing a plurality of character strings and an acoustic signal representing a sound obtained by sequentially sounding the plurality of character strings, the time during which the character strings are pronounced for each of the plurality of character strings A time analysis unit for generating time information indicating a time point on the axis;
An information processing system comprising, for each of the plurality of character strings, an information corresponding unit that associates distribution information indicating related information related to the character string with time information generated by the time analysis unit for the character string. . The time analysis unit
A character string specifying unit for acquiring a recognized character string representing the pronunciation content of the sound represented by the acoustic signal;
By collating the plurality of character strings represented by the character string information with the recognized character string acquired by the character string specifying unit, for each of the plurality of character strings, on the time axis where the character string is pronounced The information processing system of Claim 1. The collation process part which produces | generates the time information which shows this time is included. The time analysis unit
An estimation processing unit that estimates a feature amount of a sound produced by pronounced the plurality of character strings;
A feature extraction unit for extracting a feature amount of the acoustic signal;
A time indicating a time point on the time axis at which the character string is pronounced for each of the plurality of character strings by collating the feature amount estimated by the estimation processing unit with the feature amount extracted by the feature extraction unit. The information processing system according to claim 1, further comprising: a matching processing unit that generates information. Computer system
By analyzing character string information representing a plurality of character strings and an acoustic signal representing a sound obtained by sequentially sounding the plurality of character strings, the time during which the character strings are pronounced for each of the plurality of character strings Generate time information indicating the time on the axis,
An information processing method that associates, for each of the plurality of character strings, distribution information indicating related information related to the character string with time information generated for the character string.

Images