WO2019182075A1

WO2019182075A1 - Information processing method and information processing device

Info

Publication number: WO2019182075A1
Application number: PCT/JP2019/011933
Authority: WO
Inventors: 佳孝浦谷; 克己石川; 康之介加藤
Original assignee: ヤマハ株式会社
Priority date: 2018-03-23
Filing date: 2019-03-20
Publication date: 2019-09-26
Also published as: JP2019169852A; JP7102826B2

Abstract

The purpose of the present invention is to achieve temporal correspondence in playback between a plurality of contents, without embedding synchronization data in the contents. This information processing device (30) is provided with: an analysis unit (352) which compares a sound signal (Y) generated by collecting a sound (Vc) emitted by playing back a first content (C1) with a reference signal (Z) temporally corresponding to a second content (C2), thereby analyzing temporal correspondence between the first content (C1) and the second content (C2); and a playback control unit (353) which, on the basis of the temporal correspondence analyzed by the analysis unit (352), causes a playback device (33) to play back the second content (C2) in conjunction with the playback of the first content (C1).

Description

Information processing method and information processing apparatus

This disclosure relates to a technology for reproducing content.

A technology for playing back other content in conjunction with playback of specific content has been proposed. For example, in Patent Document 1, watermark data including a time code is embedded in the sound of content reproduced in a movie theater or the like. The user terminal of the user who uses the movie theater reproduces another content in conjunction with the reproduction of the content by detecting the sound.

Japanese Patent No. 6163680

However, the technique of Patent Document 1 has a problem that it is necessary to embed watermark data in the content in advance. In addition, there is a problem that the audio of the content changes by embedding the watermark data for synchronization. In view of the above circumstances, it is an object of the present disclosure to temporally support the reproduction of a plurality of contents without embedding synchronization data in the contents.

In order to solve the above-described problem, an information processing method according to an aspect of the present disclosure temporally corresponds to an acoustic signal generated by collecting sound emitted by playing back the first content and the second content. The temporal correspondence between the first content and the second content is analyzed by comparing with the reference signal being played, and the reproduction of the first content is performed based on the analyzed temporal correspondence. The second content is played back by the playback device in conjunction with the above.
An information processing apparatus according to an aspect of the present disclosure compares an acoustic signal generated by collecting sound emitted by playing back first content and a reference signal temporally corresponding to the second content. Thus, an analysis unit that analyzes a temporal correspondence between the first content and the second content, and a time correspondence analyzed by the analysis unit, in conjunction with the reproduction of the first content A reproduction control unit for causing the reproduction apparatus to reproduce the second content.

It is a block diagram which illustrates the composition of a playback system. It is a schematic diagram of a 1st content, a reference signal, and a 2nd content. It is an example of a display of the 2nd content by a reproducing | regenerating apparatus. It is a flowchart of a process of a control apparatus. It is a schematic diagram of the 1st content which concerns on a modification, a reference signal, and 2nd content. It is a schematic diagram of the 2nd content which concerns on a modification.

FIG. 1 is a block diagram illustrating a configuration of a playback system 100 according to an aspect of the present disclosure. The reproduction system 100 is a computer system for reproducing the first content C1 and the second content C2 in conjunction with each other. The playback system 100 of this embodiment includes a playback device 20 and an information processing device 30.

The playback device 20 is an output device that plays back the first content C1. The first content C1 is a moving image work composed of sound Vc and video Mc. Specifically, the first content C1 is represented by an audio signal representing the sound Vc and a video signal representing the video Mc. A movie or a television program is exemplified as the first content C1. For example, the playback device 20 plays back the first content C1 stored on an optical disk such as a DVD. The playback device 20 includes a display device 23 (for example, a liquid crystal display panel) that displays the video Mc included in the first content C1, and a sound emitting device 21 (for example, a speaker) that emits the sound included in the first content C1. It has. That is, the reproduction by the reproduction apparatus 20 includes video display and sound emission. Note that the playback device 20 may include only the sound emitting device 21. That is, the display device 23 can be omitted from the playback device 20.

The information processing apparatus 30 is carried by a user who views the first content C1 reproduced by the reproduction apparatus 20. For example, an information terminal such as a mobile phone, a smartphone, or a tablet terminal is used as the information processing apparatus 30. The information processing apparatus 30 according to the present embodiment plays back the second content C2. Specifically, the second content C2 is played back in conjunction with the playback of the first content C1 by the playback device 20. The second content C2 is information (for example, sound or image) related to the first content C1, for example. For example, a privilege video for the purchaser of the first content C1 is provided to the information processing apparatus 30 as the second content C2. Information that is continuous over the entire time axis (from the start point to the end point) of the first content C1 is exemplified as the second content C2. The second content C2 of the present embodiment is an image showing a time series of subtitles (for example, a character string representing the line of characters) representing the details of the first content C1, for example. The second content C2 may include a section where no image is actually displayed.

The information processing apparatus 30 includes a sound collection device 31, a reproduction device 33, a control device 35, and a storage device 37. The control device 35 is a processing circuit such as a CPU (Central Processing Unit), and controls each element constituting the information processing device 30. The control device 35 includes at least one circuit. The playback device 33 is an output device that plays back various types of information under the control of the control device 35. For example, the playback device 33 is a liquid crystal display panel.

The storage device 37 (memory) is configured by a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of a plurality of types of recording media, and is used by a program executed by the control device 35 and the control device 35 Stores various data. The storage device 37 of the present embodiment stores the second content C2. The second content C2 is acquired in advance from, for example, a distribution device (for example, a WEB server). Note that a storage device 37 (for example, a cloud storage) separate from the information processing device 30 is prepared, and the control device 35 performs writing and reading with respect to the storage device 37 via a mobile communication network or a communication network such as the Internet. May be. That is, the storage device 37 can be omitted from the information processing device 30.

The sound collection device 31 is an acoustic device (microphone) that collects surrounding sounds. Specifically, the sound collection device 31 collects the sound Vc emitted by the reproduction of the first content C1, and generates the sound signal Y representing the waveform of the sound Vc.

FIG. 2 schematically shows the first content C1. The sound Vc of the portion (hereinafter referred to as “playback portion”) T1 of the first content C1 played by the playback device 20 is collected. For example, sound collection by the sound collection device 31 is started in response to a user operation on the information processing device 30. Actually, when sound collection by the sound collection device 31 is started, the sound Vc of the reproduction portion T1 is continuously collected over a predetermined period (for example, a period sufficiently short with respect to the time length of the first content C1). The As the playback of the first content C1 progresses, the playback portion T1 moves in the future direction (direction in which time passes) on the time axis.

The control device 35 realizes a plurality of functions (analysis unit 352 and reproduction control unit 354) for reproducing the second content C2 by executing a plurality of tasks according to the program stored in the storage device 37. Note that the function of the control device 35 may be realized by a set of a plurality of devices (that is, a system), or part or all of the function of the control device 35 is realized by a dedicated electronic circuit (for example, a signal processing circuit). Also good.

The analysis unit 352 analyzes the temporal correspondence between the first content C1 and the second content C2. The analysis unit 352 of the present embodiment identifies a portion (hereinafter referred to as “processing portion”) T2 that temporally corresponds to the reproduction portion T1 of the first content C1 in the second content C2. By comparing the acoustic signal Y generated by the sound collection device 31 with the reference signal Z stored in the storage device 37, the processing portion T2 is specified.

The reference signal Z is a signal temporally corresponding to the second content C2. Specifically, the reference signal Z has the same time axis as the second content C2. In the present embodiment, the start point of the reference signal Z and the start point of the second content C2 match, and the end point of the reference signal Z and the end point of the second content C2 match. For example, a signal representing the waveform of the sound Vc emitted by the reproduction of the first content C1 is used as the reference signal Z. The reference signal Z is a signal representing the sound Vc over the entire time axis of the first content C1. The reference signal Z is distributed from the distribution device together with the second content C2, for example, and stored in the storage device 37 in advance.

Specifically, the analysis unit 352 specifies a portion P (hereinafter referred to as “similar portion”) P similar to the waveform of the acoustic signal Y in the reference signal Z. The reference signal Z is divided into a plurality of sections (hereinafter referred to as “analysis sections”) on the time axis, and an index (hereinafter referred to as “similar index”) indicating similarity to the acoustic signal Y is calculated for each analysis section. The time length of the analysis section is equal to the time length of the acoustic signal Y, for example. Each analysis interval may overlap each other.

The cross-correlation calculated for the analysis section and the acoustic signal Y is used as a similarity index between the analysis section and the acoustic signal Y. When a common sound is included in the analysis section and the sound signal Y (that is, when the similarity is high), the absolute value of the cross-correlation becomes maximum. An analysis section in which the absolute value of the cross-correlation is maximum among the plurality of analysis sections is specified as the similar portion P. That is, by calculating the cross-correlation between the acoustic signal Y and the reference signal Z, the acoustic signal Y and the reference signal Z are compared. The analysis unit 352 specifies a portion of the second content C2 that corresponds temporally to the similar portion P of the reference signal Z as the processing portion T2. A portion of the second content C2 that matches the similar portion P on the time axis is specified as the processing portion T2. The analysis unit 352 analyzes the temporal correspondence between the first content C1 and the second content C2 by comparing the acoustic signal Y and the reference signal Z (that is, specifies the processing portion T2).

The playback control unit 354 causes the playback device 33 to play back the second content C2 in conjunction with the playback of the first content C1, based on the temporal correspondence between the first content C1 and the second content C2. Specifically, the reproduction control unit 354 reproduces the second content C2 in parallel with the reproduction of the first content C1. The reproduction control unit 354 causes the processing portion T2 of the second content C2 to coincide with the reproduction portion T1 of the first content C1 on the time axis (that is, in synchronization) and reproduces the second content C2. Specifically, the second content C2 is played back by the playback device 33 so that the end point of the processing part T2 coincides with the end point of the playback part T1 on the time axis.

FIG. 3 is a display example of the second content C2 by the playback device 33. The playback device 33 plays back the second content C2 in conjunction with the playback of the first content C1 under the control of the playback control unit 354. Specifically, the playback device 33 plays back the second content C2 by matching the processing portion T2 of the second content C2 with the playback portion T1 of the first content C1 in time. The playback of the second content C2 also proceeds in conjunction with the progress of the playback of the first content C1. That is, the second content C2 is played back in parallel with the playback of the first content C1 by the playback device 20. For example, the second content C2 representing the subtitles "Hello" representing the words of the characters appearing in the reproduced portion T1 of the first content C1 are displayed.

FIG. 4 is a flowchart of a process in which the control device 35 reproduces the second content C2. The process of FIG. 4 is started, for example, triggered by sound collection of the sound Vc (generation of the sound signal Y) by the sound collection device 31. When the process is started, the analysis unit 352 specifies a similar part P similar to the acoustic signal Y of the first content C1 in the reference signal Z (Sa1). Specifically, among the plurality of analysis sections in the reference signal Z, an analysis section in which the similar index (absolute value of cross-correlation) calculated for the analysis section and the sound signal Y is specified as the similar portion P. The The analysis unit 352 identifies a processing portion T2 that temporally corresponds to the similar portion P in the second content C2 (Sa2). Steps Sa1 and Sa2 are processes for analyzing the temporal correspondence between the first content C1 and the second content C2 by comparing the acoustic signal Y with the reference signal Z. The reproduction control unit 354 causes the reproduction device 33 to reproduce the second content C2 in parallel with the reproduction of the first content C1 based on the temporal correspondence between the first content C1 and the second content C2 (Sa3). .

For example, in a configuration using synchronization data (watermark data in the technique of Patent Document 1) for temporally corresponding the playback of the second content C2 to the playback of the first content C1, the first content C1 is configured. The data needs to be embedded in the acoustic signal in advance. On the other hand, in this embodiment, the temporal correspondence between the first content C1 and the second content C2 is analyzed by comparing the acoustic signal Y and the reference signal Z, and the analyzed temporal correspondence is analyzed. Then, the second content C2 is reproduced in conjunction with the reproduction of the first content C1. Therefore, it is not necessary to embed data for making the reproduction of the second content C2 temporally correspond to (ie, synchronize) the reproduction of the first content C1 in the first content C1. In addition, there is an advantage that a change in sound due to embedding synchronization data in an acoustic signal does not occur according to the present embodiment.

In the present embodiment, since the acoustic signal Y and the reference signal Z are compared by calculating the cross-correlation between the acoustic signal Y and the reference signal Z, the waveform between the acoustic signal Y and the reference signal Z is compared. Thus, the reproduction of the second content C2 can be linked with the reproduction of the first content C1 with high accuracy.

<Modification>
A modification is shown below. Two or more aspects arbitrarily selected from the following examples may be appropriately combined as long as they do not contradict each other.

(1) The second content C2 is reproduced in parallel with the reproduction of the first content C1, but the second content C2 may be reproduced subsequent to the reproduction of the first content C1, for example. For example, the second content C2 is reproduced immediately after the first content C1 ends. In the above configuration, the end point of the reference signal Z coincides with the start point of the second content C2. That is, the reference signal Z temporally corresponds to the second content C2. The reproduction control unit 354 causes the reproduction device 33 to reproduce the second content C2 when the time elapses from the end point of the similar portion P of the reference signal Z to the end point of the reference signal Z. The configuration for reproducing the second content C in conjunction with the reproduction of the first content C1 includes the configuration for reproducing the second content C2 in parallel with the reproduction of the first content C1, and the reproduction of the first content C1. And a configuration for reproducing the second content C2. However, according to the above-described embodiment in which the second content C2 is reproduced in parallel with the reproduction of the first content C1, the second content C2 is reproduced subsequent to the reproduction of the first content C1, There is an advantage that the user can easily grasp the temporal correspondence between the first content C1 and the second content C2.

(2) Although the signal representing the sound Vc emitted by the reproduction of the first content C1 is exemplified as the reference signal Z, the reference signal Z is not limited to the above illustration. For example, a signal obtained by reducing the amount of information from the sound Vc (for example, a signal whose sound quality has been reduced by data compression on the sound Vc signal) may be used as the reference signal Z.

Further, the reference signal Z is not limited to a signal representing the sound Vc. For example, a signal generated by performing some processing on the sound Vc may be used as the reference signal Z. For example, a signal obtained by extracting a portion of the sound Vc of the first content C1 whose volume exceeds a predetermined value or a signal in which a pulse is arranged at the rising point (attack portion) of the sound Vc of the first content C1 is the reference signal Z. It is good. However, according to the configuration in which the signal representing the sound Vc emitted by the reproduction of the first content C1 is used as the reference signal Z, it is not necessary to prepare the reference signal Z separately from the first content C1. As long as the signal can be temporally analyzed with the acoustic signal Y, the target and waveform represented by the reference signal Z are arbitrary.

(3) The signal representing the sound Vc throughout the time axis of the first content C1 is used as the reference signal Z. For example, the sound Vc of the first content C1 is used for each of a plurality of sections separated from each other on the time axis. The signal to be represented may be the reference signal Z. That is, a signal obtained by intermittently removing the sound Vc of the first content C1 is used as the reference signal Z.

(4) Although the first content C1 is composed of the sound Vc and the video Mc, the first content C1 may be composed of only the sound Vc. That is, it is not essential that the first content C1 includes the video Mc.

(5) In the above-described embodiment, the image showing the caption of the first content C1 is exemplified as the second content C2. However, the second content C2 is not limited to the image showing the caption. For example, an image (still image or video) that introduces an actor appearing in the first content C1 or a story of the first content C1 may be used as the second content C2. Further, the sound related to the first content C1 may be used as the second content C2. In the configuration in which the second content C2 includes sound, a sound emitting device (for example, a speaker) that emits the sound is used as the playback device 33. Further, the second content C2 may be composed of both sound and images. That is, reproduction by the reproduction device 33 includes image display and sound emission. Various types of information related to the first content C1 can be adopted as the second content C2. Further, information irrelevant to the first content C1 (for example, advertisement) may be used as the second content C2.

In addition, when the sound related to the sound Vc of the first content C1 (for example, the sound represented by the sound Vc included in the first content C1 with high sound quality) is the second content C2, a signal representing the sound related to the sound Vc. May be used as the reference signal Z. In the above configuration, it is not necessary to prepare the reference signal Z and the second content C2 separately.

(6) The second content C2 that is continuous over the entire time axis of the first content C1 is adopted, but as illustrated in FIG. 5, the second content corresponding to a part of the first content C1 on the time axis. C2 may be adopted. That is, it is assumed that there is no processing portion T2 corresponding to the similar portion P of the reference signal Z in the second content C2. In the above configuration, when the reproduction portion T1 of the first content C1 reaches the second content C2, reproduction of the second content C2 is started. Specifically, when the end point of the similar portion P has passed to the start point of the second content C2, the second content C2 is played back by the playback device 33.

Further, as illustrated in FIG. 6, the second content C2 may be composed of a plurality of parts separated from each other on the time axis. Based on the temporal correspondence between the first content C1 and the second content C2, a portion corresponding to the reproduction portion T1 of the first content C1 is reproduced among a plurality of portions constituting the second content C2.

(7) Although the cross-correlation calculated for the analysis section and the acoustic signal Y is used as the similarity index, the similarity index indicating the degree of similarity between the acoustic signal Y and the reference signal Z is not limited to the cross-correlation. That is, the method of comparing the acoustic signal Y and the reference signal Z is not limited to the calculation of the cross-correlation between the acoustic signal Y and the reference signal Z.

(8) The sound collection device 31 starts collecting the sound Vc in response to a user operation on the information processing device 30, and the sound collection by the sound collection device 31 is performed in parallel with the reproduction of the first content C1, for example. May be executed periodically.

(9) The information processing apparatus 30 may store a plurality of second contents C2 of different types. For example, the second content C2 selected by the user among the plurality of second contents C2 respectively associated with the first content C1 is reproduced. With the above configuration, it is possible to reproduce the second content C2 desired by the user.

(10) Although the configuration in which the playback device 20 and the information processing device 30 are separated is illustrated, the playback device 20 and the information processing device 30 may be configured integrally. For example, the playback device of the information processing device 30 plays back the second content C2 in conjunction with the playback of the first content C1.

(11) As described above, the information processing apparatus 30 is realized by cooperation between a computer (specifically, the control apparatus 35) and a program. The program according to the above-described form can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium The recording medium of the form may be included. Note that the non-transitory recording medium includes an arbitrary recording medium excluding a transient propagation signal (transitory, “propagating signal”) and does not exclude a volatile recording medium. It is also possible to provide a program to a computer in the form of distribution via a communication network.

<Appendix>
For example, the following configuration is grasped from the above-exemplified form.

The information processing method according to the first aspect of the present disclosure includes an acoustic signal generated by collecting sound that is emitted by playing back the first content, and a reference signal temporally corresponding to the second content. By comparing, the temporal correspondence between the first content and the second content is analyzed, and the second content is linked to the reproduction of the first content based on the analyzed temporal correspondence. Play the content on the playback device. In the above aspect, the first content is compared by comparing the acoustic signal generated by collecting the sound emitted by the reproduction of the first content with the reference signal temporally corresponding to the second content. And the second content are reproduced in conjunction with the reproduction of the first content based on the analyzed temporal correspondence. That is, it is not necessary to embed data (watermark data in the technique of Patent Document 1) for making the reproduction of the second content temporally correspond to the reproduction of the first content in the first content.

The reference signal may be a signal representing the sound emitted by the reproduction of the first content, or a signal obtained by processing the signal. In the above aspect, since the signal representing the sound emitted by the reproduction of the first content or the signal obtained by processing the signal is used as the reference signal, it is necessary to prepare the reference signal separately from the first content. There is no.

The acoustic signal and the reference signal may be compared by calculating a cross-correlation between the acoustic signal and the reference signal. In the above aspect, since the sound signal and the reference signal are compared by calculating the cross-correlation between the sound signal and the reference signal, the similarity of the waveform between the sound signal and the reference signal is taken into account. Thus, the reproduction of the second content can be linked with the reproduction of the first content with high accuracy.

In parallel with the reproduction of the first content, the second content may be reproduced by a reproduction device. In the above aspect, since the second content is reproduced in parallel with the reproduction of the first content, the first content and the first content are compared with the configuration in which the second content is reproduced subsequent to the reproduction of the first content. The user can easily grasp the temporal correspondence with the two contents.

The aspect of the present disclosure can also be realized as an information processing apparatus that executes the information processing method exemplified above, or a program that causes a computer to execute the information processing method of each aspect exemplified above.

This application is based on Japanese Patent Application No. 2018-056349 filed on Mar. 23, 2018, the contents of which are incorporated herein by reference.

DESCRIPTION OF SYMBOLS 100 ... Reproduction system, 20 ... Reproduction apparatus, 30 ... Information processing apparatus, 31 ... Sound collection apparatus, 33 ... Reproduction apparatus, 35 ... Control apparatus, 37 ... Storage apparatus, 352 ... Analysis part, 354 ... Reproduction control part.

Claims

The first content and the second content are compared by comparing the acoustic signal generated by collecting the sound emitted by the reproduction of the first content with the reference signal temporally corresponding to the second content. Analyzing the temporal correspondence with content,
An information processing method realized by a computer that causes a playback device to play back the second content in conjunction with the playback of the first content based on the analyzed temporal correspondence.
The information processing method according to claim 1, wherein the reference signal is a signal representing the sound emitted by reproducing the first content, or a signal obtained by processing the signal.
The information processing method according to claim 1 or 2, wherein the acoustic signal and the reference signal are compared by calculating a cross-correlation between the acoustic signal and the reference signal.
The information processing method according to any one of claims 1 to 3, wherein the second content is played back by a playback device in parallel with the playback of the first content.
The first content and the second content are compared by comparing the acoustic signal generated by collecting the sound emitted by the reproduction of the first content with the reference signal temporally corresponding to the second content. An analysis unit that analyzes temporal correspondence with content;
An information processing apparatus comprising: a reproduction control unit that causes the reproduction device to reproduce the second content in conjunction with reproduction of the first content based on the temporal correspondence analyzed by the analysis unit.
The information processing apparatus according to claim 5, wherein the reference signal is a signal representing the sound emitted by reproducing the first content, or a signal obtained by processing the signal.
The information processing apparatus according to claim 5, wherein the analysis unit compares the acoustic signal with the reference signal by calculating a cross-correlation between the acoustic signal and the reference signal.
The information processing apparatus according to claim 5, wherein the reproduction control unit causes the reproduction apparatus to reproduce the second content in parallel with the reproduction of the first content.