JP6411274B2 - Timing correction system, method and program thereof - Google Patents

Description

  The present invention relates to a technique for displaying text information input by a person viewing a video so as to be superimposed on the video.

  Non-Patent Document 1 is known as a prior art of a technique for displaying text information input by a person viewing a video in a superimposed manner. In Non-Patent Document 1, a viewer can post a comment while watching a moving image.

“Watching videos and posting comments”, [online], NIWANGO.INC, [Search February 2, 2015], Internet <URL: http://info.nicovideo.jp/help/player/howto/>

  However, in the conventional technology, it is necessary to enter a comment after clicking the comment post button or pressing the enter key after you want to comment on the video, so it is delayed from the timing when the viewer wants to comment Comments may be displayed. Conversely, if you know the content of the video in advance, you can enter comments in advance and the viewer can try to click the comment post button or press the enter key. , It may be earlier or later than when the viewer wants to comment. For example, when commenting the text information “8” meaning applause in time with the tempo of a music video or live video composition, there are many cases where the discrepancy is greater than when applause is actually performed, or the discrepancy is larger. Prone.

  An object of the present invention is to provide a timing correction system, a method thereof, and a program for correcting the timing when a comment is superimposed and displayed on a video in accordance with the video.

  In order to solve the above-described problem, according to one aspect of the present invention, a timing correction system is inputted by a person viewing a target video signal, is displayed superimposed on the target video signal, and means a predetermined action A key input detection unit for detecting visual information and metadata including the input time of the visual information; (i) a target video signal; (ii) a target audio signal corresponding to the target video signal; and (iii) a target video signal. A reference timing that is a timing that is a reference of an action that the visual information detected by the key input detection unit from at least one of other already input visual information and its metadata displayed superimposed on A timing detection unit for detecting, and a correction unit for correcting the timing for displaying the visual information detected by the key input detection unit on the display unit based on the reference timing.

  In order to solve the above-described problem, according to another aspect of the present invention, a timing correction method includes a key input detection unit that is input by an object that views a target video signal and is displayed superimposed on the target video signal. A key input detection step for detecting metadata including visual information meaning a predetermined action and input time of the visual information, and a timing detection unit corresponding to (i) the target video signal and (ii) the target video signal Meaning of the visual information detected in the key input detection step from the target audio signal and (iii) other input visual information already superimposed on the target video signal and / or its metadata A timing detection step for detecting a reference timing, which is a reference timing for the act to be performed, and a correction unit for displaying the visual information detected in the key input detection step on the display unit The timing, and a correction step of correcting, based on the reference timing.

  According to the present invention, it is possible to correct the timing when a comment is superimposed on a video and displayed in accordance with the video.

The functional block diagram of the timing correction system which concerns on 1st embodiment. The figure which shows the example of the processing flow of the timing correction system which concerns on 1st embodiment. The figure which shows the example of the data memorize | stored in the memory | storage part with which a key input detection part is provided. The figure which shows the correction example of a correction | amendment part. The figure which shows the example in case text information txt _in (r) is displayed from the right end of a display part at the input time t _in (r), moves toward the left end, and disappears from the left end. The figure which shows the example in case text information txt _in (q) is displayed from the right end of a display part at timing _tout (q), moves toward the left end, and disappears from the left end.

  Hereinafter, embodiments of the present invention will be described. In the drawings used for the following description, constituent parts having the same function and steps for performing the same process are denoted by the same reference numerals, and redundant description is omitted.

<Video distribution system 1 according to the first embodiment>
FIG. 1 is a functional block diagram of a timing correction system 100 according to the first embodiment, and FIG. 2 shows a processing flow thereof.

  The moving image distribution system 1 includes one or more viewer terminals 91, a timing correction system 100, and a moving image distribution server 92 that distributes the target video signal to the viewer terminals 91. Each viewer terminal 91, the timing correction system 100, and the moving image distribution server 92 are communicable via a communication line. It should be noted that the viewer terminals 91, the timing correction system 100, and the moving image distribution server 92 included in the moving image distribution system 1 are preferably synchronized in time by NTP (Network Time Protocol) or the like.

<Viewer terminal 91>
The viewer terminal 91 is operated by an object (for example, a moving image viewer) that views a target video signal (for example, a moving image), and has an input unit (keyboard, mouse, touch panel, etc.) and a display unit (display, touch panel, etc.). For example, a personal computer, a smart phone, a tablet or the like. The viewer can request the moving image distribution server 92 to play back the target video via the input unit of the viewer terminal 91. In addition, the target video signal can be viewed via the display unit of the viewer terminal 91. Furthermore, the viewer can input text information (for example, a comment) displayed superimposed on the target video signal via the input unit. When the text information is input, the viewer terminal 91 acquires the input time based on, for example, the system time, and transmits it as metadata along with the text information.

<Video distribution server 92>
The moving image distribution server 92 receives moving images from the moving image database and the video camera, and distributes the moving images stored in the moving image database or the moving images recorded by the video camera in real time in response to a request from the viewer terminal 91. In addition to moving images recorded by a video camera, CG synthesized and edited in real time or CG synthesized from motion capture may be distributed in real time. In the present embodiment, the moving image means a video signal provided together with an audio signal synchronized with the time axis. The moving image database stores text information added to the moving image together with the moving image. Further, metadata is added to the text information. The metadata includes the input time of text information, the size of text information, its color, its appearance method, its moving speed, moving position, and the like. For example, the size, color, appearance method, moving speed, moving position, and the like may be selectable by the text information input person, and the viewer terminal 91 transmits the metadata together with the text information as metadata to the video database. It is stored with the movie.

<Timing correction system 100>
Timing correction system 100, the text information txt _in (p) and its their target video signal input time t _in (p) metadata and with text information indicating the text information is entered, the text information txt _in (p) Metadata including timing t _in (p) or t _out (p) for displaying the text information on a display unit (for example, a display) is output. Note that p indicates an index assigned to all input text information. Note that the text information txt _in (p) is input by what looks at the target video signal and is displayed superimposed on the target video signal.

  The timing correction system 100 includes a key input detection unit 110, a timing detection unit 120, and a correction unit 130.

<Key input detection unit 110>
The key input detection unit 110 receives the text information txt _in (p) and the input time t _in (p) of the text information, and receives the text information txt _in (q) indicating the predetermined action and the input time of the text information. Metadata including t _in (q) is detected (S110) and output. In addition, q shows the index provided to all the text information meaning a predetermined action. If the number of all text information to be input is P, and the number of text information meaning a predetermined action among the input text information is Q, then P ≧ Q, and p = 1, 2,. P, q = 1, 2,..., Q.

For example, the key input detection unit 110 includes a storage unit (not shown), and a predetermined action and text information indicating the predetermined action are stored in association with each other (see FIG. 3). The key input detection unit 110 determines whether or not the received text information txt _in (p) matches the text information stored in the storage unit. If they match, the corresponding text information txt _in (q ), A predetermined action (or an index indicating the predetermined action), and an input time t _in (q) are output to the correction unit 130. On the other hand, if they do not match, the corresponding text information txt _in (r) and input time t _in (r) are output to the moving image distribution server 92. Note that r represents an index assigned to all text information other than text information meaning a predetermined action. If the number of texts other than text information meaning a predetermined action among the input text information is R, P = Q + R and r = 1, 2,...

<Timing detection unit 120>
The timing detection unit 120 is already input from the video distribution server 92 (i) the target video signal, (ii) the target audio signal corresponding to the target video signal, and (iii) the superimposed video signal displayed on the target video signal. Receive at least one of other text information and its metadata. For example, a moving image with text information includes all of (i) to (iii).

The timing detection unit 120 detects a reference timing t _out (s) that is a timing that is a reference of an action that the text information detected by the key input detection unit 110 detects from at least one of (i) to (iii). (S120) and output. Note that s indicates an index representing an appearance number for a predetermined action included in at least one of (i) to (iii), s = 1, 2,..., S, and S is (i) to (iii) ) Is the number of appearances of the predetermined action included in at least one of the above.

As a method for detecting the reference timing t _out (s), various methods are conceivable depending on the action that the text information means.

For example, a method of detecting the reference timing when the action that the text information means is “applause” is illustrated according to the data to be used. When the action that the text information means is “applause”, the action that the text information means is “applause” that is performed at a constant tempo and “applause” that is performed without a constant tempo. "You could think so. Hereinafter, simply “applause” means “applause” performed without a certain tempo. Since “hand clapping” and “clapping” are different in time interval and volume difference between claps (Reference 1), for example, (ii) text information based on the target audio signal corresponding to the target video signal, etc. It is possible to determine which action means.
(Reference 1) Yu Kamamoto, Kazuhiko Kawahara, Akira Omoto, Takehiro Moriya, “Fundamental study for low-delay transmission of applause and clapping sounds excited during music appreciation”, Acoustical Society of Japan 2014 Autumn Study Presentation, 1-Q-17, 2014.
In this embodiment, a method for detecting the reference timing when the action that the text information means is “hand clapping” will be described.

(i) When the action is “applause” and the data is the target video signal If the target video signal has a certain tempo, it is assumed that the applause is in accordance with the tempo and the tempo is detected as the reference timing ( (See Reference 2).
(Reference 2) Amami Mikami, Tomomi Matsumoto, Koji Kadota, Harumi Kawamura, Akira Kojima, "Delayed Synchronous Video Feedback System for Motion Learning", Information Processing Society of Japan, IPSJ Transactions Consumer Devices & Systems, 2014 Year, vol.4, No.1, pp22-31.
(ii) When the action is “hand applause” and the data is the target acoustic signal If the target acoustic signal has a certain tempo, it is assumed that the applause matches the tempo and the tempo is detected as the reference timing ( References 2 to 5).
(Reference 3) Mio Tsunoo, Kenichi Miyamoto, Junki Ono, Shigeki Hiyama, “Extraction of Rhythm Features from Musical Sound Signals Using Harmonic / Percussive Sound Separation Techniques”, Spring Meeting of Acoustical Society of Japan Proceedings of the conference, Mar. 2008, pp.905-906.
(Reference 4) Mio Tsunoo, Junki Ono, Shigeki Hiyama, “Rhythm Map: Extraction of Unit Rhythm Patterns from Musical Acoustic Signals and Analysis of Musical Structure”, Information Processing Society of Japan, Aug.2008, vol .2008-MUS-76, no.25, pp.149-154.
(Reference 5) Mio Tsunoo, Junki Ono, Shigeki Hiyama, “Nodal Boundary Estimation of Musical Acoustic Signals Based on Extraction of Unit Rhythm Patterns Considering Harmonic Boundaries”, Proc. , Sep. 2009, no. 3-5-10, pp. 897-898.
(iii) In the case where the action is “applause” and the data is displayed in a superimposed manner on the target video signal and the other text information and its metadata already input. The other text information that means the same action as the action that the text information detected by the key input detection unit 110 means is extracted from the text information. For example, with reference to a storage unit (not shown) (see FIG. 3), the timing detection unit 120 receives the received other text information and the text information stored in the storage unit and detected by the key input detection unit 110. It is determined whether or not the text information corresponding to the act meaning "?" Matches, and if it matches, the display time is extracted.

  The timing detection unit 120 obtains a reference timing based on the extracted statistics of the display time. For example, using the extracted display time, a predetermined time interval (for example, if the video is a music video and the tempo of the song is 148 BPM (Beats per Minutes), the interval of one beat is about 405 ms. Every 405 ms), a representative value for each time interval (any value representing a plurality of display times such as an average value, a mode value, a minimum value, and a maximum value) is obtained and detected as a reference timing. For example, a histogram is created using the extracted display time, and the reference timing is obtained by majority vote. That is, the mode value is set as the reference timing.

(How to give timing fluctuation)
You may give fluctuation to the timing of applause (refer to reference 1).

  For example, in the case of (iii), using the extracted display time, the average value and variance of the display time are obtained for each predetermined time interval, and a random number according to a Gaussian distribution having the average value and variance is used as the reference timing. Also good. By this method, fluctuations in the timing of applause can be provided, and text information corresponding to a predetermined action can be displayed at a more natural timing.

  The variance may be obtained by a similar method and combined with the methods (i) and (ii). For example, the reference timing is obtained by the methods (i) and (ii), and a random number according to a Gaussian distribution having the obtained variance around the reference timing is set as a new (finally used) reference timing. By such a method, text information corresponding to a predetermined action can be displayed at a more natural timing.

  Alternatively, a dispersion value generally generated when clapping is obtained in advance, and the reference timing may be obtained based on the dispersion. For example, a reference timing is obtained by the methods (i), (ii), and (iii), and a random number according to a Gaussian distribution having a variance that generally occurs when clapping is performed around the reference timing is newly (final). Used as a reference timing).

(If the action is other than “hand clapping”)
In the case of other text information and its metadata already input (iii) displayed superimposed on the target video signal, it is easily applied even if the action is an action other than “clapping” be able to. For example, when the action is “applause” or “laughter”, a predetermined time interval is set to the maximum time that the series of actions “applause” or “laughter” can continue. For example, if it is assumed that it will be about 30 seconds at the most to send "applause" for some event, within 1 minute after other text information that means "applause" is first displayed A representative value is obtained from other text information indicating “applause” displayed on the screen and detected as a reference timing. The timing of “applause” and “laughter” may be given fluctuation by the above-described method.

  However, when distributing the target video signal for the first time (when text information is not added to the target video signal), or when distributing a video recorded by a video camera in real time, Since there is no text information, (iii) cannot be used.

As a method for detecting the reference timing of actions other than “hand clapping” from (i) the target video signal or (ii) target sound signal, for example, a predetermined action (for example, “applause” or “laughter”) is taken in advance. A feature amount is extracted from the video signal or the audio signal and stored in a storage unit (not shown). Further, the feature amount is extracted from (i) the target video signal or (ii) the target acoustic signal received from the video distribution server 92, the similarity with the feature amount stored in the storage unit is obtained, and the similarity is a threshold value In the case described above, it is determined that a predetermined action is being performed, a representative value is obtained, and detected as a reference timing. For example, by applying an existing face authentication technology, (i) a smile may be detected from the target video signal, and the reference timing of “laughter” may be obtained from the detected time. Further, for example, a motion compensation vector (see Reference Document 6) that is a feature amount for moving image compression may be used.
(Reference 6) Atsumi Murakami, Kotaro Asai, Shunichi Sekiguchi, “High-efficiency video coding technology HEVC / H.265 and its application”, Ohm, 2013, p.20-28,125-132
A video signal obtained by capturing “applause” in advance is compression-encoded to obtain a time series of motion compensation vectors, which are stored in a storage unit (not shown). Furthermore, (i) the target video signal received from the video distribution server 92 is compression-encoded to obtain a time series of motion compensation vectors (in the first place, if the target video signal is compression-encoded, motion compensation is performed. Since a vector is included, it may be used as it is), and the degree of similarity with the time series of motion compensation vectors stored in the storage unit may be obtained.

<Correction unit 130>
The correction unit 130 receives text information txt _in (q) meaning a predetermined action and metadata including an input time t _in (q) of the text information and a reference timing t _out (s), and detects key input. The timing for displaying the text information detected by the unit 110 on the display unit is corrected based on the reference timing t _out (s) (S130), and the text information txt _in (q) and the text information are displayed. The metadata including the timing t _out (q) is output to the moving image distribution server 92. For example, t _out (s) that is closest to t _in (q) among S t _out (s) is set as timing t _out (q) for displaying text information txt _in (q) (t _out (q) ← t _out (s)). For example, FIG. 4 shows an example in which the text information txt _in (q) is “8”.

The moving image distribution server 92 superimposes the text information on the target video signal and outputs it. The video distribution server 92 displays the text information txt _in (q) or txt _in (r) on the display unit of the viewer terminal 91 at the timing t _out (q) or the input time t _in (r), respectively. And delivered together with the target video signal. For example, if text information txt _in (p) is displayed from the right edge of the display at timing t _out (q) or input time t _in (r), moves toward the left edge, and disappears from the left edge, it is corrected. FIG. 5 shows an example of the case where it is not corrected (displayed at the input time t _in (r)), and FIG. 6 shows an example of the case where it is corrected (displayed at the timing t _out (q)).

<When to apply timing t _out (q)>
Note that the video distribution server 92 can consider two timings for distributing the text information txt _in (q) together with the target video signal so that the text information txt _in (q) is displayed on the display unit of the viewer terminal 91 at timing t _out (q). .

(1) when the viewer inputs the text information txt _in (q), the at the time of reproduction is displayed on the display unit of the viewer terminal 91 by text information txt _in (q) enter the time t _in (q) As described above, it is distributed together with the target video signal. Then, the video distribution server 92 stores the text information txt _in (q) and the timing t _out (q) (not the input time t _in (q)) together with the target video signal in the video database. Is reproduced together with the target video signal so that the text information txt _in (q) is displayed on the display unit of the viewer terminal 91 at the timing t _out (q).

(2) when the viewer inputs the text information txt _in (q), at the time of the reproduction, so as to be displayed on the display unit of the viewer terminal 91 text information txt _in (q) at the timing t _out (q) To the target video signal. In this case, the delivery method differs depending on the magnitude relationship between the timing t _out (q) and the input time t _in (q).

(2-A) When the timing t _out (q) is later than the input time t _in (q), the text information txt _in (q) is displayed on the display unit at the timing t _out (q). The display timing may be delayed and distributed together with the target video signal.

(2-B) When the timing t _out (q) is earlier than the input time t _in (q), distribution is performed by the following method.

(2-B-1) For example, as shown in FIGS. 5 and 6, when the display method of the text information txt _in (q) represents the passage of time, the passage of time is used to obtain the text information txt _in Distribute together with the target video signal as if (q) was displayed at timing t _out (q). For example, in the case of FIG. 6, the timing t _out (q) is earlier than the input time t _in (q), so when t = t _out (q) (the top diagram in FIG. 6), the video distribution server 92, because it does not receive the text information txt _in (q), so as to display the text information txt _in (q), it can not be delivered. If the text information txt _in (q) is received (input time t = t _in (q)) and is displayed on the display unit of the viewer terminal 91 at the timing t _out (q), t = t to display the text information txt _in (q) to position the text information txt _in (q) is displayed in the _in (q). In other words, without displaying the movement (transition) of the text information txt _in (q) from the top diagram in FIG. 6 to the middle diagram in FIG. 6, the text is suddenly displayed at t = t _in (q). Information txt _in (q) is displayed as shown in the middle diagram of FIG. 6, and the text information txt _in (q) is moved to the bottom diagram of FIG. In this case, the moving image distribution server 92 in the moving image database may be stored text information txt _in the (q) and a timing t _out (q) with target video signal, when the next playback, text information txt _in (q) is distributed together with the target video signal so as to be displayed on the display unit of the viewer terminal 91 at timing t _out (q).

(2-B-2) When the predetermined action indicated by the text information is a periodic action such as clapping, the acquired text information txt _in (q) is displayed in the next period. That is, the text information txt _in (q) is distributed together with the target video signal so as to be displayed on the display unit of the viewer terminal 91 at timing t _out (q + 1). In this case, the video distribution server 92 may store the text information txt _in (q) and the timing t _out (q) together with the target video signal in the video database, or the text information txt _in (q). And timing t _out (q + 1) may be stored. At the next playback time, the text information txt _in (q) is distributed together with the target video signal so as to be displayed on the display unit of the viewer terminal 91 at the timing t _out (q) or the timing t _out (q + 1).

<Effect>
With such a configuration, it is possible to correct the timing when the text information is displayed superimposed on the video according to the video. For example, when matching the music tempo, the sense of unity of the viewer can be improved by matching the display timing of the text information.

<Modification>
In the present embodiment, the timing correction system 100 is configured as a separate device from the viewer terminal 91 or the video distribution server 92, but may be configured to be incorporated in the viewer terminal 91 or the video distribution server 92. Further, the timing correction system 100 includes the key input detection unit 110, the timing detection unit 120, and the correction unit 130, and has been described as an independent device. However, each unit is incorporated in the viewer terminal 91 or the moving image distribution server 92. It is good also as a structure. For example, the key input detection unit 110 may be incorporated in the viewer terminal 91 and the timing detection unit 120 and the correction unit 130 may be incorporated in the moving image distribution server 92.

  The moving image distribution server 92 may be a server that distributes only moving images without adding text information. In that case, the timing correction system 100 may include a text information database and add text information.

In the present embodiment, an example of text information is shown as information input by a viewer and displayed superimposed on a target video signal, but other visual information may be used. Here, the “visual information” is information visually recognizable via the display unit, and is, for example, a character, a figure, a symbol, a combination thereof, or a combination of these and a color. Further, the image is not limited to a still image, and may be a moving image. For example, (1) text information (eg, “w”, “8”, etc.) indicating a predetermined action such as “laughter” or “applause”, (2) “laughter” other than text information, as in this embodiment ”Or“ applause ”means a predetermined action such as bit information on a computer for identification, (3) emoticons, pictograms, etc. that are not normal text information. For example, common pictograms between mobile phones of different carriers (see Reference 7), (4) ASCII art, etc., which are generally like pictures using text information and text information layout information (Reference 8) (5) Net slang corresponding to the above (1) to (4). For example, there is a net slang such as “I grew up” for text information “www...
(Reference 7) “docomo / au common pictograms”, NTT DOCOMO, Inc., [online], [Search February 9, 2015], Internet <URL: https://www.nttdocomo.co.jp/service / developer / smart_phone / make_contents / pictograph />
(Reference 8) "ASCII art", [online], February 2, 2015, Wikipedia, [Search February 9, 2015], Internet <URL: http://en.wikipedia.org/wiki/% E3% 82% A2% E3% 82% B9% E3% 82% AD% E3% 83% BC% E3% 82% A2% E3% 83% BC% E3% 83% 88>

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by the electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

Claims (11)

A storage unit that stores a predetermined action and visual information that means the predetermined action in association with each other;
Entered by what you see an object video signal, the visual information displayed superimposed on the subject image signal, in the case where it is determined whether it matches the visual information stored in the storage unit, matches, Visual information displayed superimposed on the target video signal, predetermined action corresponding to the visual information displayed superimposed on the target video signal , and input of visual information displayed superimposed on the target video signal A key input detection unit for detecting metadata including time;
(i) the target video signal, (ii) a target audio signal corresponding to the target video signal, and (iii) other already input visual information displayed superimposed on the target video signal and the already A timing detection unit that detects a reference timing that is a timing that is a reference of an action that the visual information detected by the key input detection unit from at least one of metadata of other visual information that has been input;
The timing for displaying the visual information detected by said key input detection unit on the display unit, seen including a correction portion that corrects, based on the reference timing,
Visual information indicating a predetermined action stored in the storage unit is superimposed on the target video signal at the reference timing corrected by the correction unit.
Timing correction system.   The timing correction system of claim 1,
  The timing detector
  (1) (i) the tempo when the target video signal has a certain tempo,
  (2) (ii) the tempo when the target acoustic signal has a certain tempo,
  (3) (i) When the similarity between the feature amount extracted from the target video signal and the feature amount extracted from the video signal obtained by photographing a predetermined action in advance is equal to or greater than a threshold value, for each predetermined time interval Typical value,
  (4) (ii) A predetermined value when the similarity between the feature value extracted from the target sound signal and the feature value extracted from the sound signal corresponding to the video signal obtained by capturing a predetermined action is equal to or greater than a threshold value Representative value for each time interval,
  Detecting at least one of the reference timing,
  Timing correction system. The timing correction system of claim 1,
The timing detection unit is based on at least a plurality of (iii) other already input visual information that is displayed superimposed on the target video signal and statistics of metadata of the other already input visual information. a reference timing is intended Mel asked Te,
Extract the display time of each of the already input visual information that means an action that matches the predetermined action included in the metadata detected by the key input detection unit from the other already input visual information. The mode value for each predetermined time interval of the extracted display time is detected as the reference timing for each predetermined time interval,
Timing correction system. The timing correction system of claim 1,
The timing detection unit is based on at least a plurality of (iii) other already input visual information that is displayed superimposed on the target video signal and statistics of metadata of the other already input visual information. Te is shall request the reference timing,
Extract the display time of each of the already input visual information that means an action that matches the predetermined action included in the metadata detected by the key input detection unit from the other already input visual information. Using the extracted display time, for each predetermined time interval, the average value and variance of the display time are obtained, and a random number according to a Gaussian distribution having the obtained average value and variance is used as a reference timing.
Timing correction system.   The timing correction system according to any one of claims 1 to 3,
  The timing detection unit obtains in advance a value of variance that occurs for each type of predetermined action, and the metadata detected by the key input detection unit out of the obtained variance with the reference timing as a center. A random number according to a Gaussian distribution corresponding to a predetermined action included is used as a new reference timing.
  Timing correction system. In the storage unit, a predetermined action and visual information indicating the predetermined action are stored in association with each other,
Key input detection unit is input by what you see an object video signal, the visual information displayed superimposed on the subject image signal, determines whether or not to match the visual information stored in the storage unit, If they match, the visual information displayed superimposed on the target video signal, a predetermined action corresponding to the visual information displayed superimposed on the target video signal, and displayed superimposed on the target video signal A key input detection step for detecting metadata including input time of visual information to be displayed;
The timing detection unit includes (i) the target video signal, (ii) a target audio signal corresponding to the target video signal, and (iii) another already input that is displayed superimposed on the target video signal. Timing detection for detecting a reference timing, which is a reference timing of an action meant by the visual information detected in the key input detection step, from at least one of visual information and metadata of other visual information that has already been input. Steps,
Correction section, viewed contains a correction step of the timing for displaying the visual information detected by said key input detection step on the display unit, to correct based on the reference timing,
Visual information indicating a predetermined action stored in the storage unit is superimposed on the target video signal at the reference timing corrected in the correction step.
Timing correction method.   The timing correction method according to claim 6, comprising:
  In the timing detection step,
  (1) (i) the tempo when the target video signal has a certain tempo,
  (2) (ii) the tempo when the target acoustic signal has a certain tempo,
  (3) (i) When the similarity between the feature amount extracted from the target video signal and the feature amount extracted from the video signal obtained by photographing a predetermined action in advance is equal to or greater than a threshold value, for each predetermined time interval Typical value,
  (4) (ii) A predetermined value when the similarity between the feature value extracted from the target sound signal and the feature value extracted from the sound signal corresponding to the video signal obtained by capturing a predetermined action is equal to or greater than a threshold value Representative value for each time interval,
  Detecting at least one of the reference timing,
  Timing correction method. The timing correction method according to claim 6 , comprising:
The timing detection step is based on at least a plurality of (iii) other already input visual information displayed superimposed on the target video signal and statistics of metadata of the already input other visual information. a reference timing is intended Mel asked Te,
The display time of each of the already input other visual information that means an action that matches the predetermined action included in the metadata detected in the key input detection step is extracted from the other already input visual information. The mode value for each predetermined time interval of the extracted display time is detected as the reference timing for each predetermined time interval,
Timing correction method. The timing correction method according to claim 6 , comprising:
The timing detection step is based on at least a plurality of (iii) other already input visual information displayed superimposed on the target video signal and statistics of metadata of the already input other visual information. Te is shall request the reference timing,
Extract the display time of each of the already input visual information that means an action that matches the predetermined action included in the metadata detected by the key input detection unit from the other already input visual information. Using the extracted display time, for each predetermined time interval, the average value and variance of the display time are obtained, and a random number according to a Gaussian distribution having the obtained average value and variance is used as a reference timing.
Timing correction method.   A timing correction method according to any one of claims 6 to 8,
  In the timing detection step, a variance value generated for each type of the predetermined action is obtained in advance, and the metadata detected in the key input detection step in the obtained variance is centered on the reference timing. A random number according to a Gaussian distribution corresponding to a predetermined action included is used as a new reference timing.
  Timing correction method. The program for functioning a computer as a timing correction system in any one of Claims 1-5 .

Images