JP2006121388A - Output timing control apparatus, video image output unit, output timing control system, output unit, integrated data providing apparatus, output timing control program, output device control program, integrated data providing apparatus control program, method of controlling the output timing control apparatus, method of controlling the output unit and method of controlling the integrated data providing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an output timing controlling apparatus, a video image output unit, an output timing control system, an output unit, an integrated data providing apparatus, an output timing control program, an output device control program, an integrated data providing apparatus control program, a method of controlling the output timing control apparatus, a method of controlling the output timing control apparatus, a method of controlling the output device, and a method of controlling the integrated data providing apparatus, suitable for making arrival timing of multiple kinds of output elements, at a predetermined position for a televiewer. <P>SOLUTION: A video image display system 1 includes an integrated data transmission unit 2 for transmitting integrated data to a video image reproducing device 3 and a voice-reproducing unit 4; and the video image reproducing device 3 which reproduces the video image based on the integrated data, performs optical projection outputting of a regenerated video image in a video image projecting part 33, and synchronizes the voice in a televiewer seat and timing in the acquisition of the video image, based on the integrated data and the voice information, in which the reproducing output is performed with the voice reproducing unit 4 acquired in the voice information acquisition part 32 which arranges to the televiewer seat. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an apparatus for controlling the output timing of a plurality of output elements output from a plurality of output apparatuses, and in particular, an output timing control apparatus suitable for synchronizing arrival timings of a plurality of output elements at a predetermined position, Video output device, output timing control system, output device, integrated data providing device, output timing control program, output device control program, integrated data providing device control program, output timing control device control method, output device control method, and integrated data providing device It relates to a control method.

  2. Description of the Related Art Conventionally, there is a system in which audio is output from a speaker device different from a projector in accordance with video output from a projector or the like, and a viewer views it from a viewer's seat or the like. However, since the sound output from the speaker device installed beside the screen with respect to the video (light) displayed on the screen has a significant difference in propagation speed between them, there is a viewer especially from the speaker device. When the position is far away, there is a possibility that the video and audio are not synchronized at the position and the timing of both is shifted.

Conventionally, as a technique for preventing such synchronization deviation, there are a digital AV system described in Patent Document 1, a time difference adjusting device described in Patent Document 2, a video / audio synchronization method described in Patent Document 3, and the like.
In the digital AV system described in Patent Document 1, a device that reproduces video and a device that reproduces audio cooperate with each other. Each device corrects the delay based on the time information.

Also, the time difference adjustment apparatus of Patent Document 2 uses a reference signal to synchronize video and audio, and adds the reference signal to audio and video signals.
Also, the video / audio synchronization method of Patent Document 3 is a method of measuring time lag by detecting video and audio of a predetermined gesture.
JP 2001-128167 A JP 7-38771 A JP 11-88847 A

  However, since the digital AV system of Patent Document 1 does not measure and synchronize information (including time) between video and audio at the position where the viewer is actually located, It is not possible to solve the synchronization error occurring at the position. Furthermore, there is a problem that a function for synchronizing each device has to be added, leading to an increase in cost.

In addition, in the time difference adjusting apparatus of Patent Document 2, since a reference signal for synchronization is added to an audio or video signal, a process for separating them or a process for comparing them is necessary, and synchronization is achieved in real time. However, there is a risk that the processing becomes heavy.
Further, the video / audio synchronization method of Patent Document 3 has a problem that processing for recognizing a gesture becomes heavy.

  Therefore, the present invention has been made paying attention to such an unsolved problem of the conventional technique, and is an output timing control device suitable for synchronizing arrival timings of a plurality of output elements at predetermined positions. , Video output device, output timing control system, output device, integrated data providing device, output timing control program, output device control program, integrated data providing device control program, output timing control device control method, output device control method, and integrated data provision An object is to provide a device control method.

[Invention 1] An output result obtained by integrating N (N is an integer of 2 or more) output elements is obtained separately from M (1 <M ≦ N) output devices. In the system configured by outputting, an output timing control device for controlling the output timing of the output element from the output device,
Synchronization information storage means for storing information related to synchronization timing when integrating the N output elements;
Output information acquisition means for acquiring information on output elements output from the M output devices at a predetermined position;
Synchronization deviation calculating means for calculating respective deviations of the arrival timings of the N output elements at the predetermined position with respect to the synchronization timing based on the acquisition result of the output information acquisition means and the stored contents of the synchronization information storage means; ,
Output timing control means for controlling the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result of the synchronization deviation calculation means It is characterized by providing these.

  If it is such a structure, the information which concerns on the output element each output from M output devices will be acquired in a predetermined position by an output information acquisition means. When the information related to the output element is acquired, the synchronization deviation calculation means synchronizes the arrival timing of the N output elements at a predetermined position based on the acquisition result of the output information acquisition means and the stored contents of the synchronization information storage means. Each deviation with respect to is calculated. When the deviation is calculated, the output timing control means determines the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result. Be controlled.

  Therefore, since the information related to the output element output from the output device is acquired at a predetermined position, the deviation of the arrival timing at the predetermined position of the output element from the synchronization timing is calculated based on the acquired information. The deviation of the arrival timing of the output element at the predetermined position from the synchronization timing can be calculated, and the output timing of the output element from the output device is controlled based on the deviation thus calculated. Therefore, the effect that the arrival timings of the N output elements output from the output device can be synchronized at a predetermined position can be obtained.

  Here, for example, in the case of a work such as a movie, the output element is an output element that includes a video and an audio (an actor's dialogue, BGM, etc.). In addition, a plurality of types of video are displayed synchronously. In such a work, each video is an output element, and in a work where multiple types of sounds (violin, piano, flute, etc.) are played synchronously, each sound is an output element and is synchronized with the video by fogging with dry ice etc. In a work that generates odor, the video and fog are output elements, and in a work that generates an odor such as a floral scent in synchronization with the video, the video and odor are output elements. In other words, when a single work is configured by integrating elements such as video and audio output from a plurality of output devices in a movie or a show, each element constituting the work is an output element. Become. The output element propagates (moves) in the air or the like with the passage of time when output from the output device, such as video (light) or audio, and various propagation ( Moving) speed. Further, it also includes an image propagating through a relay means such as a screen, such as an image output from a light projection type output device such as a projector (there is another reflecting mirror or the like). Hereinafter, the same applies to the output timing control program of the invention 18 and the output timing control device control method of the invention 28.

  Further, the information related to the synchronization timing when integrating the output elements is at least information indicating a temporal relationship between the output elements output from each output device. It includes information that the output timing of each output element can be understood. Hereinafter, the same applies to the output timing control program of the invention 18 and the output timing control device control method of the invention 28.

  The predetermined position is mainly a position away from the output position of the output element, such as a theater seat in a movie theater, and indicates a position where a viewer who views the result of integrating the plurality of output elements is present. Hereinafter, the same applies to the output timing control program of the invention 18 and the output timing control device control method of the invention 28.

[Invention 2] Furthermore, the output timing control device of Invention 2 is the output timing control device of Invention 1,
The information relating to the output element includes information relating to the arrival timing of the output element to the predetermined position,
The information related to the synchronization timing includes information related to a part of the data and information related to the output timing of the data when the output element is data.

  With such a configuration, a part of the data included in the output element can be used as a detection marker for knowing the synchronization timing, so that special information is added to the output element. Therefore, there is an effect that the deviation of the arrival timing with respect to the synchronization timing can be calculated.

[Invention 3] Further, the output timing control device of Invention 3 is the output timing control device of Invention 2,
Information related to a part of the data is a feature amount extracted from the part of the data,
The synchronization deviation calculation unit extracts a feature amount from the acquisition result based on the acquisition result of the output information acquisition unit, and stores the feature amount, information related to the arrival timing, and storage content of the synchronization information storage unit. Based on the above, a deviation from the synchronization timing is calculated.
With such a configuration, it is possible to reduce the processing when detecting a part of the data from the acquired output element.

[Invention 4] Furthermore, the output timing control device of Invention 4 is the output timing control device of Invention 1,
At least one of the M output devices is configured to output a periodic signal having a periodicity together with the output element in synchronization with the output timing of the output element output from the own device,
The output information acquisition means acquires the periodic signal as information relating to the output element.

  With such a configuration, for example, it is possible to continuously flow audio outside the audible band, which is not reflected in the output element integration result, while shifting it in a predetermined cycle, and to detect a phase shift of frequency modulation as a synchronization shift. As a result, there is no need for a marker for detecting the synchronization timing, and the effect that the deviation from the synchronization timing can be easily calculated is obtained.

[Invention 5] Furthermore, the output timing control device of Invention 5 is the output timing control device of any one of Inventions 1 to 4,
When the M output devices are configured to include a video output device that outputs video as the output element and an audio output device that outputs audio as the output element,
The output information acquisition means acquires information on the video and audio output from the video output device and the audio output device, respectively, at the predetermined position,
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the storage content of the synchronization information storage means, and the arrival timing of the video output from the video output device at the predetermined position, and the audio output device The difference between the arrival timing of the sound output from the above and the synchronization timing is calculated.

With such a configuration, the video output device and the audio output so that the difference in arrival timing at a predetermined position between the video output from the video output device and the audio output from the audio output device matches the synchronization timing. The effect that the output timing of the apparatus can be controlled is obtained.
Here, the sound output device may output not only sound but also other sounds such as sound effects and BGM, and the synchronization timing of these sounds and the video may be synchronized. Hereinafter, the same applies to the output timing control program of the invention 22 and the output timing control device control method of the invention 32.

[Invention 6] Furthermore, the output timing control device of Invention 6 is the output timing control device of any one of Inventions 1 to 5,
When the M output devices are configured to include two or more video output devices that output video as the output elements,
The output information acquisition means acquires information relating to the video respectively output from the two or more video output devices at the predetermined position,
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the storage content of the synchronization information storage means, and the arrival timing of the video output from the two or more video output devices at the predetermined position, A deviation from the synchronization timing is calculated.
With such a configuration, the output timings of the plurality of video output devices are controlled so that the deviation in arrival timing at a predetermined position of the plurality of videos output separately from the plurality of video output devices matches the synchronization timing. The effect that it can do is acquired.

[Invention 7] The output timing control device according to Invention 7 is the output timing control device according to any one of Inventions 1 to 4, wherein the M output devices output two or more devices that output sound as the output element. When configured to include an audio output device,
The output information acquisition means acquires information on the sound output from each of the two or more sound output devices at the predetermined position,
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the storage content of the synchronization information storage means, and the arrival timing of the voices output from the two or more voice output devices at the predetermined position, A deviation from the synchronization timing is calculated.
With such a configuration, the output timing of the plurality of audio output devices is controlled so that the arrival timing deviation at a predetermined position of the plurality of audios output separately from the plurality of audio output devices matches the synchronization timing. The effect that it can do is acquired.

[Invention 8] Further, the output timing control device of Invention 8 is the output timing control device of any one of Inventions 1 to 7,
When the plurality of output devices are configured to include an odor output device that outputs odor as the output element,
The output information acquisition means is configured to output information related to an odor output from the odor output device and information related to an output element output from an output device different from at least one odor output device. Acquired in
The synchronization shift calculation means is based on the acquisition result of the output information acquisition means and the stored contents of the synchronization information storage means, the arrival timing of the odor output from the odor output device at the predetermined position, the odor output device, Is characterized by calculating a deviation of the arrival timing of output elements output from different output devices with respect to the synchronization timing.
With such a configuration, the video output device and the video output device so that the difference in arrival timing at a predetermined position between the odor output from the odor output device and the output element output from another output device matches the synchronization timing. The effect that the output timing of the audio output device can be controlled is obtained.

[Invention 9] Furthermore, the output timing control device of Invention 9 is the output timing control device of any one of Inventions 1 to 8,
Integrated data acquisition means for acquiring integrated data comprising information relating to the N output elements;
Output means capable of outputting x (1 ≦ x <N) output elements out of the N output elements based on the integrated data acquired by the integrated data acquiring means.

With such a configuration, the integrated data including information related to the N output elements is acquired by the integrated data acquisition unit. When the integrated data is acquired, the output means outputs x (1 ≦ x <N) output elements out of the N output elements based on the integrated data.
Therefore, for example, in the case of a work composed of video and audio, it is possible to configure an output timing control device that also has a function as an output device that outputs video or audio.

[Invention 10] On the other hand, the video output device of Invention 10 is
An output timing control device of the invention 5 or 6;
Integrated data acquisition means for acquiring integrated data including information relating to video as the output element;
Video output means capable of outputting the video based on the information related to the video included in the integrated data acquired by the integrated data acquisition means.

If it is such a structure, the integrated data which contains the information which concerns on an image | video will be acquired by the integrated data acquisition means. When the integrated data is acquired, the video is output by the video output unit based on the information related to the video included in the integrated data.
Therefore, the output timing control device can be configured as, for example, a projection display device such as a projector having a light projection type video output means, or a display device such as a liquid crystal display or a plasma display.

[Invention 11] On the other hand, the output timing control system of Invention 11 is
An output timing control device according to any one of claims 1 to 8, and the M output devices.
The output timing control device and the M output devices are connected so that data communication is possible,
The output timing control device includes:
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device;
Output timing correcting means for correcting the output timing of the output element by the output means based on the control information received by the control information receiving means.

With such a configuration, in the output timing control device, the control information used for controlling the output timing of the output device is transmitted to the output device by the output timing control means.
In the output device, the output means outputs x (1 ≦ x <N) output elements out of N output elements, and the control information receiving means transmits the control transmitted from the output timing control device. Information is received. When the control information is received, the output timing correction means corrects the output timing of the output element by the output means.

  Therefore, the output timing control device acquires information on the output element output from the output device at a predetermined position, and calculates the deviation of the arrival timing of the output element at the predetermined position from the synchronization timing based on the acquired information. As a result, the deviation of the arrival timing of the output element at the predetermined position from the synchronization timing can be calculated. Furthermore, based on the deviation calculated in this way, control information used to control the output timing of the output element from the output device is generated and transmitted to the output device. Since the output timing can be controlled, it is possible to synchronize the arrival timings of the output elements output from the plurality of output devices at a predetermined position.

[Invention 12] The output timing control system of Invention 12 is
An output timing control device according to any one of the inventions 1 to 8, the M output devices, and an integrated data providing device;
The output timing control device, the M output devices, and the integrated data providing device are connected so that data communication is possible,
The output timing control device includes:
Synchronization information receiving means for receiving information related to the synchronization timing transmitted from the integrated data providing device,
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Integrated data receiving means for receiving integrated data transmitted from the integrated data providing device;
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements based on the integrated data received by the integrated data receiving means;
Control information receiving means for receiving control information transmitted from the output timing control device;
Output timing correction means for correcting the output timing of the output element based on the control information received by the control information receiving means,
The integrated data providing device includes:
Integrated data transmission means for transmitting integrated data comprising information on the N output elements to the output device;
Synchronization information transmitting means for transmitting information related to the synchronization timing to the output timing control device.

With such a configuration, in the output timing control device, the synchronization information receiving unit receives information related to the synchronization timing transmitted from the integrated data providing device, and the output timing control unit sends the information to the output device. The control information used for controlling the output timing of is transmitted.
In the output device, the integrated data receiving unit receives the integrated data transmitted from the integrated data providing device. When the integrated data is received, the output unit outputs x (1 ≦ x <N) output elements out of the N output elements based on the integrated data.

In the output device, the control information receiving means receives the control information transmitted from the output timing control device. When the control information is received, the output timing of the output element is corrected by the output timing correction means based on the control information.
Therefore, the output timing control device can acquire the synchronization information from the integrated data providing device, and the output device can acquire the integrated data from the integrated data providing device. Further, the output timing control device acquires information on the output element output from the output device at a predetermined position, and calculates a deviation of the arrival timing of the output element at the predetermined position from the synchronization timing based on the acquired information. As a result, the deviation of the arrival timing of the output element at the predetermined position from the synchronization timing can be calculated. Furthermore, based on the deviation calculated in this way, control information used to control the output timing of the output element from the output device is generated and transmitted to the output device. Since the output timing can be controlled, it is possible to synchronize the arrival timings of the output elements output from the plurality of output devices at a predetermined position.

[Invention 13] The output timing control system of Invention 13 is
An output timing control device of the invention 9 and S (S is an integer of 1 or more) output devices;
The output timing control device and the S output devices are connected to be capable of data communication,
The output timing control device includes:
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device;
Output timing correcting means for correcting the output timing of the output element by the output means based on the control information received by the control information receiving means.

With such a configuration, in the output timing control device, the control information used for controlling the output timing of the output device is transmitted to the output device by the output timing control means.
In the output device, the output means outputs x (1 ≦ x <N) output elements out of N output elements, and the control information receiving means transmits the control transmitted from the output timing control device. Information is received. When the control information is received, the output timing correction means corrects the output timing of the output element by the output means.

  Therefore, the output timing control device acquires information on the output element output from the output device at a predetermined position, and calculates the deviation of the arrival timing of the output element at the predetermined position from the synchronization timing based on the acquired information. As a result, the deviation of the arrival timing of the output element at the predetermined position from the synchronization timing can be calculated. Furthermore, based on the deviation calculated in this way, control information used to control the output timing of the output element from the output device is generated and transmitted to the output device. Since the output timing can be controlled, it is possible to synchronize the arrival timings of the output elements output from the plurality of output devices at a predetermined position.

[Invention 14] The output timing control system of the invention 14 includes
An output timing control device according to a ninth aspect of the invention, S (S is an integer of 1 or more) output devices, and an integrated data providing device;
The output timing control device, the S output devices, and the integrated data providing device are connected so that data communication is possible,
The output timing control device includes:
Synchronization information receiving means for receiving information related to the synchronization timing transmitted from the integrated data providing device,
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Integrated data receiving means for receiving integrated data transmitted from the integrated data providing device;
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device;
Output timing correction means for correcting the output timing of the output element based on the control information received by the control information receiving means,
The integrated data providing device includes:
Integrated data transmitting means for transmitting integrated data including information on the N output elements to the output timing control device and the output device;
Synchronization information transmitting means for transmitting information related to the synchronization timing to the output timing control device.

With such a configuration, in the output timing control device, information related to the synchronization timing transmitted from the integrated data providing device is received by the synchronization information receiving unit, and the output timing control unit outputs the output to the output device. Control information used to control the output timing of the device is transmitted.
Further, in the output device, the integrated data receiving unit receives the integrated data transmitted from the integrated data providing device. When the integrated data is received, the output unit receives the N output elements based on the integrated data. Of these, x (1 ≦ x <N) output elements are output.

In the output device, the control information receiving means receives the control information transmitted from the output timing control device. When the control information is received, the output timing of the output element is corrected by the output timing correction means based on the control information.
Therefore, the output timing control device can acquire the synchronization information from the integrated data providing device, and the output device can acquire the integrated data from the integrated data providing device. Further, the output timing control device acquires information on the output element output from the output device at a predetermined position, and calculates a deviation of the arrival timing of the output element at the predetermined position from the synchronization timing based on the acquired information. As a result, the deviation of the arrival timing of the output element at the predetermined position from the synchronization timing can be calculated. Furthermore, based on the deviation calculated in this way, control information used to control the output timing of the output element from the output device is generated and transmitted to the output device. Since the output timing can be controlled, it is possible to synchronize the arrival timings of the output elements output from the plurality of output devices at a predetermined position.

[Invention 15] The output timing control system of Invention 15 is the output timing control system of any one of Inventions 11 to 14,
The output device includes a periodic signal output means for outputting the periodic signal together with the output element.
If it is such a structure, in an output device, a periodic signal will be output with the said output element by a periodic signal output means.

  Therefore, the output timing control device can calculate the deviation of the arrival timing at the predetermined position of the output element from the synchronization timing based on the periodic signal received from the output device. There is no need for a marker, and the effect of easily calculating the deviation with respect to the synchronization timing can be obtained.

[Invention 16] Further, the output timing control device of the invention 16 includes:
The output timing control device according to any one of claims 1 to 9,
It is characterized by comprising synchronization information receiving means for receiving, from the integrated data providing device, information relating to the synchronization timing when integrating the N output elements, which are output separately from the M output devices, respectively. .
Thereby, the same operation effect as the output timing control device in the output timing control system of the invention 12 or 14 can be obtained.

[Invention 17] On the other hand, the output device of Invention 17 is
Integrated data receiving means for receiving integrated data including information related to N (N is an integer of 2 or more) output elements transmitted from the integrated data providing device;
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device and used to control the output timing of the output element by the output means;
Output timing correcting means for correcting the output timing of the output element based on the control information received by the control information receiving means.
Thereby, in the output timing control system according to the twelfth or fourteenth aspect, an operation effect equivalent to that of the output device can be obtained.

[Invention 18] On the other hand, the integrated data providing device of Invention 18 includes:
Integrated data transmission means for transmitting integrated data including information on N (N is an integer of 2 or more) output elements to the output timing control device and the output device;
Synchronization information transmitting means for transmitting information related to a synchronization timing when the N output elements are integrated to the output timing control device.
Thereby, in the output timing control system according to the twelfth or fourteenth aspect, an operational effect equivalent to that of the integrated data providing apparatus can be obtained.

[Invention 19] On the other hand, the output timing control program of Invention 19 is
By outputting the output results obtained by integrating N (N is an integer of 2 or more) output elements, the N output elements are output separately from M (1 <M ≦ N) output devices, respectively. An output timing control program used to control the output timing of the output element from the output device in the system to be configured,
A synchronization information storage step for storing information on synchronization timing when integrating the N output elements in the synchronization information storage means;
An output information acquisition step of acquiring information on output elements output from the M output devices at a predetermined position;
A synchronization deviation calculating step for calculating each deviation of the arrival timing of the N output elements at the predetermined position with respect to the synchronization timing based on the acquisition result in the output information acquisition step and the stored contents of the synchronization information storage means; ,
An output timing control step for controlling the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result in the synchronization deviation calculation step It includes a program used to cause a computer to execute a process consisting of:
With this configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the first aspect can be obtained.

[Invention 20] Furthermore, the output timing control program of Invention 20 is the output timing control program of Invention 19,
The information relating to the output element includes information relating to the arrival timing of the output element to the predetermined position,
The information related to the synchronization timing includes information related to a part of the data and information related to the output timing of the data when the output element is data.
With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of aspect 2 can be obtained.

[Invention 21] Further, the output timing control program of the invention 21 is the output timing control program of the invention 20,
Information related to a part of the data is a feature amount extracted from the part of the data,
In the synchronization deviation calculation step, based on the acquisition result in the output information acquisition step, a feature amount is extracted from the acquisition result, the feature amount, information related to the arrival timing, and storage content of the synchronization information storage unit Based on the above, a deviation from the synchronization timing is calculated.
With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the invention 3 can be obtained.

[Invention 22] Further, the output timing control program of the invention 22 is the output timing control program of the invention 20,
At least one of the M output devices is configured to output a periodic signal having a periodicity together with the output element in synchronization with the output timing of the output element output from the own device,
In the output information acquisition step, the periodic signal is acquired as information relating to the output element.
With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the fourth aspect are obtained.

[Invention 23] The output timing control program of Invention 23 is the output timing control program of any one of Inventions 19 to 22,
When the M output devices are configured to include a video output device that outputs video as the output element and an audio output device that outputs audio as the output element,
In the output information acquisition step, information related to video and audio output from the video output device and the audio output device, respectively, is acquired at the predetermined position;
In the synchronization deviation calculation step, the arrival timing of the video output from the video output device at the predetermined position based on the acquisition result in the output information acquisition step and the storage content of the synchronization information storage means, and the audio output The difference between the arrival timing of the voice output from the apparatus and the synchronization timing is calculated.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the fifth aspect are obtained.

[Invention 24] Furthermore, the output timing control program of Invention 24 is the output timing control program of any one of Inventions 19 to 23,
When the M output devices are configured to include two or more video output devices that output video as the output elements,
In the output information acquisition step, information related to the video output from each of the two or more video output devices is acquired at the predetermined position,
In the synchronization deviation calculation step, the arrival timing of the video output from the two or more video output devices at the predetermined position is determined based on the acquisition result in the output information acquisition step and the storage content of the synchronization information storage means. , Calculating a deviation from the synchronization timing.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the sixth aspect are obtained.

[Invention 25] Further, the output timing control program of the invention 25 is the output timing control program of any one of the inventions 19 to 22,
When the M output devices are configured to include two or more sound output devices that output sound as the output elements,
In the output information acquisition step, information related to the sound output from each of the two or more sound output devices is acquired at the predetermined position,
In the synchronization deviation calculation step, the arrival timing of the voices output from the two or more voice output devices at the predetermined position based on the acquisition result in the output information acquisition step and the stored contents of the synchronization information storage means. , Calculating a deviation from the synchronization timing.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the seventh aspect are obtained.

[Invention 26] Further, the output timing control program of Invention 26 is the output timing control program of any one of Inventions 19 to 25,
When the plurality of output devices are configured to include an odor output device that outputs odor as the output element,
In the output information acquisition step, the information related to the odor output from the odor output device and the information related to the output element output from at least one output device different from the odor output device are the predetermined information. Get in position,
In the synchronization deviation calculation step, the arrival timing of the odor output from the odor output device at the predetermined position based on the acquisition result in the output information acquisition step and the stored content of the synchronization information storage means, and the odor output device The difference between the arrival timing of the output element output from the output device different from the above and the synchronization timing is calculated.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the eighth aspect are obtained.

[Invention 27] Furthermore, the output timing control program of Invention 27 is the output timing control program of any one of Inventions 19 to 26,
The information processing apparatus further includes a program used to cause a computer to execute a synchronization information receiving step of receiving information related to synchronization timing when integrating the N output elements from the integrated data providing apparatus.

  With this configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the output timing control device of the ninth aspect are obtained.

[Invention 28] On the other hand, the output device control program of Invention 28 is
A program used to control an output device capable of outputting x (1 ≦ x <N) output elements among N (N is an integer of 2 or more) output elements,
An integrated data receiving step for receiving integrated data including information on the N output elements transmitted from the integrated data providing device;
An output step of outputting the x output elements among the N output elements;
A control information receiving step for receiving control information transmitted from the output timing control device and used to control the output timing of the output element in the output step;
A program used for causing a computer to execute processing including an output timing correction step of correcting an output timing of an output element based on the control information received in the control information reception step is included.

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as the output device of the seventeenth aspect can be obtained.

[Invention 29] On the other hand, the integrated data providing apparatus control program of Invention 29 is
An integrated data transmission step of transmitting integrated data including information on N (N is an integer of 2 or more) output elements to the output timing control device and the output device;
A program used for causing a computer to execute a process including a synchronization information transmission step of transmitting information related to synchronization timing when integrating the N output elements to the output timing control device is included. .

  With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the integrated data providing apparatus of the eighteenth aspect are obtained.

[Invention 30] On the other hand, the output timing control device control method of Invention 30 includes:
By outputting the output results obtained by integrating N (N is an integer of 2 or more) output elements, the N output elements are output separately from M (1 <M ≦ N) output devices, respectively. An output timing control device control method used for controlling an output timing control device for controlling an output timing of the output element from the output device in a system comprising:
The output timing control device is
An output information acquisition step of acquiring information on output elements output from the M output devices at a predetermined position;
Based on the acquisition result in the output information acquisition step and the stored contents of the synchronization information storage means storing information related to the synchronization timing when the N output elements are integrated, the N outputs at the predetermined position A synchronization deviation calculating step for calculating each deviation of the arrival timing of the element with respect to the synchronization timing;
An output timing control step for controlling the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result in the synchronization deviation calculation step It is characterized by including.
Thereby, the same operation effect as the output timing control device of aspect 1 can be obtained.

[Invention 31] Furthermore, the output timing control device control method of the invention 31 is the output timing control device control method of the invention 30,
The information relating to the output element includes information relating to the arrival timing of the output element to the predetermined position,
The information related to the synchronization timing includes information related to a part of the data and information related to the output timing of the data when the output element is data.
Thereby, the same effect as that of the output timing control device of aspect 2 can be obtained.

[Invention 32] Furthermore, the output timing control device control method of Invention 32 is the output timing control device control method of Invention 31,
Information related to a part of the data is a feature amount extracted from the part of the data,
In the synchronization deviation calculation step, based on the acquisition result in the output information acquisition step, a feature amount is extracted from the acquisition result, the feature amount, information related to the arrival timing, and storage content of the synchronization information storage unit Based on the above, a deviation from the synchronization timing is calculated.
Thereby, the same effect as that of the output timing control device of aspect 3 can be obtained.

[Invention 33] Further, the output timing control device control method of Invention 33 is the output timing control device control method of Invention 31,
At least one of the M output devices is configured to output a periodic signal having a periodicity together with the output element in synchronization with the output timing of the output element output from the own device,
In the output information acquisition step, the periodic signal is acquired as information relating to the output element.
Thereby, the same effect as that of the output timing control device of aspect 4 can be obtained.

[Invention 34] Furthermore, the output timing control device control method of the invention 34 is the output timing control device control method of any one of the inventions 30 to 33,
When the M output devices are configured to include a video output device that outputs video as the output element and an audio output device that outputs audio as the output element,
In the output information acquisition step, information related to video and audio output from the video output device and the audio output device, respectively, is acquired at the predetermined position;
In the synchronization deviation calculation step, the arrival timing of the video output from the video output device at the predetermined position based on the acquisition result in the output information acquisition step and the storage content of the synchronization information storage means, and the audio output The difference between the arrival timing of the voice output from the apparatus and the synchronization timing is calculated.
Thereby, the same effect as that of the output timing control device of aspect 5 can be obtained.

[Invention 35] Furthermore, the output timing control device control method according to Invention 35 is the output timing control device control method according to any one of Inventions 30 to 34,
When the M output devices are configured to include two or more video output devices that output video as the output elements,
In the output information acquisition step, information related to the video output from each of the two or more video output devices is acquired at the predetermined position,
In the synchronization deviation calculation step, the arrival timing of the video output from the two or more video output devices at the predetermined position is determined based on the acquisition result in the output information acquisition step and the storage content of the synchronization information storage means. , Calculating a deviation from the synchronization timing.
Thereby, the same effect as that of the output timing control device of aspect 6 can be obtained.

[Invention 36] Further, the output timing control device control method of the invention 36 is the output timing control device control method of any one of the inventions 30 to 33,
When the M output devices are configured to include two or more sound output devices that output sound as the output elements,
In the output information acquisition step, information related to the sound output from each of the two or more sound output devices is acquired at the predetermined position,
In the synchronization deviation calculation step, the arrival timing of the voices output from the two or more voice output devices at the predetermined position based on the acquisition result in the output information acquisition step and the stored contents of the synchronization information storage means. , Calculating a deviation from the synchronization timing.
Thereby, the same effect as that of the output timing control device of aspect 7 can be obtained.

[Invention 37] Furthermore, the output timing control device control method of Invention 37 is the output timing control device control method of any one of Inventions 30 to 36,
When the plurality of output devices are configured to include an odor output device that outputs odor as the output element,
In the output information acquisition step, the information related to the odor output from the odor output device and the information related to the output element output from at least one output device different from the odor output device are the predetermined information. Get in position,
In the synchronization deviation calculation step, the arrival timing of the odor output from the odor output device at the predetermined position based on the acquisition result in the output information acquisition step and the stored content of the synchronization information storage means, and the odor output device The difference between the arrival timing of the output element output from the output device different from the above and the synchronization timing is calculated.
Thereby, the same effect as that of the output timing control device of aspect 8 can be obtained.

[Invention 38] The output timing control device control method of Invention 38 is the output timing control device control method of any one of Inventions 30 to 37,
The information processing apparatus further includes a program used to cause a computer to execute a synchronization information receiving step of receiving information related to synchronization timing when integrating the N output elements from the integrated data providing apparatus.
Thereby, the same effect as that of the output timing control device of aspect 9 can be obtained.

[Invention 39] On the other hand, an output device control method according to Invention 39 includes:
An output device control method used to control an output device capable of outputting x (1 ≦ x <N) output elements among N (N is an integer of 2 or more) output elements,
The output device is
An integrated data receiving step for receiving integrated data including information on the N output elements transmitted from the integrated data providing device;
An output step of outputting the x output elements among the N output elements;
A control information receiving step for receiving control information transmitted from the output timing control device and used to control the output timing of the output element in the output step;
An output timing correcting step of correcting the output timing of the output element based on the control information received in the control information receiving step.
Thereby, the same effect as that of the output device of the seventeenth aspect can be obtained.

[Invention 40] On the other hand, the integrated data providing apparatus control method according to Invention 40 includes:
An integrated data transmission step of transmitting integrated data including information on N (N is an integer of 2 or more) output elements to the output timing control device and the output device;
A synchronization information transmission step of transmitting information related to synchronization timing when the N output elements are integrated to the output timing control device.
Thereby, the same effect as that of the integrated data providing apparatus of the eighteenth aspect can be obtained.

[First Embodiment]
A first embodiment of the present invention will be described below with reference to the drawings. 1 to 10 are an output timing control device, a video output device, an output timing control system, an output device, an integrated data providing device, an output timing control program, an output device control program, an integrated data providing device control program, It is a figure which shows 1st Embodiment of an output timing control apparatus control method, an output apparatus control method, and an integrated data provision apparatus control method.

  In the present embodiment, video and audio are output from separate output devices based on integrated data consisting of video data and audio data that constitute an output result of a movie or the like that integrates video and audio. In the display system, the output timing of video and audio from each output device is controlled so that these arrival timings are synchronized in the audience seat located at a position away from the speaker that outputs audio and the screen that displays the video. .

First, the configuration of the video display system 1 according to the present invention will be described with reference to FIG. FIG. 1 is an overview of a video display system 1 according to the present invention.
As shown in FIG. 1, the video display system 1 includes an integrated data transmission device 2 that transmits integrated data composed of video data and audio data to a video playback device 3 and an audio playback device 4 described later, and an integrated data transmission device 2. The video is reproduced based on the integrated data received from the image, and the reproduced video is optically projected and output in the video projection unit 33 described later, and the integrated data and the audio information acquisition unit arranged in the viewer's seat described later 32 reproduces audio based on the integrated data received from the integrated data transmitting device 2 and the video reproducing device 3 that synchronizes the arrival timing of the audio and video at the viewer's seat based on the audio information acquired in 32 An audio reproduction device 4 for outputting audio from speakers 4a and 4b, which will be described later, a screen 5 for displaying an image optically projected from the video reproduction device 3, and And it has a inclusive configuration.

  In the present embodiment, the distance between the speakers 4a and 4b and the viewer's seat where the viewer is located is greatly separated. Therefore, the time for the video to reach the viewer's seat and the time for the sound to reach the viewer's seat are There will be a difference. In other words, since the sound arrives later than the image viewed by the viewer, there is a difference between the image and the sound. Therefore, in the video reproduction device 3 according to the present embodiment, the audio information acquisition unit 32 acquires the audio information that is output from the speakers 4a and 4b of the audio reproduction device 4 and reaches the viewer's seat. By controlling the video output timing based on the audio information, the video arrival timing and the audio arrival timing reaching the viewer's seat are synchronized. Here, to synchronize is to make the video corresponding to the audio reaching the viewer's seat the same arrival timing as the audio.

  Further, the integrated data transmission device 2 transmits synchronization information for synchronizing audio and video together with the integrated data to the video reproduction device 3. Here, the synchronization information includes information for detecting the marker when a part of the audio data is a synchronization marker. The synchronization marker is described in a predetermined description language as metadata together with other synchronization information, and the characteristic parameters of the synchronization marker are described together with the contents, data format, size, and the like.

Further, a detailed configuration of the video reproduction device 3 will be described with reference to FIG. FIG. 2 is a block diagram showing a detailed configuration of the video reproduction device 3.
As shown in FIG. 2, the video playback device 3 includes a video playback unit 30, a synchronization marker detection unit 31, an audio information acquisition unit 32, a video projection unit 33, and a data communication unit 34. ing.

The video reproduction unit 30 includes a clock generation circuit 30a, a synchronization timing calculation unit 30b, a synchronization deviation calculation unit 30c, and a reproduction circuit / delay circuit 30d.
The clock generation circuit 30a is composed of a crystal oscillator or the like, and has a function of generating a clock signal having a predetermined period. Then, the generated clock signal is supplied to the synchronization timing calculation unit 30b, the reproduction circuit / delay circuit 30d, and the synchronization marker detection unit 31.

  The synchronization timing calculation unit 30b calculates a playback start time T2 corresponding to the playback video of the audio corresponding to the synchronization marker in the video playback device 3 from the playback start time of the video and the playback time of the audio corresponding to the synchronization marker. The calculated reproduction start time T2 is transmitted to the synchronization deviation calculation unit 30c. Further, the synchronization information received from the integrated data transmission unit 2 via the data communication unit 34 is transmitted to the synchronization marker detection unit 31.

  Based on the reproduction start time T2 received from the synchronization timing calculation unit 30b and the arrival time T1 received from the synchronization marker detection unit 31, the synchronization deviation calculation unit 30c determines the synchronization timing of the video and audio arrival timings at the viewer's seat. The deviation dT [s] (= T1-T2) is calculated, and the calculated deviation dT (hereinafter simply referred to as deviation dT) is transmitted to the reproduction circuit / delay circuit 30d.

The reproduction circuit / delay circuit 30d reproduces the video based on the video data included in the integrated data received from the integrated data transmission device 2 via the data communication unit 34, and based on the shift dT received from the synchronization shift calculation unit 30d. Correct the video playback speed.
The synchronization marker detection unit 31 receives a synchronization marker from the audio information based on the synchronization information received from the synchronization timing calculation unit 30b and stored in a storage unit (not shown) and the audio information received from the audio information acquisition unit 32. The matching information is detected, and the arrival time T1 of the detected information is calculated based on the clock signal from the clock generation circuit 30a. Then, the calculated arrival time T1 is transmitted to the synchronization deviation calculation unit 30c. In the present embodiment, the synchronization marker detection process performs A / D conversion on the acquired audio information by an A / D converter, and performs FFT (Fast Fourier Transform) processing on the converted data. The frequency component is extracted from the audio information as a feature amount, and this feature amount is compared with a synchronization marker (feature parameter) included in the synchronization information. In the present embodiment, if the error between the feature quantity and the feature parameter is equal to or smaller than a predetermined value, it is determined that the two match each other, and the matched data portion in the audio data is detected as a synchronization marker.

The voice information acquisition unit 32 includes a microphone for collecting sound, acquires voice information (analog signal) at the viewer's seat via the microphone, and transmits the acquired voice information to the synchronization marker detection unit 31. To do. Therefore, the microphone is arranged at the viewer's seat.
The video projection unit 33 has a function of optically projecting a video composed of the video signal onto the screen 5 based on the video signal from the reproduction circuit / delay circuit 30d, based on the same principle as a known liquid crystal projector. . Specifically, a light source composed of an ultra-high pressure mercury lamp, a xenon lamp or the like, a fly-eye lens for obtaining a uniform illuminance distribution on the irradiated surface, and RGB primary colors in the wavelength range of incident light based on the video signal The color modulation part which modulates each brightness | luminance, and the projection part which light-projects the light which injected from the color modulation part on the screen 5 are comprised.

The color modulator is composed of three liquid crystal light valves with a plurality of pixels whose transmittance can be controlled independently, arranged in a matrix, and a field lens. And a dichroic prism that synthesizes incident light of each color.
The projection unit includes a projection lens, and is configured to enlarge and project incident light onto the screen 5 by the projection lens.

The data communication unit 34 receives the integrated data and the synchronization information from the integrated data transmission device 2 and stores them in a storage unit (not shown), or transmits the reproduction start timing information to the audio reproduction device 4. 3 and a function of performing data transmission / reception between devices capable of data communication.
Here, the video reproduction apparatus 3 includes a processor, a RAM (Random Access Memory), and a ROM (Read Only Memory) in which a dedicated program is stored, which is not shown, and executes the dedicated program by the processor. This fulfills the function of each of the above components. In addition, each of the above-described components includes a component that fulfills its function only by a dedicated program, a component that controls its hardware by a dedicated program, and fulfills its function.

Further, the video reproduction apparatus 3 is configured to activate a predetermined program stored in a predetermined area of the ROM by the processor and execute the reproduction timing control process shown in the flowchart of FIG. 3 according to the program.
FIG. 3 is a flowchart showing a playback timing control process in the video playback device 3.

When the reproduction timing control process is executed by the processor, first, as shown in FIG. 3, the process proceeds to step S100.
In step S100, the data communication unit 34 determines whether or not synchronization information has been received from the integrated data transmission device 2. If it is determined that the synchronization information has been received (Yes), the data communication unit 34 stores the synchronization information in a storage unit (not shown) and proceeds to step S102. If not (No), wait until receiving.

If the process proceeds to step S102, it is determined whether or not the integrated data corresponding to the synchronization information received in step S100 is received via the data communication unit 34. If it is determined that the integrated data is received (Yes), step S104 is performed. If not (No), wait until receiving.
When the process proceeds to step S104, the reproduction circuit / delay circuit 30d starts reproducing the video based on the video data included in the integrated data received in step S102, and the process proceeds to step S106.

In step S106, based on the synchronization information received in step S100 and the integrated data received in step S102, the playback start time T2 corresponding to the playback video in the audio video playback device 3 corresponding to the synchronization marker included in the synchronization information is calculated. Then, the process proceeds to step S108.
In step S108, the voice information acquisition unit 32 determines whether or not voice information has been acquired. If it is determined that the voice information has been acquired (Yes), the process proceeds to step S110. If not (No), the voice information is determined until acquisition. Continue processing.

When the process proceeds to step S110, the synchronization marker detection unit 31 performs a synchronization marker detection process, and the process proceeds to step S112.
In step S112, it is determined whether or not a synchronization marker is detected based on either the detection success notification or the detection failure notification from the synchronization marker detection unit 31, and if it is determined that the detection is detected (Yes), step S114 is performed. If not (No), the process proceeds to step S120.

When the process proceeds to step S114, the synchronization deviation calculation unit 30c determines the image at the viewer's seat based on the reproduction start time T2 calculated by the synchronization timing calculation unit 30b and the arrival time T1 calculated by the synchronization marker detection unit 31. The shift dT from the audio synchronization timing is calculated, and the process proceeds to step S116.
In step S116, the reproduction circuit / delay circuit 30d determines whether or not there is a deviation from the synchronization timing based on the notification result from the synchronization deviation calculation unit 30c, and if it is determined that there is a deviation (Yes). The process proceeds to step S118. Otherwise (No), the process proceeds to step S120. In the present embodiment, when a notification of erroneous detection is received, the synchronization marker is excluded from the correction target.

When the process proceeds to step S118, the reproduction circuit / delay circuit 30d corrects the video reproduction timing, and the process proceeds to step S120. Here, when the deviation dT is dT> 0, the playback timing is corrected by increasing the video playback speed by a necessary time from the current speed, while if dT ≦ 0, the video playback speed is increased by a time required by the current speed. Slow down.
In step S120, the reproduction circuit / delay circuit 30d reproduces the video based on the video data included in the integrated data, and the process proceeds to step S122.

In step S122, the playback circuit / delay circuit 30d determines whether or not the video playback has ended. If it is determined that the playback has ended (Yes), the process proceeds to step S100; The process proceeds to S108.
That is, when video playback is started in the video playback device 3, the processing from step S108 to step S122 is repeated until video playback ends.

Next, based on FIG. 4, the synchronous marker detection process of step S110 is demonstrated in detail.
FIG. 4 is a flowchart showing the synchronization marker detection process in the video playback device 3.
The synchronization marker detection process is a process for detecting the synchronization marker from the audio information based on the synchronization information and the audio information, and calculating the arrival time of the detected synchronization marker, and when executed in step S110, FIG. As shown in FIG. 1, first, the process proceeds to step S200.

In step S200, the synchronization marker detection unit 31 performs A / D conversion on the audio information acquired from the audio information acquisition unit 32 by the A / D converter, and the process proceeds to step S202.
When the process proceeds to step S202, the synchronization marker detection unit 31 performs FFT processing on the audio information A / D converted in step S200, and the process proceeds to step S204.
In step S204, the result of the FFT process in step S202 is compared with the synchronization marker (feature parameter) included in the synchronization information acquired from the synchronization timing calculation unit 30b, and the process proceeds to step S206. In the present embodiment, the feature parameter included in the metadata is compared with the FFT processing result.

In step S206, the synchronization marker detection unit 31 determines whether or not the feature amounts match based on whether or not the FFT processing result and the feature parameter are within a predetermined error range. If yes (Yes), the process proceeds to step S208. If not (No), the process proceeds to step S210.
When the process proceeds to step S208, the synchronization marker detection unit 31 supplies the clock information acquired from the voice information acquisition unit 32 as a synchronization marker, and the start time is supplied from the clock generation circuit 30a as the arrival time T1. The calculation result is transmitted to the synchronization deviation calculation unit 30c together with the detection success notification, and the series of processes is terminated and the process returns to the original process.

On the other hand, when the process proceeds to step S210, the synchronization marker detection unit 31 transmits a detection failure notification to the synchronization deviation calculation unit 30c on the assumption that the synchronization information is not included in the audio information acquired from the audio information acquisition unit 32. Then, the series of processes is terminated and the process returns to the original process.
Next, based on FIG. 5, the synchronization deviation calculation process of step S114 will be described in detail.
FIG. 5 is a flowchart showing the synchronization deviation calculation process in the video reproduction device 3.

The synchronization shift calculation process is a process of calculating a shift dT with respect to the synchronization timing between the audio and the video based on the reproduction start time T2 and the arrival time T1, and when executed in step S114, as shown in FIG. First, the process proceeds to step S300. Here, the video and audio synchronization timing is the timing at which the audio corresponding to the playback video is played back,
In step S300, the synchronization deviation calculation unit 30c acquires the reproduction start time T2 stored in the storage unit, and proceeds to step S302.

In step S302, the deviation dT with respect to the synchronization timing of the video and audio at the viewer's seat based on the reproduction start time T2 read in step S300 and the arrival time T1 acquired from the synchronization marker detection unit 31 in the synchronization deviation calculation unit 30c. (= T1-T2) is calculated, and the process proceeds to step S304.
In step S304, the synchronization deviation calculation unit 30c determines whether the magnitude of the deviation dT is greater than or equal to a predetermined value. If it is determined that the magnitude is greater than or equal to the predetermined value (Yes), the process proceeds to step S306; (No) moves to step S308.

When the process proceeds to step S306, the synchronization deviation calculation unit 30c notifies the reproduction circuit / delay circuit 30d that it is a false detection, ends a series of processes, and returns to the original process.
On the other hand, when the process proceeds to step S308, the synchronization deviation calculation unit 30c notifies the reproduction circuit / delay circuit 30d of the calculation result, ends the series of processes, and returns to the original process.
Next, the reproduction timing correction process in step S118 will be described in detail with reference to FIG.

FIG. 6 is a flowchart showing a playback timing correction process in the video playback device 3.
The reproduction timing correction process is a process of correcting the reproduction timing of the reproduction circuit based on the calculation result from the synchronization deviation calculation unit 30c and correcting the deviation dT with respect to the synchronization timing between the audio and the video, and is executed in step S118. Then, as shown in FIG. 6, first, the process proceeds to step S400.

In step S400, the reproduction circuit / delay circuit 30d determines whether dT is greater than 0. If it is determined that it is greater than 0 (Yes), the process proceeds to step S402. If not (No), step S400 is performed. The process proceeds to S404.
When the process proceeds to step S402, the playback circuit / delay circuit 30d performs control for changing the video playback speed for the calculated time, ends the series of processes, and returns to the original process.

On the other hand, when the process proceeds to step S404, the playback circuit / delay circuit 30d performs control to reduce the video playback speed, ends the series of processes, and returns to the original process.
Next, the operation of the present embodiment will be described with reference to FIGS.
Here, FIG. 7 is a diagram showing a transmission state of synchronization information and integrated data (audio data and video data), and FIG. 8 is a diagram showing a series of flow until a synchronization marker is detected from audio information. . FIG. 9 is a diagram illustrating a description example of metadata, and FIG. 10 is a diagram illustrating a table example of a sound reproduction start time corresponding to a synchronization marker.

First, the integrated data transmission device 2 transmits synchronization information to the video reproduction device 3, and then transmits integrated data corresponding to the synchronization information to the video reproduction device 3 and the audio reproduction device 4.
The video reproduction device 3 receives the synchronization information via the data communication unit 34, stores it in a storage unit (not shown) (step S100), and receives integrated data corresponding to the synchronization information (step S102). The video is reproduced by the reproduction circuit / delay circuit 30d based on the video data included in (Step S104). In the present embodiment, the playback start timing of video and audio is the timing according to an instruction from the user of the video playback device 3, and information on this playback start timing is also transmitted to the audio playback device 4. Therefore, the video reproduction device 3 and the audio reproduction device 4 start reproduction at the time when the instruction is given in accordance with the instruction from the user. Note that the video playback device 3 transmits the video playback start time to the synchronization timing calculator 30b. Further, the synchronization information and the integrated data are transmitted from the integrated data transmitting apparatus 2 in the order shown in FIG. As shown in FIG. 7, the synchronization information (metadata) includes information on the synchronization marker. In addition, the synchronization information includes information on the reproduction time of the video data corresponding to the synchronization marker, and the integrated data is data in which audio data and video data are integrated. Includes data corresponding to the synchronization marker. When the video is reproduced by the reproduction circuit / delay circuit 30d, the video projection unit 33 enlarges and projects the reproduced video on the screen 5.

  In the synchronization timing calculation unit 30b of the video playback device 3, when the playback of the video is started by the playback circuit / delay circuit 30d and the playback start time of the video is acquired from the playback circuit / delay circuit 30d, the playback start time Based on the reproduction time of the audio corresponding to the synchronization marker included in the synchronization information, the reproduction start time T2 in real time corresponding to the video reproduced in the video reproduction device 3 of the audio corresponding to the synchronization marker is calculated. . The audio reproduction start time T2 corresponding to the synchronization marker is described between the tag <time> and the tag </ time> in the metadata shown in FIG. In addition, the synchronization marker identifies its type by giving an identification number to the <data id> tag, for example, <data id = 1>. For example, if the video playback start time in the video playback device 3 is “09:00:00”, the synchronization marker of identification number 1 is described as “00: 03: 12: 04” as the playback start time. Therefore, the actual reproduction start time T2 of the audio corresponding to the synchronization marker is “09: 03: 12: 04” obtained by adding the reproduction start time of the video. Similarly, since “00: 06: 25: 36” is described as the reproduction start time for the synchronization marker of identification number 1, the reproduction start time T2 in real time is “09: 06: 25: 36”. It becomes. The start time T2 calculated as described above is configured as a data table in which the start time T2 and the identification number of the synchronization marker are associated with each other as shown in FIG. 10, and is stored in a storage unit (not shown).

On the other hand, when receiving the integrated data from the integrated data transmitting device 2, the audio reproducing device 4 starts reproducing the audio based on the information of the reproduction start timing from the video reproducing device 3. And the reproduced | regenerated audio | voice is output from the speakers 4a and 4b.
When the video and audio are reproduced, the audio output from the speakers 4a and 4b arrives at the audio information acquisition unit 32 disposed in the viewer's seat located away from the speakers 4a and 4b. Thereby, the voice information acquisition unit 32 acquires the voice information that has arrived through the microphone (step S108). When the voice information is acquired by the voice information acquisition unit 32, the synchronization marker detection process is started in the synchronization marker detection unit 31 (step S110).

  Since the audio information acquired from the audio information acquisition unit 32 is an analog signal as shown in FIG. 8A, the synchronization marker detection unit 31 first performs A / D conversion (step S200). Then, an FFT process is performed on the converted audio information to extract a feature amount (step S202). Here, since the time length of the synchronization marker is known, the size of the window used in the FFT processing is set according to this. The window function used for window weighting is selected from a rectangular window, a Hamming window, a Hanning window, a Blackman-Harris window, and the like. Therefore, the feature amount as shown in FIG. 8B is extracted by performing FFT processing by shifting the set window by a predetermined time width.

  When the feature amount is extracted from the audio information, the feature amount (see FIG. 8B) and a synchronization marker having the same feature parameter as the extracted feature amount as shown in FIG. In comparison (step S204), when the error (%) between the two is less than or equal to a predetermined value (for example, 3% or less), it is determined that the two match each other. (Step S206). Here, as shown in FIG. 9, the synchronization marker is described as metadata together with other synchronization information. For each synchronization marker of each identification number, between the <data id> tag and the </ data> tag. Describes the characteristic parameter values.

  When the synchronization marker of each identification number is compared with the extracted feature quantity and it is determined that they match, the matched data portion in the audio data is based on the clock signal supplied from the clock generation circuit 30a. Is calculated as arrival time T1. The calculated arrival time T1 and the corresponding identification number (Id) are transmitted to the synchronization deviation calculation unit 30c together with information indicating successful detection (step S208).

On the other hand, the synchronization marker of each identification number is compared with the extracted feature amount, and if it is determined that there is no match, information indicating a detection failure is transmitted to the synchronization deviation calculation unit 30c (step S1). S210).
When the synchronization deviation calculation unit 30c acquires information indicating the detection result from the synchronization marker detection unit 31, the synchronization deviation calculation unit 30c determines whether or not a synchronization marker is detected based on the acquired information. If it is determined that it has been detected and information indicating a detection failure is acquired, it is determined that it has not been detected (step S114). If the synchronization deviation calculation unit 30c determines that a synchronization marker has been detected, a synchronization deviation calculation process is started (step S116).

  First, the synchronization shift calculation unit 30c reads the start time T2 stored in the storage unit (step S300). Next, based on the read T2, the arrival time T1 and the identification number acquired from the synchronization marker detection unit 31, a synchronization shift dT (= T1-T2) is calculated for each identification number of the synchronization marker (step S302). Here, it is assumed that time “09: 07: 22: 04” is transmitted as the arrival time T1 and “Id = 1” is transmitted as the identification number from the synchronization marker detection unit 31. When T2 is “09: 03: 12: 04” from the start time T2 when Id is 1 and the arrival time T1 shown in FIG. 10, dT (= T1-T2) is “00:04:10 : 00 ", which is dT1. On the other hand, when T2 is “09: 07: 21: 48”, dT (= T1−T2) is calculated as “00: 00: 00: 16”, which is set as dT2. Then, when dT is calculated, the absolute value of the calculated dT is compared with a predetermined value (here, 10 [s] (“00: 00: 10: 00”)) (step S304). Based on this comparison result, when the absolute value of dT is 10 [s] or more, the detected synchronization marker is determined to be an erroneously detected synchronization marker, and information indicating the erroneous detection is transmitted to the reproduction circuit / delay circuit 30d. (Step S306). On the other hand, if the absolute value of dT is less than 10 [s], the calculated dT is transmitted to the reproduction circuit / delay circuit 30d (step S308).

  The reproduction circuit / delay circuit 30d determines whether or not there is a synchronization shift based on the notification result from the synchronization shift calculation unit 30c (step S116). Here, the determination as to whether or not there is a synchronization error is that the calculation result of dT acquired from the synchronization error calculation unit 30c is dT = 0 (in this embodiment, “00:00:00”). In this case, it is determined that there is no synchronization shift, and when dT ≠ 0, it is determined that there is a synchronization shift.

If it is determined that there is a synchronization error, the reproduction circuit / delay circuit 30d starts a process for correcting the video reproduction timing (step S118).
The reproduction circuit / delay circuit 30d first determines whether or not dT acquired from the synchronization deviation calculation unit 30c is greater than “0” (step S400). That is, when dT is larger than “0”, the sound corresponding to the synchronization marker reaches the viewer's seat first with respect to the video corresponding to the synchronization marker, while dT is “ When it is smaller than “0”, it means that the sound corresponding to the synchronization marker reaches the viewer's seat with a delay from the video corresponding to the synchronization marker.

  Therefore, when dT is larger than “0”, the playback circuit / delay circuit 30d calculates a control amount for increasing the video playback speed (step S402). On the other hand, when dT is smaller than “0”. Then, in the reproduction circuit / delay circuit 30d, a control amount for reducing the video reproduction speed is calculated (step S404). For example, when the dT is larger than “0”, the control amount for the reproduction speed is calculated as (1 + a) times (for example, a is 0.1), while the dT is “0”. Is smaller than the control amount for multiplying the reproduction speed by (1−a) times (for example, a is 0.1), and the time required to eliminate the deviation is calculated.

When the control amount and the time are calculated, the playback circuit / delay circuit 30d performs a video playback process by changing the playback speed for the calculated time based on the control amount (step S120).
In this way, by increasing or decreasing the video playback speed, the shift dT relative to the synchronization timing of the video and audio arrival timing at the viewer's seat is corrected.

  In this way, in the present embodiment, audio information from the speakers 4a and 4b located at a position away from the viewer's seat is acquired at the viewer's seat, and a preset synchronization marker is detected from the acquired audio information. Thus, the arrival time T1 of the voice is calculated, and the deviation dT between the two at the synchronization timing is calculated from the arrival time T1 and the actual reproduction start time T2 of the voice corresponding to the synchronization marker. Based on the deviation dT, the deviation dT can be corrected by changing the reproduction speed of the video in the video reproduction device 3. Therefore, it is possible to correct the deviation of the arrival timing of the video and audio at the viewer's seat with respect to the synchronization timing.

  In the first embodiment, the process of storing the synchronization information from the integrated data transmission device 2 and storing it in the storage unit (not shown) is the first, third, fifth, sixth, eighth, nineteenth, twenty-first, twenty-third, twenty-third, twenty-fourth, twenty-sixth, and twenty-sixth. , 30, 32, 34, 35, and 37, a series of calculating synchronization deviation dT performed by the synchronization timing calculation unit 30b, the synchronization deviation calculation unit 30c, and the synchronization marker detection unit 31. This processing corresponds to the synchronization deviation calculation means of any one of the inventions 1, 3, 5, 6, and 8, and the audio information acquisition unit 32 corresponds to the output information acquisition means of the inventions 1, 3, 5, 6, and 8. Correspondingly, the video playback speed correction processing in the playback circuit / delay circuit 30d corresponds to the output timing control means of the invention 1 or 14.

In the first embodiment, the process of receiving synchronization information via the data communication unit 34 corresponds to the synchronization information receiving means of the invention 14 and receives integrated data via the data communication unit 34. The processing corresponds to the integrated data receiving means of the fourteenth aspect.
In the first embodiment, step S100 corresponds to the synchronization information receiving step of the invention 27 or 37, and step S102 corresponds to the integrated data receiving step of the invention 28 or 39, and steps S110 and Step S114 corresponds to the synchronization deviation calculation step of any one of Inventions 19, 21, 23, 24, 26, 30, 32, 34, 35, and 37, and Steps S116 and S118 correspond to those of Invention 19 or Invention 30, respectively. Corresponds to the output timing control step.

In the first embodiment, the synchronization timing shift dT is not limited to correction by changing the playback speed. When the audio is advanced, the video is skipped by dT, or the audio is skipped. Is delayed, it may be performed by temporarily stopping the reproduction of the video by dT.
In the first embodiment, a part of audio data in the audible band that can be listened to by the viewer is used as the synchronization marker. However, the present invention is not limited to this, and the audio data is outside the audible band (20 to 20 kHz). These data may be superimposed on the audio signal and this signal may be detected as a synchronization marker. In this case, signal start, signal end, frequency change, and volume change can all be treated as synchronization markers.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. 11 to 15 show an output timing control device, an output timing control system, an output device, an integrated data providing device, an output timing control program, an output device control program, an integrated data providing device control program, and an output timing control device according to the present invention. It is a figure which shows 2nd Embodiment of a control method, an output device control method, and an integrated data provision apparatus control method.

  In the present embodiment, video and audio are output from separate output devices based on integrated data consisting of video data and audio data that constitute an output result of a movie or the like that integrates video and audio. In the display system, the output timing of video and audio from each output device is controlled so that these arrival timings are synchronized in the audience seat located at a position away from the speaker that outputs audio and the screen that displays the video. .

  The difference from the first embodiment is that the video displayed on the screen 5 is captured from the viewer's seat, video information is acquired, a synchronization marker is detected from the video information, and the detected synchronization marker The difference dT with respect to the synchronization timing of the arrival timing of audio and video at the viewer's seat is calculated based on the start time, and the difference is corrected by changing the audio reproduction speed based on the calculated dT.

First, the configuration of the video display system 6 according to the present invention will be described with reference to FIG. FIG. 11 is an overview of the video display system 6 according to the present invention.
As shown in FIG. 11, the video display system 6 includes an integrated data transmission device 2 that transmits integrated data including video data and audio data to an audio reproduction device 7 and a video reproduction device 8 described later, and the integrated data transmission device 2. Audio is reproduced on the basis of the integrated data received from the computer, and the reproduced audio is output from speakers 73 and 74, which will be described later. In addition, the integrated data and the video information acquisition unit 72 provided in the viewer's seat, which will be described later. Based on the acquired video information, the audio reproduction device 7 that synchronizes the audio and video arrival timing at the viewer's seat and the integrated data received from the integrated data transmission device 2 reproduces the video, The image playback device 8 that projects light onto the screen 5 and the screen 5 that displays the image light projected from the image playback device 8 are included. There.

  In the present embodiment, the distance between the speakers 73 and 74 and the viewer's seat where the viewer is located is greatly separated. Therefore, the time for the video to reach the viewer's seat and the time for the sound to reach the viewer's seat are There will be a difference. In other words, since the sound arrives later than the image viewed by the viewer, there is a difference between the image and the sound. Therefore, in the audio playback device 7 according to the present embodiment, the video information acquisition unit 72 acquires video information that is projected and displayed on the screen 5 from the video playback device 8 and reaches the viewer's seat. By controlling the audio output timing based on the above, the audio arrival timing and the video arrival timing reaching the viewer's seat are synchronized. Here, to synchronize is to make the sound corresponding to the video reaching the viewer's seat the same arrival timing as the video.

  Further, the integrated data transmission device 2 transmits synchronization information for synchronizing audio and video together with the integrated data to the video reproduction device 3. Here, the synchronization information includes information for detecting the marker when a part of the video data is a synchronization marker. The synchronization marker is described in a predetermined description language as metadata together with other synchronization information, and the characteristic parameters of the synchronization marker are described together with the contents, data format, size, and the like.

Furthermore, the detailed structure of the audio | voice reproduction apparatus 7 is demonstrated based on FIG. FIG. 12 is a block diagram showing a detailed configuration of the audio reproduction device 7.
As shown in FIG. 12, the audio reproduction device 7 includes an audio reproduction unit 70, a synchronization marker detection unit 71, a video information acquisition unit 72, speakers 73 and 74, and a data communication unit 75. ing.

The audio reproduction unit 70 includes a clock generation circuit 70a, a synchronization timing calculation unit 70b, a synchronization deviation calculation unit 70c, and a reproduction circuit / delay circuit 70d.
Since the clock generation circuit 70a is the same as the clock generation circuit 30a in the first embodiment, description thereof is omitted.
The synchronization timing calculation unit 70b calculates the reproduction start time T2 corresponding to the reproduction audio in the audio reproduction device 7 of the video corresponding to the synchronization marker from the audio reproduction start time and the reproduction time of the video corresponding to the synchronization marker. The calculated reproduction start time T2 is transmitted to the synchronization deviation calculation unit 70c. Further, the synchronization information received from the integrated data transmission unit 2 via the data communication unit 75 and stored in a storage unit (not shown) is transmitted to the synchronization marker detection unit 71.

  Based on the reproduction start time T2 received from the synchronization timing calculation unit 70b and the arrival time T1 received from the synchronization marker detection unit 71, the synchronization deviation calculation unit 70c determines the synchronization timing of the arrival timing of video and audio at the viewer's seat. The deviation dT [s] (= T1-T2) is calculated, and the calculated deviation dT (hereinafter simply referred to as deviation dT) is transmitted to the reproduction circuit / delay circuit 30d.

The reproduction circuit / delay circuit 70d reproduces sound based on the sound data included in the integrated data received from the integrated data transmission device 2 via the data communication unit 75, and based on the shift dT received from the synchronization shift calculation unit 70d. Correct the audio playback speed.
The synchronization marker detection unit 71 detects information matching the synchronization marker from the video information based on the synchronization information received from the synchronization timing calculation unit 70b and the video information received from the video information acquisition unit 72. The arrival time T1 of the detected information is calculated based on the clock signal from the clock generation circuit 70a. Then, the calculated arrival time T1 is transmitted to the synchronization deviation calculation unit 70c. In the present embodiment, the synchronization marker detection process extracts a screen area (an area corresponding to the video portion of the integrated data) from the acquired video information, divides the video of the extracted screen area into four parts, An average value of luminance values of each of the four divided areas is calculated, and the feature value is used as a feature value, and the feature value is compared with a synchronization marker (feature parameter) included in the synchronization information. In the present embodiment, if the difference between the feature quantity and the feature parameter is equal to or smaller than a predetermined value, it is determined that the two match each other, and the matched data portion in the audio data is detected as a synchronization marker.

The video information acquisition unit 72 includes a digital video camera, captures video displayed on the screen 5 by the digital video camera from the viewer's seat, and uses the captured video information (digital data) as a synchronization marker detection unit. 71 is transmitted. Therefore, the digital video camera is disposed at the viewer's seat.
The speakers 73 and 74 convert the audio signal reproduced by the reproduction circuit / delay circuit 70d into sound waves and output the sound waves.

The data communication unit 75 is a device capable of data communication with the audio reproduction device 7 such as receiving integrated data and synchronization information from the integrated data transmission device 2 and transmitting information of reproduction start timing to the video reproduction device 8. Data transmission / reception to / from.
Here, the audio reproducing device 7 includes a processor, a RAM (Random Access Memory), and a ROM (Read Only Memory) in which a dedicated program is stored, which is not shown, and executes the dedicated program by the processor. This fulfills the function of each of the above components. In addition, each of the above-described components includes a component that fulfills its function only by a dedicated program, a component that controls its hardware by a dedicated program, and fulfills its function.

Furthermore, the audio reproducing device 7 starts a predetermined program stored in a predetermined area of the ROM by the processor, and executes the reproduction timing control process shown in the flowchart of FIG. 13 according to the program.
FIG. 13 is a flowchart showing playback timing control processing in the audio playback device 7.

When the reproduction timing control process is executed by the processor, as shown in FIG. 13, first, the process proceeds to step S500.
In step S500, the data communication unit 75 determines whether or not the synchronization information has been received from the integrated data transmission apparatus 2. If it is determined that the synchronization information has been received (Yes), the synchronization information is stored in a storage unit (not shown). Then, the process proceeds to step S502. If not (No), the process waits until reception.

When the process proceeds to step S502, it is determined whether or not the integrated data corresponding to the synchronization information received in step S500 is received via the data communication unit 75. If not (No), wait until receiving.
When the process proceeds to step S504, the reproduction circuit / delay circuit 70d starts reproducing the sound based on the audio data included in the integrated data received in step S502, and the process proceeds to step S506.

In step S506, based on the synchronization information received in step S500 and the integrated data received in step S502, the reproduction start time T2 corresponding to the reproduction audio of the video corresponding to the synchronization marker included in the synchronization information in the audio reproduction device 7 is calculated. Then, the process proceeds to step S508.
In step S508, the video information acquisition unit 72 determines whether or not video information has been acquired. If it is determined that the video information has been acquired (Yes), the process proceeds to step S510; Continue processing.

When the process proceeds to step S510, the synchronization marker detection unit 71 performs a synchronization marker detection process, and the process proceeds to step S512.
In step S512, it is determined whether a synchronization marker is detected based on either the detection success notification or the detection failure notification from the synchronization marker detection unit 71. If it is determined that the synchronization marker is detected (Yes), step S514 is performed. If not (No), the process proceeds to step S520.

When the process proceeds to step S514, the synchronization deviation calculation unit 70c determines the image at the viewer's seat based on the reproduction start time T2 calculated by the synchronization timing calculation unit 70b and the arrival time T1 calculated by the synchronization marker detection unit 71. The shift dT from the synchronization timing with the sound is calculated, and the process proceeds to step S516.
In step S516, the reproduction circuit / delay circuit 70d determines whether or not there is a deviation from the synchronization timing based on the notification result from the synchronization deviation calculation unit 70c, and when it is determined that there is a deviation (Yes). The process proceeds to step S518, and if not (No), the process proceeds to step S520. In the present embodiment, when a notification of erroneous detection is received, the synchronization marker is excluded from the correction target.

When the process proceeds to step S518, the reproduction circuit / delay circuit 70d corrects the video reproduction timing, and the process proceeds to step S520. Here, in the correction of the reproduction timing, if the deviation dT is dT> 0, the audio reproduction speed is made faster than the current speed, and if dT ≦ 0, the audio reproduction speed is made slower than the current speed.
In step S520, the reproduction circuit / delay circuit 70d reproduces the video based on the audio data included in the integrated data, and the process proceeds to step S522. At this time, when the reproduction speed is changed, the sound is reproduced at the reproduction speed changed for the calculated time.

In step S522, the playback circuit / delay circuit 70d determines whether or not the playback of the sound has ended. If it is determined that the playback has ended (Yes), the process proceeds to step S500; The process proceeds to S508.
That is, when the audio reproduction is started in the audio reproduction device 7, the processes from step S508 to step S522 are repeated until the audio reproduction is completed.

Next, based on FIG. 14, the synchronous marker detection process of step S510 will be described in detail.
FIG. 14 is a flowchart showing the synchronization marker detection process in the audio playback device 7.
The synchronization marker detection process is a process for detecting the synchronization marker from the video information based on the synchronization information and the video information, and calculating the arrival time of the detected synchronization marker. As shown in FIG. 6, first, the process proceeds to step S600.

In step S600, the synchronization marker detection unit 71 extracts a screen area in which the display content of the video data included in the integrated data is displayed from the video information acquired from the video information acquisition unit 72, and the process proceeds to step S602. That is, only the portion corresponding to the display content of the video data is extracted from the video including the frame portion of the screen 5 other than the display content of the video data.
In step S602, the synchronization marker detection unit 71 divides the screen area extracted in step S600 into four, calculates the average value of the luminance values of each area based on the luminance value of each divided area, and proceeds to step S604. Transition. That is, the average luminance value is extracted from the video information as a feature amount.

In step S604, the result extracted in step S602 is compared with the synchronization marker (feature parameter) included in the synchronization information acquired from the synchronization timing calculation unit 70b, and the process proceeds to step S606. In the present embodiment, the feature parameter included in the metadata is compared with the extraction result (luminance average value).
In step S606, the synchronization marker detection unit 71 determines whether or not the feature amounts match depending on whether or not the extraction result and the feature parameter are within a predetermined error range. If (Yes), the process proceeds to step S608. If not (No), the process proceeds to step S610.

  When the process proceeds to step S608, the synchronization marker detection unit 71 supplies the clock information from the clock generation circuit 70a with the start time as the arrival time T1, assuming that the video information acquired from the video information acquisition unit 72 includes the synchronization marker. The calculation result is transmitted to the synchronization deviation calculation unit 70c together with the detection success notification, and the series of processes is terminated and the process returns to the original process.

On the other hand, when the process proceeds to step S610, the synchronization marker detection unit 71 transmits a detection failure notification to the synchronization deviation calculation unit 70c on the assumption that the synchronization information is not included in the video information acquired from the video information acquisition unit 72. Then, the series of processes is terminated and the process returns to the original process.
Here, the synchronization deviation calculation process in step S514 and the reproduction timing correction process in step S518 are the synchronization deviation calculation process in step S118 and the reproduction timing correction process in step S118 in the flowchart of FIG. 3 in the first embodiment. Since this is the same process as in FIG.

Next, the operation of the present embodiment will be described with reference to FIG.
Here, FIGS. 15A to 15C are diagrams showing a series of flows until the synchronization marker is detected from the video information.
First, the integrated data transmission device 2 transmits synchronization information to the audio reproduction device 7, and then transmits integrated data corresponding to the synchronization information to the audio reproduction device 7 and the video reproduction device 8.

  When the audio reproducing device 7 receives the synchronization information via the data communication unit 75 and stores it in a storage unit (not shown) (step S500) and receives the integrated data corresponding to the synchronization information (step S502), the integrated data Is reproduced by the reproduction circuit / delay circuit 70d based on the audio data included in the (step S504). In the present embodiment, the playback start timing of video and audio is in accordance with an instruction from the user of the audio playback device 7, and information on this playback start timing is also transmitted to the video playback device 8. Therefore, the audio reproduction device 7 and the video reproduction device 8 start reproduction at the time when the instruction is given in accordance with the instruction from the user. Note that the audio reproduction device 7 transmits the time at which audio reproduction is started to the synchronization timing calculation unit 70b. When audio is reproduced by the reproduction circuit / delay circuit 70d, the speakers 73 and 74 convert the reproduced audio signals into sound waves and output them.

  In the synchronization timing calculation unit 70b of the audio reproduction device 7, when the reproduction of the audio is started by the reproduction circuit / delay circuit 70d and the audio reproduction start time is acquired from the reproduction circuit / delay circuit 70d, the reproduction start time Based on the playback time of the video corresponding to the synchronization marker included in the synchronization information, the playback start time T2 in real time corresponding to the audio played back by the audio playback device 7 of the video corresponding to the synchronization marker is calculated. . The sound reproduction start time T2 corresponding to the synchronization marker is described between the tag <time> and the tag </ time> in the metadata, as in the first embodiment. Accordingly, the real-time reproduction start time T2 of the video corresponding to the synchronization marker is calculated as the audio reproduction start time added to the reproduction time of the video corresponding to the synchronization marker. The start time T2 calculated in this way is configured as a data table in which the start time T2 and the identification number of the synchronization marker are associated with each other, as in the first embodiment, and is stored in a storage unit (not shown). Remembered.

On the other hand, when receiving the integrated data from the integrated data transmission device 2, the video playback device 8 starts playback of the video based on the information of the playback start timing from the audio playback device 7. Then, the reproduced video is optically projected and displayed on the screen 5.
When the video and audio are reproduced, the video displayed on the screen 5 is photographed by the digital video camera by the video information acquisition unit 72 disposed in the viewer's seat located away from the screen 5 and the speakers 73 and 74. (Step S508). When the video information acquisition unit 72 acquires the video information, the synchronization marker detection unit 71 starts a synchronization marker detection process (step S510).

  First, the synchronization marker detection unit 71 extracts the video in the screen area from the video information acquired from the video information acquisition unit 72 (step S600). In the video extraction of the screen area, first, a plurality of frames of the video are compared, and the part where the video content is changed in the predetermined area (the end part excluding the central part of the screen) is the frame part of the screen 5 or the like It is determined that the area is a peripheral area of the image, and the other area is determined as the screen area. Such comparison processing needs to be performed only once, and once the screen area is understood, only the necessary video portion can be extracted from the video information using the same area range. By this extraction process, an image as shown in FIG. 15A is extracted.

When the video of the screen area is extracted from the video information, the video of the screen area is divided into four equal parts as shown in FIG. 15B, and each video in the area is divided based on the video data in each divided area. The average value of the luminance values in the divided area is calculated from the luminance values of the pixels (step S602). The average value of the calculated luminance values becomes the feature amount of each video information.
When the feature quantity is extracted from the video information, the extracted feature quantity is compared with a synchronization marker having the same feature parameter as the extracted feature quantity as shown in FIG. 15C (step S604). ), When the difference between the two is less than or equal to a predetermined value (for example, 8 or less), it is determined that the two match, and when the difference is greater than the predetermined value, it is determined that they do not match (step S606). . Here, as in the first embodiment, the synchronization marker is described as metadata together with other synchronization information. For each synchronization marker of each identification number, a <data id> tag and a </ data> Characteristic parameter values are described between the tags.

  When the synchronization marker of each identification number is compared with the extracted feature quantity and it is determined that they match, the matched data portion in the video data is based on the clock signal supplied from the clock generation circuit 70a. Is calculated as arrival time T1. The calculated arrival time T1 and the corresponding identification number (Id) are transmitted to the synchronization deviation calculation unit 70c together with information indicating successful detection (step S608).

On the other hand, the synchronization marker of each identification number is compared with the extracted feature amount, and if it is determined that there is no match between them, information indicating a detection failure is transmitted to the synchronization deviation calculation unit 70c (step) S610).
As in the first embodiment, the synchronization deviation calculation unit 70c is synchronized based on the table of the start time T2 stored in the storage unit, the arrival time T1 and the identification number acquired from the synchronization marker detection unit 71. A synchronization shift dT (= T1-T2) is calculated for each marker identification number (step S302). Then, when dT is calculated, the absolute value of the calculated dT is compared with a predetermined value (here, 10 [s] (“00: 00: 10: 00”)) (step S304). Based on this comparison result, when the absolute value of dT is 10 [s] or more, the detected synchronization marker is determined to be an erroneously detected synchronization marker, and information indicating the erroneous detection is transmitted to the reproduction circuit / delay circuit 70d. (Step S306). On the other hand, if the absolute value of dT is less than 10 [s], the calculated dT is transmitted to the reproduction circuit / delay circuit 30d (step S308).

The reproduction circuit / delay circuit 30d determines whether or not there is a synchronization deviation based on the notification result from the synchronization deviation calculation unit 70c (step S516). If it is determined that there is a synchronization deviation, the reproduction circuit / delay circuit 30d. In the circuit 70d, the video reproduction timing correction process is started (step S518).
Similarly to the reproduction circuit / delay circuit 30d in the first embodiment, the reproduction circuit / delay circuit 70d determines whether or not dT acquired from the synchronization deviation calculation unit 70c is larger than “0” (step) S400). Based on the determination result, when the video corresponding to the synchronization marker has reached the viewer's seat first with respect to the audio corresponding to the synchronization marker, the amount of control is different from the control amount for increasing the audio playback speed. (Step S402), and if the video corresponding to the synchronization marker is delayed and reaches the viewer's seat with respect to the audio corresponding to the synchronization marker, A control amount for reducing the reproduction speed and a time required to eliminate the deviation are calculated (step S404).

When the control amount and the time are calculated, the playback circuit / delay circuit 30d performs a sound playback process by changing the playback speed for the calculated time based on the control amount (step S520).
In this way, by increasing or decreasing the audio reproduction speed, the deviation dT of the arrival timing of the video and audio at the viewer's seat with respect to the synchronization timing is corrected.

Note that the synchronization timing shift dT is not limited to correction by changing the playback speed. If the video is advanced, the audio is skipped by dT or if the video is delayed, It may be performed by pausing the playback of the sound for the number of minutes.
In this way, in the present embodiment, the video information displayed on the screen 5 at a position away from the viewer's seat is acquired at the viewer's seat, and a preset synchronization marker is detected from the acquired video information. Thus, the arrival time T1 of the video is calculated, and from this arrival time T1 and the actual reproduction start time T2 of the video corresponding to the synchronization marker, a shift dT between the two with respect to the synchronization timing is calculated. Based on the deviation dT, it is possible to correct the deviation dT by changing the audio reproduction speed in the audio reproduction device 7. Therefore, it is possible to correct the deviation of the arrival timing of the video and audio at the viewer's seat with respect to the synchronization timing.

  In the second embodiment, the process of storing the synchronization information from the integrated data transmitting apparatus 2 and storing it in a storage unit (not shown) is the first, third, fifth, sixth, seventh, nineteenth, twenty-first, twenty-third, twenty-third, twenty-fourth, twenty-sixth, and twenty-sixth. , 30, 32, 34, 35, and 37, a series of calculating synchronization deviation dT performed by the synchronization timing calculation unit 70b, the synchronization deviation calculation unit 70c, and the synchronization marker detection unit 71, corresponding to any one of the synchronization information storage means. This processing corresponds to the synchronization deviation calculation means of any one of the inventions 1, 3, 5, 6 and 8, and the video information acquisition unit 72 corresponds to the output information acquisition means of the inventions 1, 3, 5, 6 and 8. Correspondingly, the sound reproduction speed correction processing in the reproduction circuit / delay circuit 70d corresponds to the output timing control means of the invention 1 or 14.

In the second embodiment, the process of receiving synchronization information via the data communication unit 75 corresponds to the synchronization information receiving means of the invention 14 and receives integrated data via the data communication unit 75. The processing corresponds to the integrated data receiving means of the fourteenth aspect.
In the second embodiment, step S500 corresponds to the synchronization information receiving step of the invention 27 or 38, step S502 corresponds to the integrated data receiving step of the invention 28 or 39, and steps S510 and Step S514 corresponds to the synchronization deviation calculating step of any one of Inventions 19, 21, 23, 24, 26, 30, 32, 34, 35 and 37, and Steps S516 and S518 are those of Invention 19 or Invention 30. Corresponds to the output timing control step.

In the second embodiment, as the feature amount, the screen area is divided into four and the average luminance value of each divided area is obtained. However, the present invention is not limited to this, and the average color is used instead of the average luminance. Other values may be obtained as feature values, such as obtaining the average luminance or average color of the entire screen area.
In the above embodiment, the synchronization marker is a part of the video data that is actually displayed on the screen and perceived by the viewer. However, the present invention is not limited to this, and the red color outside visible light (400 to 700 nm). An external signal may be superimposed on the video signal, and this signal may be detected as a synchronization marker. In this case, signal start, signal end, wavelength change (color change), and luminance change can all be handled as synchronization markers. Also, one frame of the video is used as a screen indicating synchronization (for example, a predetermined pattern such as a color test pattern). If it is one frame, it is unlikely that the viewer explicitly perceives it like the subliminal effect. That is, the inserted frame can be handled as a synchronization marker.

  In the above embodiment, a part of the video data desired to be shown to the viewer is used as the synchronization marker. However, the present invention is not limited to this, and a video for synchronizing with the video desired to be shown to the viewer may be added. good. For example, a graphic of a predetermined shape and color is displayed at a predetermined position of an image such as a punch mark indicating a film exchange in a movie (a black circle is displayed on the upper right of the screen), and this is detected as a synchronization marker. That is, a punch mark can be handled as a synchronization marker.

[Third Embodiment]
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. 16 to 20 show an output timing control device, an output timing control system, an output device, a video output device, an integrated data providing device, an output timing control program, an output device control program, an integrated data providing device control program according to the present invention, It is a figure which shows 3rd Embodiment of an output timing control apparatus control method, an output apparatus control method, and an integrated data provision apparatus control method.

  In the present embodiment, video and audio are output from separate output devices based on integrated data consisting of video data and audio data that constitute an output result of a movie or the like that integrates video and audio. In the display system, the output timing of video and audio from each output device is controlled so that these arrival timings are synchronized in the audience seat located at a position away from the speaker that outputs audio and the screen that displays the video. .

The difference from the first embodiment and the second embodiment is that an audio signal is used without using a synchronization marker to obtain a deviation dT with respect to the synchronization timing of the arrival timing of audio and video at the viewer's seat. This is in that a signal having a frequency outside the audible band is used.
First, the configuration of the video display system 9 according to the present invention will be described with reference to FIG. FIG. 16 is an overview of the video display system 9 according to the present invention.

  As shown in FIG. 16, the video display system 9 transmits the audio data to the audio playback device 4 to be described later, and transmits the video data to the video playback device 10 among the integrated data including the video data and the audio data. From the transmission apparatus 2, the audio reproduction apparatus 4 that reproduces audio based on the audio data received from the integrated data transmission apparatus 2, and outputs the reproduced audio from speakers 4 a and 4 b, which will be described later, and the integrated data transmission apparatus 2. A video is reproduced based on the received integrated data, the video is optically projected onto the screen 5, and the integrated data and audio information acquired by the audio information acquisition unit 120 provided in the viewer's seat described later are used. Based on this, the video playback device 10 that synchronizes the arrival timing of audio and video at the viewer's seat, and the video light projected from the video playback device 10 And Shimesuru screen 5, and has a configuration that includes a.

  In the present embodiment, the distance between the speakers 4a and 4b and the viewer's seat where the viewer is located is greatly separated. Therefore, the time for the video to reach the viewer's seat and the time for the sound to reach the viewer's seat. And there is a difference. In other words, since the sound arrives later than the image viewed by the viewer, there is a difference between the image and the sound. Therefore, in the video reproduction device 10 according to the present embodiment, the audio information acquisition unit 120 acquires the audio information that is output from the speakers 4a and 4b of the audio reproduction device 4 and reaches the viewer's seat. By controlling the video output timing based on the audio information, the video arrival timing and the audio arrival timing reaching the viewer's seat are synchronized.

  Further, the integrated data transmission device 2 uses, as audio data to be transmitted to the audio reproduction device 4, data of a signal in which a synchronization signal having a frequency outside the audible frequency band (hereinafter referred to as a synchronization superimposed signal) is superimposed on the audio signal. Generate. The synchronization superposition signal is a modulation signal that switches to another frequency at a predetermined period, and information on the modulation period and information on the initial phase are transmitted to the video reproduction apparatus 10 as synchronization information.

Further, a detailed configuration of the video reproduction apparatus 10 will be described with reference to FIG. FIG. 17 is a block diagram illustrating a detailed configuration of the video reproduction device 10.
As shown in FIG. 17, the video reproduction device 10 includes a video reproduction unit 100, a synchronization deviation detection unit 110, an audio information acquisition unit 120, a video projection unit 130, and a data communication unit 140. ing.

The video reproduction unit 100 includes a clock generation circuit 100a and a reproduction circuit / delay circuit 100b.
The clock generation circuit 100a includes a crystal oscillator or the like, and has a function of generating a clock signal having a predetermined period. Then, the generated clock signal is supplied to a reproduction circuit / delay circuit 100b, a synchronization signal generation circuit 110a, a phase detection unit 110b, and a frequency analysis unit 110c, which will be described later.

The reproduction circuit / delay circuit 100b reproduces the video based on the video data received from the integrated data transmission device 2 via the data communication unit 140, and at the viewer's seat of the video and audio received from the synchronization shift detection unit 110. The video playback speed is corrected based on the deviation dT of the arrival timing with respect to the synchronization timing.
The synchronization shift detection unit 110 includes a synchronization signal generation circuit 110a, a phase difference detection unit 110b, a frequency analysis unit 110c, and a band pass filter 110d.

The synchronization signal generation circuit 110a generates, from the synchronization information from the integrated data transmission device 2, a synchronization signal having the same modulation period as the modulation period a of the synchronization superimposed signal superimposed on the audio data and the initial phase c. .
The phase difference detection unit 110b detects a phase difference between the synchronization superimposed signal and the synchronization signal generated in the synchronization signal generation circuit 110a based on the frequency analysis result of the synchronization superimposed signal. In the present embodiment, the phase difference between the peak frequency of the synchronization superimposed signal and the peak frequency of the synchronization signal at time t is detected. That is, this phase difference is the same as the shift dT with respect to the synchronization timing of the arrival timing of the audio and the video at the viewer's seat in the first and second embodiments. The detected phase difference dT is transmitted to the reproduction circuit / delay circuit 100b.

The frequency analysis unit 110c performs A / D conversion on the synchronization superimposed signal extracted from the audio signal in the bandpass filter 110d, performs FFT processing on the signal after the A / D conversion, and peaks based on the FFT processing result. Find the frequency.
The band-pass filter 110d is a band filter that extracts a synchronization superimposed signal included in the audio signal from the audio signal acquired by the audio information acquisition unit 120.

Here, since the audio information acquisition unit 120, the video projection unit 130, and the data communication unit 140 are the same as the audio information acquisition unit 32, the video projection unit 33, and the data communication unit 34 in the first embodiment, description will be given. Is omitted.
Here, the video reproducing apparatus 10 includes a processor, a RAM (Random Access Memory), and a ROM (Read Only Memory) in which a dedicated program is stored, which is not shown, and executes the dedicated program by the processor. This fulfills the function of each of the above components. In addition, each of the above-described components includes a component that fulfills its function only by a dedicated program, a component that controls its hardware by a dedicated program, and fulfills its function.

Further, the video reproduction apparatus 10 starts a predetermined program stored in a predetermined area of the ROM by a processor, and executes a reproduction timing control process shown in the flowchart of FIG. 18 according to the program.
FIG. 18 is a flowchart showing playback timing control processing in the video playback device 10.

When the reproduction timing control process is executed by the processor, first, as shown in FIG. 18, the process proceeds to step S700.
In step S700, the data communication unit 140 determines whether or not the synchronization information has been received from the integrated data transmission device 2. If it is determined that the synchronization information has been received (Yes), the synchronization information is stored in a storage unit (not shown). The process proceeds to step S702. If not (No), the process waits until reception.

If the process proceeds to step S702, it is determined whether the video data corresponding to the synchronization information received in step S700 is received via the data communication unit 140. If it is determined that the video data is received (Yes), step S704 is performed. If not (No), wait until receiving.
When the process proceeds to step S704, the reproduction circuit / delay circuit 100b reproduces the video based on the video data received in step S702, and the process proceeds to step S706.

In step S706, it is determined whether or not audio information has been acquired by the audio information acquisition unit 120. If it is determined that the audio information has been acquired (Yes), the process proceeds to step S708, and if not (No), determination is made until acquisition. Continue processing.
When the process proceeds to step S708, the synchronization deviation detection unit 110 performs a synchronization deviation detection process, and the process proceeds to step S710.

In step S710, it is determined whether or not the detection from the synchronization deviation detection unit 110 has been acquired. If it is determined that the detection has been acquired (Yes), the process proceeds to step S712. To do.
When the process proceeds to step S712, the reproduction circuit / delay circuit 100b determines whether or not there is a synchronization shift based on the detection result acquired from the synchronization shift detection unit 110, and when it is determined that there is (Yes). The process proceeds to step S714, and if not (No), the process proceeds to step S716.

In step S714, the reproduction circuit / delay circuit 100b corrects the reproduction timing of the video, and the process proceeds to step S716. Here, the correction of the playback timing is such that if the deviation dT is dT> 0, the video playback speed is faster than the current speed, and if dT ≦ 0, the video playback speed is slower than the current speed.
In step S716, the reproduction circuit / delay circuit 100b reproduces the video based on the video data, and the process proceeds to step S718. At this time, if the playback speed is changed, the video is played back at the changed playback speed.

In step S718, the reproduction circuit / delay circuit 100b determines whether or not the reproduction of the video has ended. If it is determined that the reproduction has ended (Yes), the process proceeds to step S700; The process moves to S706.
That is, when video playback is started in the video playback device 3, the processing from step S706 to step S718 is repeatedly performed until video playback ends.

Next, based on FIG. 19, the synchronization deviation detection processing in step S708 will be described in detail.
FIG. 19 is a flowchart showing synchronization deviation detection processing in the video reproduction apparatus 10.
The synchronization shift detection process is a process of detecting a shift dT with respect to the synchronization timing of the arrival timing of the audio and video at the viewer's seat based on the synchronization superimposed signal superimposed on the audio information, and is executed in step S708. First, as shown in FIG. 19, the process proceeds to step S800.

In step S800, the band-pass filter 110d extracts a synchronization superimposed signal from the audio signal acquired from the audio information acquisition unit 120, and the process proceeds to step S802.
In step S802, the frequency analysis unit 110c performs A / D conversion on the synchronization superimposed signal extracted in step S800, and the process proceeds to step S804.
In step S804, the frequency analysis unit 110c performs FFT processing on the result of A / D conversion in step S802, and the process proceeds to step S806.

In step S806, the frequency analysis unit 110c detects the peak frequency of the synchronization superimposed signal based on the result of the FFT processing in step S804, and the process proceeds to step S808.
In step S808, the phase difference detection unit 110b detects the phase difference dT [s] between the synchronization signal and the synchronization superimposed signal from the synchronization signal from the synchronization signal generation circuit 110a and the analysis result from the frequency analysis unit 110c. Then, the process proceeds to step S810.

In step S810, the phase detection unit 110b transmits the detection result of the phase difference dT to the reproduction circuit / delay circuit 100b, ends all processing, and returns to the original processing.
Next, the operation of the present embodiment will be described based on FIG.
Here, FIG. 20 is a diagram showing a series of flows until a synchronization shift is detected from the audio information.

  First, the integrated data transmission device 2 transmits synchronization information to the video reproduction device 10, and then transmits video data corresponding to the synchronization information to the video reproduction device 10, while audio data corresponding to the synchronization information is transmitted to the audio reproduction device 10. Transmit to the playback device 4. Here, the audio data is data of a signal in which a superimposed signal for synchronization having a frequency outside the audible frequency band is superimposed on an audio signal corresponding to video. In the present embodiment, the synchronization superimposed signal is regularly changed to several kinds of frequencies in a predetermined cycle, and is a stationary signal in which the same modulation cycle is repeated in a long time. .

  The video reproduction device 10 receives the synchronization information via the data communication unit 140, stores it in a storage unit (not shown) (step S700), and receives video data corresponding to the synchronization information (step S702). Based on the above, the video is reproduced by the reproduction circuit / delay circuit 100b (step S704). In the present embodiment, the playback start timing of video and audio is the timing according to an instruction from the user of the video playback device 10, and this playback start timing information is also transmitted to the audio playback device 4. Therefore, the video reproduction device 10 and the audio reproduction device 4 start reproduction at the time when the instruction is given in accordance with the instruction from the user.

When the video is reproduced by the reproduction circuit / delay circuit 100b, the video projection unit 130 enlarges and projects the reproduced video on the screen 5.
On the other hand, when the audio reproducing device 4 receives the audio data from the integrated data transmitting device 2, the audio reproducing device 4 starts reproducing the audio based on the information of the reproduction start timing from the video reproducing device 10. And the reproduced | regenerated audio | voice is output from the speakers 4a and 4b.

  When the video and audio are reproduced, the audio output from the speakers 4a and 4b arrives at the audio information acquisition unit 120 disposed in the viewer's seat located away from the speakers 4a and 4b. Thereby, the voice information acquisition unit 120 acquires the voice information that has arrived via the microphone (step S706). When the audio information is acquired by the audio information acquisition unit 120, the synchronization error detection unit 110 starts the synchronization error detection process (step S708).

First, the synchronization shift detection unit 110 uses the band-pass filter 110d to generate a synchronization superimposed signal as shown in FIG. 20B from the audio signal as shown in FIG. 20A acquired from the audio information acquisition unit 120. Extract (step S800). That is, the band of the bandpass filter is set so that a signal in a frequency band outside the preset audible frequency band can be extracted. As shown in FIG. 20 (b), it can be seen that the frequency of the synchronization superimposed signal is not constant but has changed. Next, the frequency analysis unit 110c performs A / D conversion on the synchronization superimposed signal extracted by the band pass filter 110d (step S802), and further performs FFT processing on the converted synchronization superimposed signal (step S802). In step S804, the peak frequency is detected based on the FFT processing result (see FIG. 20C) (step S806). Here, the size of the window used in the FFT process is set in accordance with the modulation period of the synchronization superimposition signal. Further, the peak frequency of the synchronization superimposed signal at time t can be expressed as the following formula (1).

f (t) = sin (a · t + b) (1)

However, a is a modulation period, b is a phase when an audio | voice signal arrives at a viewer's seat.

When the peak frequency is detected from the synchronization superimposition signal, the phase difference detection unit 110b generates the synchronization superimposition signal peak frequency (see FIG. 20C) and the synchronization signal generation as shown in FIG. A phase difference dT with respect to the peak frequency of the synchronization signal generated from the circuit 110a is detected (step S808).
The synchronization signal generation circuit 110a is a clock supplied from the clock generation circuit 100a of the video reproduction unit 100 based on the synchronization information from the integrated data transmission device 2 and based on the modulation period a and the initial phase c of the synchronization superimposed signal. In synchronization with the signal, a synchronization signal having a modulation period a and an initial phase c is generated. Therefore, the peak frequency of the synchronization signal at time t can be expressed as the following equation (2).

f0 (t) = sin (a · t + c) (2)

That is, the phase difference detection unit 110b detects the phase difference between the peak frequency of the synchronization signal and the peak frequency of the synchronization superimposed signal at time t when the audio signal reaches the viewer's seat. Here, for detection of the phase difference, for example, a known phase detector used in a PLL is used.

When the phase difference dT is detected, the phase difference detection unit 110b transmits the detection result to the reproduction circuit / delay circuit 100b (step S810).
The reproduction circuit / delay circuit 100b determines whether or not there is a synchronization shift based on the notification result from the synchronization shift detection unit 110 (step S712). Here, whether or not there is a synchronization shift is determined by determining that the phase difference dT acquired from the synchronization shift detection unit 110 is dT = 0 (in this embodiment, “00 (h): 00 (m): 00 ( s): 00 ”), it is determined that there is no synchronization shift, and when dT ≠ 0, it is determined that there is a synchronization shift.

If it is determined that there is a synchronization error, the reproduction circuit / delay circuit 100b starts a process for correcting the reproduction timing of the video (step S714).
Since the correction timing correction process is the same as that in the first embodiment, description thereof is omitted.
Thus, in the present embodiment, audio information from the speakers 4a and 4b located away from the viewer's seat is acquired at the viewer's seat, and the audible frequency band superimposed in advance from the acquired audio information. Extracting a synchronization superposition signal composed of an external frequency, detecting the peak frequency of the extracted synchronization superposition signal, and detecting the peak frequency and the peak of the synchronization signal having the same modulation period and initial phase as the synchronization superposition signal It is possible to correct the phase difference dT by detecting the phase difference dT from the frequency and changing the video playback speed in the video playback device 10 based on the detected phase difference dT. Therefore, it is possible to correct the deviation of the arrival timing of the video and audio at the viewer's seat with respect to the synchronization timing.

  In the third embodiment, the process of storing the synchronization information from the integrated data transmission device 2 and storing it in the storage unit (not shown) is the first, fifth, sixth, eighth, nineteenth, twenty-third, twenty-fourth, twenty-sixth, thirty-fourth, thirty-fourth. , 35 and 36, a series of processes for calculating the synchronization shift dT performed by the synchronization shift detection unit 110 corresponding to the synchronization information storage means of any one of the fifth, sixth and eighth inventions. The audio information acquisition unit 120 corresponds to the output information acquisition means of the inventions 4, 5, 6 and 8, and the video playback speed correction processing in the playback circuit / delay circuit 100b is the output timing of the invention 14. Corresponds to the control means.

In the third embodiment, the process of receiving synchronization information via the data communication unit 140 corresponds to the synchronization information receiving means of the invention 14, and the process of receiving video data via the data communication unit 140 is as follows. This corresponds to the integrated data receiving means of the fourteenth aspect.
In the third embodiment, step S700 corresponds to the synchronization information receiving step of the invention 27 or 38, step S702 corresponds to the integrated data receiving step of the invention 28 or 39, and step S708 is , 23, 24, 26, 34, 35, and 37 correspond to any one of the synchronization deviation calculation steps, and steps S712 and S714 correspond to the output timing control steps of the invention 19 or the invention 30, respectively.

  In the third embodiment, a part of the audio data in the audible band that can be heard by the viewer is used as the synchronization marker. However, the present invention is not limited to this, and the audio data is outside the audible band (20 to 20 kHz). These data may be superimposed on the audio signal, and this signal may be detected as a synchronization marker. In this case, signal start, signal end, frequency change, and volume change can all be handled as synchronization markers.

  In the third embodiment, the example in which the synchronization shift (phase difference) dT is obtained using the audio signal outside the audible frequency band has been described. However, the present invention is not limited to this. Light such as infrared light is continuously flowed while shifting its luminance at a predetermined cycle, and a phase shift of luminance modulation is detected as a synchronization shift, or a luminance of a specific color (for example, black) portion in an image is detected at a predetermined cycle. Another signal may be used depending on the configuration of the system, such as detecting a phase shift as a synchronization shift by continuing to flow while shifting.

In addition to the system configuration examples described in the first to third embodiments, the present invention can be applied to other system configurations shown in the following (1) to (4). is there.
(1) A device close to a device far from a sound source FIG. 21 shows a system configuration when the present invention is applied to a system in which each video display device has a different distance from the sound source when there are a plurality of video display devices. It is a general-view figure which shows an example.

  As shown in FIG. 21, when a performance is performed on a stage and an image in accordance with the performance is displayed on a display (video playback device) at the position of each spectator located away from the stage, a stage or a speaker is used. Depending on the distance from the image, there is a difference in the timing at which audio can be heard for the video displayed on the video playback device. In such a case, it is impossible to synchronize the synchronization timing of video and audio in the integrated data transmission device, and it is necessary to synchronize the synchronization in each video reproduction device. Even in such a system, by providing the video playback device in the hand of the audience with functions equivalent to those of the video playback device 3 or the video playback device 10 in the first or third embodiment, performance and video can be obtained. Can be synchronized. In particular, since each video reproduction device synchronizes autonomously, even when there are a plurality of video reproduction devices, it is possible to achieve synchronization without increasing processing.

(2) Home theater (synchronization of voice and voice)
FIG. 22 is an overview diagram showing a system configuration example when the present invention is applied to a system such as a home theater.
In the case of a system such as a home theater composed of a plurality of audio channels, as shown in FIG. 22, the synchronization and synchronization of audio between the speakers with respect to the position of the listener also occurs due to the arrangement and sound of the speakers corresponding to each channel. . The present invention is also effective in such a case.

  For example, an audio having the same function as the audio reproduction device 7 in the second embodiment and capable of correcting the audio reproduction timing for each speaker or for each group such as a front speaker and a rear speaker. A system is configured by the playback device. As a voice synchronization marker, a part of voice data that only sounds from a speaker or a group of speakers to be synchronized is used. As shown in this example or the system of FIG. 23, transmission of integrated data is not necessarily an essential component. Here, FIG. 23 is a diagram illustrating an example of a video display system having a configuration in which an apparatus having integrated data is directly connected to a video playback apparatus and an audio playback apparatus.

(3) When the transmission path is different (synchronization between video and video)
FIG. 24 is an overview diagram showing a system configuration example when the present invention is applied to a system having a plurality of types of integrated data transmission paths.
This is a case where the images sent from the two transmission devices are synchronized. For example, a case where a video sent from overseas and a video sent from within Japan are shown in synchronization. Here, the synchronization processing method is the same as that of the second embodiment.

  As shown in FIG. 24, even if two integrated data transmission apparatuses 1 and 2 synchronize and send contents, one is via a communication satellite and the other is via a line such as a LAN. As described above, when the transmission paths are different from each other, when the screen is viewed from the viewer's position, a synchronization shift occurs between the two images displayed on the screen. Conventionally, the delay due to the transmission path is measured or predicted in advance, and the transmission timing from the transmission apparatus is shifted by that amount, or the reproduction apparatus delays the transmission timing. However, due to the stability of the transmission path, the transmission delay is not necessarily constant, and there may be a difference in delay between measurement and reproduction, or a change in transmission delay during reproduction. By applying the present invention, it is possible to synchronize with the video being played back, so that it is possible to achieve reliable real-time synchronization that has not been possible in the past.

  In the example of FIG. 24, the case where there are two integrated data transmission apparatuses and the transmission paths are different from each other is shown. However, when there are one integrated data transmission apparatus and two types of transmission paths, there are two integrated data transmission apparatuses. The same applies when there is only one type of transmission path. Furthermore, even if there is one transmission path with the integrated data transmission device, the synchronization loss may occur due to factors such as the capability of the playback device and the complexity of the content (difficulty of decoding). Even if there is, there is a possibility that a synchronization error occurs.

The video output terminal can also be used when the projector connected to the integrated data transmission device via a wired network and the video output terminal connected to the viewer via the wireless LAN are synchronized. It is effective to give the function of the present invention to the above.
(4) Playback content other than video and audio Up to now, video and audio have been described as examples of playback output. However, a playback device that outputs output content perceived by humans along the time axis. The present invention is useful for any apparatus.

FIG. 25 is an overview diagram showing a system configuration example in the case where the present invention is applied to a system for producing a scent in synchronization with an image.
For example, as shown in FIG. 25, in a system that emits a scent in synchronization with an image, the transmission state of the scent from the scent generator (scent regenerating device in the figure) to the user is Since it easily changes due to various factors such as temperature and humidity, and the arrival timing is shifted, in this case, it is possible to apply the present invention to synchronize the arrival timing of the scent with the video. In the example of FIG. 25, the information on the scent that has reached the position where the user is located is acquired by the olfactory monitor means, and based on this information, the video playback speed is controlled and synchronized. In addition, the present invention is useful for other playback contents such as when dry ice is used to produce an effect that generates fog in synchronization with the display image.

  In the first to third embodiments, the function of detecting a deviation from the synchronization timing of the video and audio at the viewer's seat and calculating a control amount for correcting the synchronization deviation is provided. Although an example of a playback device or an audio playback device has been described, the present invention is not limited to this, and is separate from an output device such as a video device or an audio device. An apparatus having a function of calculating a control amount for correcting the synchronization shift may be provided.

  In the first to third embodiments, the control shown in FIG. 3 to FIG. 6, FIG. 13, FIG. 14, FIG. 18, FIG. Although the case where the program is executed has been described, the present invention is not limited thereto, and the program may be read from the storage medium storing the program showing these procedures and executed.

  Here, the storage medium is a semiconductor storage medium such as RAM or ROM, a magnetic storage type storage medium such as FD or HD, an optical reading type storage medium such as CD, CDV, LD, or DVD, or a magnetic storage type such as MO. / Optical reading type storage media, including any storage media that can be read by a computer regardless of electronic, magnetic, optical, or other reading methods.

1 is an overview diagram of a video display system 1 according to the present invention. 3 is a block diagram showing a detailed configuration of a video reproduction device 3. FIG. 4 is a flowchart showing a playback timing control process in the video playback device 3; 6 is a flowchart showing a synchronization marker detection process in the video playback device 3; 5 is a flowchart showing a synchronization deviation calculation process in the video reproduction device 3; 4 is a flowchart showing a playback timing correction process in the video playback device 3; It is a figure which shows the transmission state of synchronous information and integrated data (audio | voice data and video data). It is a figure which shows a series of flows until it detects a synchronous marker from audio | voice information. It is a figure which shows the example of a description of metadata. It is a figure which shows the example of a table of the reproduction | regeneration start time of the audio | voice corresponding to a synchronous marker. 1 is an overview of a video display system 6 according to the present invention. 3 is a block diagram showing a detailed configuration of an audio reproduction device 7. FIG. 6 is a flowchart showing a playback timing control process in the audio playback device 7; 7 is a flowchart showing synchronization marker detection processing in the audio playback device 7; (A)-(c) is a figure which shows a series of flows until a synchronous marker is detected from video information. 1 is an overview of a video display system 9 according to the present invention. 2 is a block diagram showing a detailed configuration of a video playback device 10. FIG. 4 is a flowchart showing a playback timing control process in the video playback device 10. 4 is a flowchart showing a synchronization shift detection process in the video playback device 10. It is a figure which shows a series of flows until audio | voice synchronization detection is detected from audio | voice information. It is a general-view figure which shows the example of a system configuration at the time of applying this invention to the system from which the distance from a sound source differs in each video display apparatus when there exist a some video display apparatus. It is a general-view figure which shows the system structural example at the time of applying this invention to a system like a home theater. It is a figure which shows an example of the video display system of the structure by which the apparatus which has integrated data was directly connected to the video reproduction apparatus and the audio | voice reproduction apparatus. It is a general-view figure which shows the example of a system structure at the time of applying this invention to a system which has multiple types of transmission paths of integrated data. It is a general-view figure which shows the system structural example at the time of applying this invention to the system which outputs a fragrance synchronizing with an image | video.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,6,9 ... Video display system, 2 ... Integrated data transmission apparatus, 3 ... Video reproduction apparatus, 30 ... Video reproduction part, 30a ... Clock generation circuit, 30b ... Synchronization timing calculation part, 30c ... Synchronization deviation calculation part, 30d ... Reproduction circuit / delay circuit 31... Synchronized marker detection unit 32. Audio information acquisition unit 33. Video projection unit 34 34 Data communication unit 4. Audio reproduction device 4 a and 4 b Speakers 5. DESCRIPTION OF SYMBOLS ... Audio reproduction apparatus, 70 ... Audio reproduction part, 70a ... Clock generation circuit, 70b ... Synchronization timing calculation part, 70c ... Synchronous deviation calculation part, 70d ... Reproduction circuit / delay circuit, 71 ... Synchronization marker detection part, 72 ... Video information Acquisition unit 73, 74 ... speaker, 75 ... data communication unit, 8 ... video playback device, 10 ... video playback device, 100 ... video playback unit, 100a ... clock generation circuit, 100b ... Raw circuit / delay circuit 110... Sync deviation detection unit 110 a... Sync signal generation circuit 110 b. Phase difference detection unit 110 c. Frequency analysis unit 110 d. Bandpass filter 120 120 Audio information acquisition unit 130 130 Video projection Part, 140 ... data communication part

Claims (26)

By outputting the output results obtained by integrating N (N is an integer of 2 or more) output elements, the N output elements are output separately from M (1 <M ≦ N) output devices, respectively. An output timing control device for controlling an output timing of the output element from the output device in a system comprising:
Synchronization information storage means for storing information related to synchronization timing when integrating the N output elements;
Output information acquisition means for acquiring information on output elements output from the M output devices at a predetermined position;
Synchronization deviation calculating means for calculating respective deviations of the arrival timings of the N output elements at the predetermined position with respect to the synchronization timing based on the acquisition result of the output information acquisition means and the stored contents of the synchronization information storage means; ,
Output timing control means for controlling the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result of the synchronization deviation calculation means An output timing control apparatus comprising: The information relating to the output element includes information relating to the arrival timing of the output element to the predetermined position,
The output according to claim 1, wherein the information related to the synchronization timing includes information related to a part of the data and information related to an output timing of the data when the output element is data. Timing control device. Information related to a part of the data is a feature amount extracted from the part of the data,
The synchronization deviation calculation unit extracts a feature amount from the acquisition result based on the acquisition result of the output information acquisition unit, and stores the feature amount, information related to the arrival timing, and storage content of the synchronization information storage unit. The output timing control device according to claim 2, wherein a deviation from the synchronization timing is calculated based on At least one of the M output devices is configured to output a periodic signal having a periodicity together with the output element in synchronization with the output timing of the output element output from the own device,
The output timing control apparatus according to claim 1, wherein the output information acquisition unit acquires the periodic signal as information relating to the output element. When the M output devices are configured to include a video output device that outputs video as the output element and an audio output device that outputs audio as the output element,
The output information acquisition means acquires information on the video and audio output from the video output device and the audio output device, respectively, at the predetermined position,
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the stored contents of the synchronization information storage means, and the arrival timing of the video output from the video output device at the predetermined position, and the audio output device 5. The output timing control device according to claim 1, wherein a deviation of the arrival timing of the sound output from the terminal with respect to the synchronization timing is calculated. When the M output devices are configured to include two or more video output devices that output video as the output elements,
The output information acquisition means acquires information relating to the video respectively output from the two or more video output devices at the predetermined position,
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the storage content of the synchronization information storage means, and the arrival timing of the video output from the two or more video output devices at the predetermined position, 6. The output timing control apparatus according to claim 1, wherein a deviation with respect to the synchronization timing is calculated. When the M output devices are configured to include two or more sound output devices that output sound as the output elements,
The output information acquisition means acquires information on the sound output from each of the two or more sound output devices at the predetermined position,
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the storage content of the synchronization information storage means, and the arrival timing of the voices output from the two or more voice output devices at the predetermined position, The output timing control apparatus according to claim 1, wherein a deviation from the synchronization timing is calculated. When the plurality of output devices are configured to include an odor output device that outputs odor as the output element,
The output information acquisition means is configured to output information related to an odor output from the odor output device and information related to an output element output from an output device different from at least one odor output device. Acquired in
The synchronization deviation calculation means is based on the acquisition result of the output information acquisition means and the stored contents of the synchronization information storage means, the arrival timing of the odor output from the odor output device at the predetermined position, the odor output device, 8. The output timing control device according to claim 1, wherein a deviation of the arrival timing of an output element output from a different output device with respect to the synchronization timing is calculated. 9. Integrated data acquisition means for acquiring integrated data comprising information relating to the N output elements;
Output means capable of outputting x (1 ≦ x <N) output elements out of the N output elements based on the integrated data acquired by the integrated data acquisition means. The output timing control device according to any one of claims 1 to 8. The output timing control device according to claim 5 or 6,
Integrated data acquisition means for acquiring integrated data including information relating to video as the output element;
A video output device comprising: video output means capable of outputting the video based on information related to the video included in the integrated data acquired by the integrated data acquisition means. An output timing control device according to any one of claims 1 to 8, and the M output devices.
The output timing control device and the M output devices are connected so that data communication is possible,
The output timing control device includes:
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device;
An output timing control system comprising: output timing correction means for correcting the output timing of the output element by the output means based on the control information received by the control information reception means. An output timing control device according to any one of claims 1 to 8, the M output devices, and an integrated data providing device,
The output timing control device, the M output devices, and the integrated data providing device are connected so that data communication is possible,
The output timing control device includes:
Synchronization information receiving means for receiving information related to the synchronization timing transmitted from the integrated data providing device,
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Integrated data receiving means for receiving integrated data transmitted from the integrated data providing device;
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements based on the integrated data received by the integrated data receiving means;
Control information receiving means for receiving control information transmitted from the output timing control device;
Output timing correction means for correcting the output timing of the output element based on the control information received by the control information receiving means,
The integrated data providing device includes:
Integrated data transmission means for transmitting integrated data comprising information on the N output elements to the output device;
An output timing control system comprising: synchronization information transmitting means for transmitting information related to the synchronization timing to the output timing control device. An output timing control device according to claim 9, and S (S is an integer of 1 or more) output devices,
The output timing control device and the S output devices are connected to be capable of data communication,
The output timing control device includes:
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device;
An output timing control system comprising: output timing correction means for correcting the output timing of the output element by the output means based on the control information received by the control information reception means. An output timing control device according to claim 9, S (S is an integer of 1 or more) output devices, and an integrated data providing device,
The output timing control device, the S output devices, and the integrated data providing device are connected so that data communication is possible,
The output timing control device includes:
Synchronization information receiving means for receiving information related to the synchronization timing transmitted from the integrated data providing device,
The output timing control means transmits control information used to control the output timing of the output device to the output device,
The output device is
Integrated data receiving means for receiving integrated data transmitted from the integrated data providing device;
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device;
Output timing correction means for correcting the output timing of the output element based on the control information received by the control information receiving means,
The integrated data providing device includes:
Integrated data transmitting means for transmitting integrated data including information on the N output elements to the output timing control device and the output device;
An output timing control system comprising: synchronization information transmitting means for transmitting information related to the synchronization timing to the output timing control device.   The output timing control system according to any one of claims 11 to 14, wherein the output device includes periodic signal output means for outputting the periodic signal together with the output element. The output timing control device according to any one of claims 1 to 9,
It is characterized by comprising synchronization information receiving means for receiving, from the integrated data providing device, information related to the synchronization timing when integrating the N output elements, which are output separately from the M output devices, respectively. Output timing control device. Integrated data receiving means for receiving integrated data including information related to N (N is an integer of 2 or more) output elements transmitted from the integrated data providing device;
Output means capable of outputting x (1 ≦ x <N) output elements among the N output elements;
Control information receiving means for receiving control information transmitted from the output timing control device and used to control the output timing of the output element by the output means;
An output device comprising: output timing correcting means for correcting the output timing of the output element based on the control information received by the control information receiving means. Integrated data transmission means for transmitting integrated data including information on N (N is an integer of 2 or more) output elements to the output timing control device and the output device;
An integrated data providing device comprising: synchronization information transmitting means for transmitting information related to a synchronization timing when integrating the N output elements to the output timing control device. By outputting the output results obtained by integrating N (N is an integer of 2 or more) output elements, the N output elements are output separately from M (1 <M ≦ N) output devices, respectively. An output timing control program used to control the output timing of the output element from the output device in the system to be configured,
A synchronization information storage step for storing information on synchronization timing when integrating the N output elements in the synchronization information storage means;
An output information acquisition step of acquiring information on output elements output from the M output devices at a predetermined position;
A synchronization deviation calculating step for calculating each deviation of the arrival timing of the N output elements at the predetermined position with respect to the synchronization timing based on the acquisition result in the output information acquisition step and the stored contents of the synchronization information storage means; ,
An output timing control step for controlling the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result in the synchronization deviation calculation step An output timing control program comprising a program used for causing a computer to execute a process consisting of: An output timing control program according to claim 19,
Output timing control further comprising: a program used for causing a computer to execute a synchronization information receiving step for receiving information related to synchronization timing when integrating the N output elements from the integrated data providing apparatus. program. A program used to control an output device capable of outputting x (1 ≦ x <N) output elements among N (N is an integer of 2 or more) output elements,
An integrated data receiving step for receiving integrated data including information on the N output elements transmitted from the integrated data providing device;
An output step of outputting the x output elements among the N output elements;
A control information receiving step for receiving control information transmitted from the output timing control device and used to control the output timing of the output element in the output step;
An output device control program comprising: a program used for causing a computer to execute a process including an output timing correction step for correcting an output timing of an output element based on the control information received in the control information reception step . An integrated data transmission step of transmitting integrated data including information on N (N is an integer of 2 or more) output elements to the output timing control device and the output device;
A program used for causing a computer to execute processing including a synchronization information transmission step of transmitting information related to synchronization timing when integrating the N output elements to the output timing control device is included. Integrated data providing device control program. By outputting the output results obtained by integrating N (N is an integer of 2 or more) output elements, the N output elements are output separately from M (1 <M ≦ N) output devices, respectively. An output timing control device control method used for controlling an output timing control device for controlling an output timing of the output element from the output device in a system comprising:
The output timing control device is
An output information acquisition step of acquiring information on output elements output from the M output devices at a predetermined position;
Based on the acquisition result in the output information acquisition step and the stored contents of the synchronization information storage means storing information related to the synchronization timing when the N output elements are integrated, the N outputs at the predetermined position A synchronization deviation calculating step for calculating each deviation of the arrival timing of the element with respect to the synchronization timing;
An output timing control step for controlling the output timing of the output element from the output device so that the arrival timing of the N output elements at the predetermined position matches the synchronization timing based on the calculation result in the synchronization deviation calculation step And an output timing control device control method. An output timing control device control method according to claim 23,
The output timing control device is
A method for controlling an output timing control device, further comprising a synchronization information receiving step of receiving, from an integrated data providing device, information related to a synchronization timing when integrating the N output elements. An output device control method used to control an output device capable of outputting x (1 ≦ x <N) output elements among N (N is an integer of 2 or more) output elements,
The output device is
An integrated data receiving step for receiving integrated data including information on the N output elements transmitted from the integrated data providing device;
An output step of outputting the x output elements among the N output elements;
A control information receiving step for receiving control information transmitted from the output timing control device and used to control the output timing of the output element in the output step;
An output device control method comprising: an output timing correction step of correcting the output timing of the output element based on the control information received in the control information reception step. An integrated data transmission step of transmitting integrated data including information on N (N is an integer of 2 or more) output elements to the output timing control device and the output device;
And a synchronization information transmitting step of transmitting information related to synchronization timing when integrating the N output elements to the output timing control device.

Images