WO2023281665A1 - メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム - Google Patents

メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム Download PDF

Info

Publication number
WO2023281665A1
WO2023281665A1 PCT/JP2021/025651 JP2021025651W WO2023281665A1 WO 2023281665 A1 WO2023281665 A1 WO 2023281665A1 JP 2021025651 W JP2021025651 W JP 2021025651W WO 2023281665 A1 WO2023281665 A1 WO 2023281665A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
audio
time
signal2
return
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/025651
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
麻衣子 井元
真二 深津
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NTT Inc
Original Assignee
Nippon Telegraph and Telephone Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nippon Telegraph and Telephone Corp filed Critical Nippon Telegraph and Telephone Corp
Priority to US18/574,334 priority Critical patent/US20240321319A1/en
Priority to PCT/JP2021/025651 priority patent/WO2023281665A1/ja
Priority to JP2023532954A priority patent/JPWO2023281665A1/ja
Publication of WO2023281665A1 publication Critical patent/WO2023281665A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs

Definitions

  • One aspect of the present invention relates to a media synchronization control device, a media synchronization control method, and a media synchronization control program.
  • video/audio playback is used to digitize video/audio shot/recorded at a certain location and transmit it to a remote location in real time via a communication line such as an IP (Internet Protocol) network.
  • IP Internet Protocol
  • devices have come into use. For example, public viewing, etc., in which video and audio of a sports match being held at a competition venue or video and audio of a music concert being held at a concert venue are transmitted in real time to a remote location are being actively performed.
  • Such video/audio transmission is not limited to one-to-one one-way transmission.
  • Video and audio are transmitted from the venue where the sports competition is held (hereafter referred to as the event venue) to multiple remote locations, and images and sounds such as cheers of spectators enjoying the event are transmitted to multiple remote locations. are filmed and recorded, the video and audio are transmitted to event venues and other remote locations, and output from large video display devices and speakers at each site.
  • RTP Real-time Transport Protocol
  • RTP Real-time Transport Protocol
  • video and audio shot/recorded at event site A at time T are transmitted to two remote locations B and C, and video and audio shot/recorded at remote location B and remote location C are sent to event venue A.
  • the video/audio filmed/recorded at time T transmitted from event venue A at remote location B is played back at time T b1 , and the video/audio filmed/recorded at remote location B at time T b1 is sent to the event venue.
  • a method of synchronizing and playing multiple videos and multiple sounds transmitted from multiple remote locations at event venue A is used.
  • time is synchronized using NTP (Network Time Protocol), PTP (Precision Time Protocol), etc. so that both the sending side and the receiving side manage the same time information.
  • NTP Network Time Protocol
  • PTP Precision Time Protocol
  • the absolute time of the instant when the video/audio was sampled is given as an RTP time stamp, and the timing is adjusted by delaying at least one or more of the video and audio based on the time information on the receiving side.
  • Synchronous playback technology for audio signals distributed over IP networks (Tokumoto, Ikedo, Kaneko, Kataoka, Transactions of the Institute of Electronics, Information and Communication Engineers D-II Vol. J87-D-II No.9 pp.1870-1883)
  • the video/audio shot/recorded at time T b1 at remote site B and the video/audio recorded at time T b1 at remote site C are the video transmitted from event site A. ⁇ Those scenes with different audio are being viewed, and even if the video and audio are played back synchronously at Event Venue A, it will not lead to the elimination of the above-mentioned instinctive incomprehensibility and unnaturalness.
  • the present invention has been made in view of the above circumstances, and its object is to provide a technique for appropriately synchronizing and reproducing a plurality of video/audio signals returned from a plurality of bases through different transmission routes. That's what it is.
  • the media synchronization control device is a device at a first site, and is configured to reproduce the first media acquired at the first site at each time at the second site.
  • FIG. 1 is a block diagram showing an example of the hardware configuration of each electronic device included in the media synchronization system according to the first embodiment.
  • FIG. 2 is a block diagram showing an example of the software configuration of each electronic device that constitutes the media synchronization system according to the first embodiment.
  • FIG. 3 is a diagram showing an example of the data structure of the video synchronization control DB provided in the server of the site O according to the first embodiment.
  • FIG. 4 is a diagram showing an example of the data structure of the voice time control DB provided in the server of the site O according to the first embodiment.
  • FIG. 5 is a diagram showing an example of the data structure of the video time management DB provided in the server at the site R1 according to the first embodiment.
  • FIG. 6 is a diagram showing an example of the data structure of an audio time management DB provided in the server of the site R1 according to the first embodiment.
  • FIG. 7 is a flowchart showing a video processing procedure and processing contents of the server at the site O according to the first embodiment.
  • FIG. 8 is a flowchart showing a video processing procedure and processing contents of the server at the site R1 according to the first embodiment.
  • FIG. 9 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet storing video V signal1 of a server at site O according to the first embodiment.
  • FIG. 10 is a flow chart showing a reception processing procedure and processing contents of an RTP packet storing video V signal1 of a server at site R1 according to the first embodiment.
  • FIG. 11 is a flowchart showing a calculation processing procedure and processing contents of the presentation time t1 of the server at the site R1 according to the first embodiment.
  • FIG. 12 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet storing video V signal2 of the server at the site R1 according to the first embodiment.
  • FIG. 13 is a flow chart showing a reception processing procedure and processing contents of an RTP packet storing video V signal2 of a server at site O according to the first embodiment.
  • FIG. 14 is a flow chart showing a synchronization processing procedure and processing contents of the video V signal2 of the server at the site O according to the first embodiment.
  • FIG. 15 is a flow chart showing an audio processing procedure and processing contents of the server at the site O according to the first embodiment.
  • FIG. 16 is a flow chart showing an audio processing procedure and processing contents of the server at the site R1 according to the first embodiment.
  • FIG. 17 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing the voice A signal1 of the server at the site O according to the first embodiment.
  • FIG. 18 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing the voice A signal1 of the server at the site R1 according to the first embodiment.
  • FIG. 19 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server at the site R1 according to the first embodiment.
  • FIG. 20 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server at the site O according to the first embodiment.
  • FIG. 21 is a flowchart showing a synchronization processing procedure and processing contents of the audio A signal2 of the server at the site O according to the first embodiment.
  • FIG. 22 is a block diagram showing an example of the software configuration of each electronic device that configures the media synchronization system according to the second embodiment.
  • FIG. 23 is a flow chart showing a video processing procedure and processing contents of the server at the site O according to the second embodiment.
  • FIG. 24 is a flow chart showing a video processing procedure and processing contents of the server at the site R1 according to the second embodiment.
  • FIG. 25 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet storing video V signal2 of a server at site R1 according to the second embodiment.
  • FIG. 26 is a flow chart showing a transmission processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t video of the server at the site R1 according to the second embodiment.
  • FIG. 27 is a diagram illustrating an example of processing by the image time correction transmission unit of the server at the site R according to the second embodiment.
  • FIG. 28 is a flow chart showing a reception processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t video of the server at the base O according to the second embodiment.
  • FIG. 29 is a diagram showing an example of processing by the video time correction notification unit of the server at the site R1 according to the second embodiment.
  • FIG. 30 is a flow chart showing a reception processing procedure and processing contents of an RTP packet storing video V signal2 of a server at site O according to the second embodiment.
  • FIG. 31 is a flow chart showing an audio processing procedure and processing contents of the server at the site O according to the second embodiment.
  • FIG. 32 is a flowchart showing the voice processing procedure and processing details of the server at the site R1 according to the second embodiment.
  • FIG. 33 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server at the site R1 according to the second embodiment.
  • FIG. 30 is a flow chart showing a reception processing procedure and processing contents of an RTP packet storing video V signal2 of a server at site O according to the second embodiment.
  • FIG. 31 is a flow chart showing an audio processing procedure and processing contents of the server at the site O according
  • FIG. 34 is a flow chart showing a transmission processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t audio of the server at the site R1 according to the second embodiment.
  • FIG. 35 is a flow chart showing a reception processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t audio of the server at the base O according to the second embodiment.
  • FIG. 36 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server at the site O according to the second embodiment.
  • the time information that is uniquely determined for the absolute time when the video/audio was filmed/recorded at the site O can be obtained from multiple remote sites R 1 to R n (where n is (integer of 2 or more) is used as time information for synchronously reproducing the return video/audio.
  • the video/audio shot/recorded at the time when the video/audio having the time information was reproduced is associated with the time information.
  • all or part of the video/audio transmitted from each of the bases R1 to Rn is synchronously reproduced based on the time information.
  • Time information is transmitted and received between the base O and each of the bases R 1 to R n by any of the following means.
  • the time information is associated with video/audio shot/recorded at each of the bases R1 to Rn .
  • Time information is stored in the header extension area of RTP packets transmitted and received between site O and sites R1 to Rn .
  • the time information is in absolute time format (hh:mm:ss.fff format), but may be in millisecond format.
  • APP Application-Defined
  • RTCP RTP Control Protocol
  • the time information is in millisecond format.
  • the time information is stored in SDP (Session Description Protocol) describing initial parameters to be exchanged between the site O and each of the sites R 1 to R n at the start of transmission. In this example, the time information is in millisecond format.
  • the time information used for processing the video/audio is stored in the header extension area of the RTP packets transmitted and received between the site O and each of the sites R 1 to R n .
  • the time information is in absolute time format (hh:mm:ss.fff format).
  • Video and audio will be explained as RTP packetized and sent and received, but it is not limited to this.
  • Video and audio may be processed and managed by the same functional unit/DB (database).
  • Video and audio may both be sent and received in one RTP packet.
  • Video and audio are examples of media.
  • FIG. 1 is a block diagram showing an example of the hardware configuration of each electronic device included in the media synchronization system S according to the first embodiment.
  • the media synchronization system S includes a plurality of electronic devices included in the site O, a plurality of electronic devices included in each of the sites R 1 to R n , and the time distribution server 10 .
  • the electronic devices at each base and the time distribution server 10 can communicate with each other via an IP network.
  • Sites R 1 to R n are examples of second sites different from the first sites. In order to refer to any one of bases R 1 to R n , it is sometimes written as base R.
  • the site O includes a server 1 , an event video camera 101 , a return video presentation device 102 , an event audio recording device 103 and a return audio presentation device 104 .
  • Site O is an example of a first site.
  • the server 1 is an electronic device that controls each electronic device included in the base O.
  • FIG. The server 1 is an example of a media synchronization control device.
  • the event image capturing device 101 is a device that includes a camera that captures images of the base O.
  • the event video shooting device 101 is an example of a video shooting device.
  • the return video presentation device 102 is a device including a display that reproduces and displays the video transmitted back from each of the bases R 1 to R n to the base O.
  • the display is a liquid crystal display.
  • the return video presentation device 102 is an example of a video presentation device or a presentation device.
  • the event sound recording device 103 is a device including a microphone for recording the sound of the site O.
  • FIG. The event audio recording device 103 is an example of an audio recording device.
  • the return voice presentation device 104 is a device including a speaker that reproduces and outputs the voice transmitted back from each of the sites R 1 to R n to the site O.
  • FIG. The return audio presentation device 104 is an example of an audio presentation device or a presentation device.
  • the server 1 includes a control section 11 , a program storage section 12 , a data storage section 13 , a communication interface 14 and an input/output interface 15 .
  • Each element provided in the server 1 is connected to each other via a bus.
  • the control unit 11 corresponds to the central part of the server 1.
  • the control unit 11 includes a processor such as a central processing unit (CPU).
  • the control unit 11 includes a ROM (Read Only Memory) as a nonvolatile memory area.
  • the control unit 11 includes a RAM (Random Access Memory) as a volatile memory area.
  • the processor expands the program stored in the ROM or the program storage unit 12 to the RAM.
  • the control unit 11 implements each functional unit described later by the processor executing the program expanded in the RAM.
  • the control unit 11 constitutes a computer.
  • the program storage unit 12 is composed of a non-volatile memory that can be written and read at any time, such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) as a storage medium.
  • the program storage unit 12 stores programs necessary for executing various control processes.
  • the program storage unit 12 stores a program that causes the server 1 to execute processing by each functional unit realized by the control unit 11 and described later.
  • the program storage unit 12 is an example of storage.
  • the data storage unit 13 is composed of a non-volatile memory that can be written and read at any time, such as an HDD or SSD as a storage medium.
  • the data storage unit 13 is an example of a storage or storage unit.
  • the communication interface 14 includes various interfaces that communicatively connect the server 1 with other electronic devices using communication protocols defined by IP networks.
  • the input/output interface 15 is an interface that enables communication between the server 1 and the event video shooting device 101, return video presentation device 102, event audio recording device 103, and return audio presentation device 104, respectively.
  • the input/output interface 15 may have a wired communication interface, or may have a wireless communication interface.
  • the hardware configuration of the server 1 is not limited to the configuration described above.
  • the server 1 allows the omission and modification of the above components and the addition of new components as appropriate.
  • the base R 1 includes a server 2 , a video presentation device 201 , an offset video camera 202 , a return video camera 203 , an audio presentation device 204 and a return audio recording device 205 .
  • the server 2 is an electronic device that controls each electronic device included in the base R1 .
  • the video presentation device 201 is a device including a display that reproduces and displays video transmitted from the site O to the site R1 .
  • the image presentation device 201 is an example of a presentation device.
  • the offset video shooting device 202 is a device capable of recording shooting time.
  • the offset image capturing device 202 is a device including a camera installed so as to capture the entire image display area of the image presentation device 201 .
  • the offset video imaging device 202 is an example of video imaging device.
  • the return image capturing device 203 is a device including a camera that captures an image of the site R1 .
  • the return image capturing device 203 captures an image of the site R1 where the image presentation device 201 that reproduces and displays the image transmitted from the site O to the site R1 is installed.
  • the return video imaging device 203 is an example of a video imaging device.
  • the audio presentation device 204 is a device including a speaker that reproduces and outputs audio transmitted from the site O to the site R1 .
  • Audio presentation device 204 is an example of a presentation device.
  • the return voice recording device 205 is a device including a microphone that records the voice of the site R1 .
  • the return sound recording device 205 records the sound of the site R1 where the sound presentation device 204 that reproduces and outputs the sound transmitted from the site O to the site R1 is installed.
  • the return voice recording device 205 is an example of a voice recording device.
  • the server 2 includes a control section 21 , a program storage section 22 , a data storage section 23 , a communication interface 24 and an input/output interface 25 .
  • Each element provided in the server 2 is connected to each other via a bus.
  • the controller 21 may be configured similarly to the controller 11 .
  • the processor expands the program stored in the ROM or the program storage unit 22 to the RAM.
  • the control unit 21 implements each functional unit described later by the processor executing the program expanded in the RAM.
  • the control unit 21 constitutes a computer.
  • the program storage unit 22 can be configured similarly to the program storage unit 12 .
  • the data storage unit 23 can be configured similarly to the data storage unit 13 .
  • Communication interface 24 may be configured similarly to communication interface 14 .
  • the communication interface 14 includes various interfaces that communicatively connect the server 2 with other electronic devices.
  • Input/output interface 25 may be configured similarly to input/output interface 15 .
  • the input/output interface 25 enables communication between the server 2 and each of the video presentation device 201 , the offset video camera 202 , the return video camera 203 , the audio presentation device 204 and the return audio recording device 205 .
  • the hardware configuration of the server 2 is not limited to the configuration described above.
  • the server 2 allows omission and modification of the above components and addition of new components as appropriate.
  • the hardware configuration of the plurality of electronic devices included in each of the sites R 2 to R n is the same as that of the site R 1 described above, so description thereof will be omitted.
  • the time distribution server 10 is an electronic device that manages the reference system clock.
  • the reference system clock is absolute time.
  • FIG. 2 is a block diagram showing an example of the software configuration of each electronic device that constitutes the media synchronization system S according to the first embodiment.
  • the server 1 includes a time management unit 111, an event video transmission unit 112, a return video reception unit 113, a return video synchronization control unit 114, an event audio transmission unit 115, a return audio reception unit 116, a return audio synchronization control unit 117, and a video synchronization control unit. It has a DB 131 and an audio synchronization control DB 132 . Each functional unit is implemented by execution of a program by the control unit 11 . It can also be said that each functional unit is provided in the control unit 11 or the processor. Each functional unit can be read as the control unit 11 or a processor.
  • the video synchronization control DB 131 and the audio synchronization control DB 132 are implemented by the data storage unit 13 .
  • the time management unit 111 performs time synchronization with the time distribution server 10 using well-known protocols such as NTP and PTP, and manages the reference system clock.
  • the time management unit 111 manages the same reference system clock as the reference system clock managed by the server 2 .
  • the reference system clock managed by the time management unit 111 and the reference system clock managed by the server 2 are time-synchronized.
  • the event video transmission unit 112 transmits the RTP packet containing the video V signal1 output from the event video shooting device 101 to each server of the sites R 1 to R n via the IP network.
  • Video V signal1 is a video acquired at base O at time T video , which is absolute time. Acquiring the video V signal1 includes the event video shooting device 101 shooting the video V signal1 . Obtaining the video V signal1 includes sampling the video V signal1 shot by the event video shooting device 101 .
  • the RTP packet storing the video V signal1 is given the time T video .
  • the time T video is the time when the video V signal1 was obtained at the base O.
  • the time T video is time information for synchronizing the return video at the base O.
  • the time T video is an example of the acquisition time of the video V signal1 .
  • the event video transmission unit 112 stores the time T video associated with the video V signal1 in the video synchronization control DB 131, which will be described later, each time an RTP packet containing the video V signal1 is transmitted.
  • the image V signal1 is an example of the first image.
  • the time T video is an example of the first time.
  • An RTP packet is an example of a packet.
  • the RTP packet storing video V signal1 is an example of the second packet.
  • the event video transmission unit 112 is an example of a transmission unit.
  • the return video receiving unit 113 receives the RTP packet storing the video V signal2 from each server of the sites R 1 to R n via the IP network.
  • the image V signal2 is the image acquired at the base R at the time when the image V signal1 acquired at the base O at each time T video is reproduced at the base R.
  • Acquiring the image V signal2 includes the return image capturing device 203 capturing the image V signal2 .
  • Acquiring the image V signal2 includes sampling the image V signal2 captured by the return image capturing device 203 .
  • the RTP packet storing the video V signal2 is given a time T video related to the video V signal2 .
  • the return video receiving unit 113 Every time the return video receiving unit 113 receives an RTP packet storing the video V signal2 , it stores the video V signal2 in the video synchronization control DB 131 described later in association with the time T video associated with the video V signal2 .
  • the image V signal2 is an example of the second image.
  • the RTP packet storing video V signal2 is an example of the first packet.
  • the return video receiving unit 113 is an example of a first receiving unit.
  • the return video synchronization control unit 114 simultaneously returns the video V signal2 related to the plurality of sites R among the sites R 1 to R n associated with one time T video stored in the video synchronization control DB 131 . 102.
  • the return video synchronization control unit 114 is an example of a media synchronization control unit.
  • the event audio transmission unit 115 transmits an RTP packet storing the audio A signal1 output from the event audio recording device 103 to each server of the sites R 1 to R n via the IP network.
  • the audio A signal1 is the audio acquired at the base O at time T audio , which is absolute time.
  • Acquiring the audio A signal1 includes recording the audio A signal1 by the event audio recording device 103 .
  • Acquiring the audio A signal1 includes sampling the audio A signal1 recorded by the event audio recording device 103 .
  • An RTP packet containing audio A signal1 is given time T audio .
  • the time T audio is the time when the audio A signal1 was acquired at the base O.
  • the time T audio is time information for synchronizing return audio at the base O.
  • the time T audio is an example of the acquisition time of the audio A signal1 .
  • the event audio transmission unit 115 stores the time T audio associated with the audio A signal1 in the audio synchronization control DB 132 described later each time it transmits an RTP packet containing the audio A signal1.
  • Audio A signal1 is an example of the first audio.
  • Time T audio is an example of a first time.
  • An RTP packet containing audio A signal1 is an example of a second packet.
  • the event audio transmission unit 115 is an example of a transmission unit.
  • the return audio receiving unit 116 receives the RTP packet containing the audio A signal2 from each server of the sites R 1 to R n via the IP network.
  • the audio A signal2 is the audio acquired at the site R at the time when the audio A signal1 acquired at the site O at each time T audio is reproduced at the site R.
  • Acquiring the audio A signal2 includes the return audio recording device 205 recording the audio A signal2 .
  • Acquiring the audio A signal2 includes sampling the audio A signal2 recorded by the return audio recording device 205 .
  • the RTP packet containing the audio A signal2 is given the time T audio associated with the audio A signal2 .
  • the return audio receiving unit 116 Every time the return audio receiving unit 116 receives an RTP packet containing the audio A signal2 , it stores the audio A signal2 in the audio synchronization control DB 132 described later in association with the time T audio related to the audio A signal1 .
  • Audio A signal2 is an example of the second audio.
  • the RTP packet containing the audio A signal2 is an example of the first packet.
  • Return voice receiving section 116 is an example of a first receiving section.
  • the turn-back audio synchronization control unit 117 simultaneously turns back the audio A signal2 related to a plurality of locations R among the locations R 1 to R n associated with one time T audio stored in the audio synchronization control DB 132. 104.
  • the return audio synchronization control section 117 is an example of a media synchronization control section.
  • FIG. 3 is a diagram showing an example of the data structure of the video synchronization control DB 131 provided in the server 1 of the site O according to the first embodiment.
  • the video synchronization control DB 131 associates and stores the time T video and the video V signal2 stored in the RTP packets received by the return video receiving unit 113 from the n sites R 1 to R n .
  • the video synchronization control DB 131 has a video synchronization reference time column and n video data columns relating to bases R 1 to R n .
  • the video synchronization reference time column stores time T video .
  • the video data 1 column is a column related to base R1 .
  • the video data 1 column stores the video V signal2 returned from the site R1 .
  • the video data n column is a column related to base R n .
  • the video data n column stores the video V signal2 transmitted back from the site R n .
  • the video synchronization control DB 131 is an example of a storage unit.
  • FIG. 4 is a diagram showing an example of the data structure of the audio synchronization control DB 132 provided in the server 1 of the site O according to the first embodiment.
  • the audio synchronization control DB 132 associates and stores the time T audio and the audio A signal2 stored in the RTP packets received by the return audio receiving unit 116 from the n sites R 1 to R n .
  • the audio synchronization control DB 132 has an audio synchronization reference time column and n audio data columns.
  • the audio synchronization reference time column stores time T audio .
  • the voice data 1 column stores the voice A signal2 returned from the site R1 .
  • the voice data n column stores voice A signal2 returned from base R n .
  • Let r be the line number of a record in the audio synchronization control DB 132 . Let r be an integer with an initial value of 0.
  • the audio synchronization control DB 132 is an example of a storage unit.
  • the server 2 includes a time management unit 211, an event video reception unit 212, a video offset calculation unit 213, a return video transmission unit 214, an event audio reception unit 215, a return audio transmission unit 216, a video time management DB 231, and an audio time management DB 232.
  • Each functional unit is implemented by execution of a program by the control unit 21 . It can also be said that each functional unit is provided in the control unit 21 or the processor. Each functional unit can be read as the control unit 21 or the processor.
  • the video time management DB 231 and the audio time management DB 232 are realized by the data storage unit 23.
  • the time management unit 211 performs time synchronization with the time distribution server 10 using well-known protocols such as NTP and PTP, and manages the reference system clock.
  • the time management unit 211 manages the same reference system clock as the reference system clock managed by the server 1 .
  • the reference system clock managed by the time management unit 211 and the reference system clock managed by the server 1 are time-synchronized.
  • the event video reception unit 212 receives the RTP packet containing the video V signal1 from the server 1 via the IP network.
  • the event video reception unit 212 outputs the video V signal1 to the video presentation device 201 .
  • the video offset calculator 213 calculates the presentation time t 1 that is the absolute time when the video V signal 1 was reproduced by the video presentation device 201 .
  • the return video transmission unit 214 transmits the RTP packet containing the video V signal2 to the server 1 via the IP network.
  • the RTP packet containing the video V signal2 contains the time T video associated with the presentation time t1 that matches the absolute time t when the video V signal2 was captured.
  • the event audio receiver 215 receives the RTP packet containing the audio A signal1 from the server 1 via the IP network.
  • the event audio reception unit 215 outputs audio A signal1 to the audio presentation device 204 .
  • the return audio transmission unit 216 transmits the RTP packet containing the audio A signal2 to the server 1 via the IP network.
  • the RTP packet containing audio A signal2 includes time T audio .
  • FIG. 5 is a diagram showing an example of the data structure of the video time management DB 231 provided in the server 2 of the site R1 according to the first embodiment.
  • the video time management DB 231 is a DB that associates and stores the time T video acquired from the video offset calculation unit 213 and the presentation time t 1 .
  • the video time management DB 231 has a video synchronization reference time column and a presentation time column.
  • the video synchronization reference time column stores time T video .
  • the presentation time column stores the presentation time t1.
  • FIG. 6 is a diagram showing an example of the data structure of the voice time management DB 232 provided in the server 2 of the site R1 according to the first embodiment.
  • the audio time management DB 232 is a DB that associates and stores the time T audio acquired from the event audio reception unit 215 and the audio A signal1 .
  • the audio time management DB 232 has an audio synchronization reference time column and an audio data column.
  • the audio synchronization reference time column stores time T audio .
  • the audio data column stores audio A signal1 .
  • Each server at base R 2 to base R n includes the same functional unit and DB as the server 1 at base R 1 , and executes the same processing as the server 1 at base R 1 .
  • a description of the processing flow and DB structure of the functional units included in each server of base R 2 to base R n will be omitted.
  • base O and the base R1 will be described as an example.
  • the operation of the bases R 2 to R n may be the same as the operation of the base R 1 , and the description thereof will be omitted.
  • the notation of base R 1 may be read as base R 2 to base R n .
  • FIG. 7 is a flowchart showing video processing procedures and processing contents of the server 1 at the site O according to the first embodiment.
  • the event video transmission unit 112 transmits the RTP packet storing the video V signal1 to the server of each site R via the IP network (step S11). A typical example of the processing of step S11 will be described later.
  • the return video receiving unit 113 receives the RTP packet containing the video V signal2 from the server of each site R via the IP network (step S12).
  • the return video receiving unit 113 stores the video V signal2 in the video synchronization control DB 131 based on the time T video stored in the RTP packet storing the video V signal2 .
  • a typical example of the processing of step S12 will be described later.
  • the return video synchronization control unit 114 simultaneously returns the video V signal2 related to the plurality of sites R among the sites R 1 to R n associated with one time T video stored in the video synchronization control DB 131 . 102 (step S13). A typical example of the processing of step S13 will be described later.
  • FIG. 8 is a flow chart showing a video processing procedure and processing contents of the server 2 at the site R1 according to the first embodiment.
  • the event video reception unit 212 receives the RTP packet containing the video V signal1 from the server 1 via the IP network (step S14). A typical example of the processing of step S14 will be described later.
  • the video offset calculator 213 calculates the presentation time t1 at which the video V signal1 was reproduced by the video presentation device 201 (step S15). A typical example of the processing of step S15 will be described later.
  • the return video transmission unit 214 transmits the RTP packet containing the video V signal2 to the server 1 via the IP network (step S16). A typical example of the processing of step S16 will be described later.
  • step S11 of the server 1 Typical examples of the processing of steps S11 to S13 of the server 1 and the processing of steps S14 to S16 of the server 2 are described below.
  • step S11 of the server 1 the process of step S14 of the server 2, the process of step S15 of the server 2, the process of step S16 of the server 2, and the process of step S12 of the server 1 processing, and the processing of step S13 of the server 1 will be described in this order.
  • FIG. 9 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing video V signal1 of the server 1 at the site O according to the first embodiment.
  • FIG. 9 shows a typical example of the processing of step S11.
  • the event video transmission unit 112 acquires the video V signal1 output from the event video shooting device 101 at regular intervals I video (step S111).
  • the event video transmission unit 112 generates an RTP packet containing the video V signal1 (step S112).
  • step S112 for example, the event video transmission unit 112 stores the acquired video V signal1 in an RTP packet.
  • the event video transmission unit 112 acquires the time T video that is the absolute time at which the video V signal1 is sampled from the reference system clock managed by the time management unit 111 .
  • the event video transmission unit 112 stores the acquired time T video in the header extension area of the RTP packet.
  • the event video transmission unit 112 stores the acquired time T video in the video synchronization reference time column of the video synchronization control DB 131 (step S113).
  • the event video transmission unit 112 transmits the RTP packet containing the generated video V signal1 to the IP network (step S114).
  • FIG. 10 is a flow chart showing a reception processing procedure and processing contents of an RTP packet storing video V signal1 of the server 2 at the site R1 according to the first embodiment.
  • FIG. 10 shows a typical example of the processing of step S14 of the server 2.
  • the event video reception unit 212 receives the RTP packet containing the video V signal1 transmitted from the event video transmission unit 112 via the IP network (step S141).
  • the event video reception unit 212 acquires the video V signal1 stored in the RTP packet storing the received video V signal1 (step S142).
  • the event video reception unit 212 outputs the acquired video V signal1 to the video presentation device 201 (step S143).
  • the video presentation device 201 reproduces and displays the video V signal1 .
  • the event video reception unit 212 acquires the time T video stored in the header extension area of the RTP packet storing the received video V signal1 (step S144). The event video reception unit 212 transfers the acquired video V signal1 and time T video to the video offset calculation unit 213 (step S145).
  • FIG. 11 is a flow chart showing a calculation processing procedure and processing contents of the presentation time t1 of the server 2 at the site R1 according to the first embodiment.
  • FIG. 11 shows a typical example of the processing of step S15 of the server 2.
  • FIG. The video offset calculator 213 acquires the video V signal1 and the time T video from the event video receiver 212 (step S151).
  • the image offset calculation unit 213 calculates the presentation time t1 based on the obtained image V signal1 and the image input from the offset image capturing device 202 (step S152).
  • the video offset calculation unit 213 extracts a video frame including the video V signal1 from the video shot by the offset video shooting device 202 using a known image processing technique.
  • the video offset calculation unit 213 acquires the shooting time given to the extracted video frame as the presentation time t1.
  • the shooting time is absolute time.
  • the video offset calculator 213 stores the acquired time T video in the video synchronization reference time column of the video time management DB 231 (step S153).
  • the video offset calculator 213 stores the acquired presentation time t1 in the presentation time column of the video time management DB 231 (step S154).
  • FIG. 12 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet storing video V signal2 of the server 2 at the site R1 according to the first embodiment.
  • FIG. 12 shows a typical example of the processing of step S16 of the server 2.
  • the return video transmission unit 214 acquires the video V signal2 output from the return video camera 203 at regular intervals I video (step S161).
  • the video V signal2 is a video acquired at the site R1 at the time when the video presentation device 201 reproduces the video V signal1 acquired at each time T video at the site O at the site R1 .
  • the return video transmission unit 214 calculates the time t, which is the absolute time when the acquired video V signal2 was captured (step S162).
  • the return video transmission unit 214 refers to the video time management DB 231 and extracts a record having time t1 that matches the acquired time t (step S163).
  • the return video transmission unit 214 refers to the video time management DB 231 and acquires the time T video in the video synchronization reference time column of the extracted record (step S164).
  • the return video transmission unit 214 generates an RTP packet containing the video V signal2 (step S165).
  • step S165 for example, the return video transmission unit 214 stores the acquired video V signal2 in the RTP packet.
  • the return video transmission unit 214 stores the acquired time T video in the header extension area of the RTP packet.
  • the return video transmission unit 214 transmits the RTP packet containing the generated video V signal2 to the IP network (step S166).
  • FIG. 13 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing video V signal2 of the server 1 at the site O according to the first embodiment.
  • FIG. 13 shows a typical example of the processing of step S12 of the server 1.
  • the return video reception unit 113 receives the RTP packet containing the video V signal2 transmitted from the return video transmission unit 214 via the IP network (step S121).
  • the return video reception unit 113 acquires the video V signal2 stored in the RTP packet storing the received video V signal2 (step S122).
  • the return video receiving unit 113 acquires the time T video stored in the header extension area of the RTP packet storing the received video V signal2 (step S123).
  • the return video receiving unit 113 acquires the transmission source base R x (x is any one of 1, 2, . S124).
  • the return video receiving unit 113 refers to the video synchronization control DB 131 and determines that the time T video stored in the video synchronization reference time column is the time T video associated with the video V signal2 obtained from the RTP packet storing the video V signal2 . (step S125).
  • the return video receiving unit 113 stores the acquired video V signal2 in the video data x column related to the acquired transmission source site R x among the extracted records (step S126).
  • Storing the video V signal2 in the record of the video synchronization control DB 131 is an example of storing the video V signal2 in the video synchronization control DB 131 in association with the time T video . For example, when the return video receiving unit 113 receives an RTP packet containing video V signal2 from the server 2 of the site R1 , it stores the video V signal2 in the video data 1 column related to the transmission source site R1.
  • FIG. 14 is a flow chart showing the synchronization processing procedure and processing details of the video V signal2 of the server 1 at the site O according to the first embodiment.
  • FIG. 14 shows a typical example of the processing of step S13 of the server 1.
  • the return video synchronization control unit 114 simultaneously outputs all the video V signal2 stored in the n video data columns of the r-th record in the video synchronization control DB 131 to the return video presentation device 102 (step S131).
  • step S131 for example, the return video synchronization control unit 114 starts processing from the 0th record.
  • Return video synchronization control unit 114 starts outputting video V signal2 to return video presentation device 102 after time t video_start has elapsed from the start timing of transmission of the RTP packet storing video V signal1 by event video transmission unit 112 .
  • the time t video_start is from the start timing of transmission of the RTP packet storing the video V signal1 by the event video transmission unit 112 to all the n video data columns of the 0th record in the video synchronization control DB 131 . may be the time until is stored.
  • the time t video_start may be calculated by the return video synchronization control unit 114 .
  • the time t video_start may be a predetermined value.
  • the return video synchronization control unit 114 extracts one line from the r-th record.
  • the return video synchronization control unit 114 simultaneously outputs all the video V signal2 stored in the n video data columns of the r-th record to the return video presentation device 102 .
  • the r-th record is a record of one time T video . All the video V signal2 stored in the n video data columns of the r-th record are the video V signal2 related to a plurality of sites R among sites R 1 to R n associated with one time T video . is an example.
  • the rth record may store video V signal2 in all n video data columns.
  • the r-th record stores video V signal2 for all sites R among sites R 1 to R n .
  • the return video synchronization control unit 114 simultaneously outputs all the video V signal2 stored in all n video data columns of the r-th record to the return video presentation device 102 .
  • the rth record may store video V signal2 in part of the n video data columns.
  • the r-th record stores a video V signal2 related to a plurality of sites R that are part of sites R 1 to R n .
  • the return video synchronization control unit 114 simultaneously outputs all the video V signal2 stored in the plurality of video data columns that are part of the n video data columns of the r-th record to the return video presentation device 102 .
  • the return video synchronization control unit 114 outputs this output to the return video presentation device 102 in the processing of the (r-1)th record in the video data column related to the site R in which the video V signal2 of the r-th record is not stored.
  • the image V signal2 related to the site R may be repeatedly output to the image presentation device 102 in return. Note that when r is 0, the return video synchronization control unit 114 does not output the video V signal2 to the return video presentation device 102 in the video data column related to the site R where the video V signal2 of the 0th record is not stored. .
  • the return video synchronization control unit 114 determines whether or not an unprocessed record exists in the video synchronization control DB 131 (step S132). If there is no unprocessed record (step S132, NO), the process ends. If there is an unprocessed record (step S132, YES), the process transitions from step S132 to step S133. The return video synchronization control unit 114 increments the row number r by 1 (step S133).
  • the return video synchronization control unit 114 determines whether or not a certain interval I video has passed after processing the (r-1)th record (step S134). If the interval I video has not elapsed (step S134, NO), the return video synchronization control unit 114 repeats the process of step S134. If the interval I video has passed (step S134, YES), the process returns from step S134 to step S131.
  • the return video synchronization control unit 114 extracts records line by line from the video synchronization control DB 131 at regular intervals I video . Each time the return video synchronization control unit 114 extracts a record, it simultaneously outputs all the video V signal2 stored in the n video data columns of the extracted record to the return video presentation device 102 . In other words, even if there is an RTP packet that has not arrived at the base O by the playback time, which is the processing time of the record, the return video synchronization control unit 114 detects all the video V that has arrived at the hub O by the playback time. At the same time, the signal2 is output to the image presentation device 102 in return. Even if the RTP packet arrives at the base O after the reproduction time, the return video synchronization control unit 114 does not output the video V signal2 stored in the RTP packet to the return video presentation device 102 .
  • FIG. 15 is a flow chart showing the voice processing procedure and processing contents of the server 1 at the site O according to the first embodiment.
  • the event audio transmission unit 115 transmits the RTP packet storing the audio A signal1 to the server of each site R via the IP network (step S17). A typical example of the processing of step S17 will be described later.
  • the return audio receiving unit 116 receives the RTP packet containing the audio A signal2 from the server of each site R via the IP network (step S18).
  • the return audio receiving unit 116 stores the audio A signal2 in the audio synchronization control DB 132 based on the time T audio stored in the RTP packet storing the audio A signal2.
  • a typical example of the processing of step S18 will be described later.
  • the turn-back audio synchronization control unit 117 simultaneously turns back the audio A signal2 related to a plurality of locations R among the locations R 1 to R n associated with one time T audio stored in the audio synchronization control DB 132. 104 (step S19).
  • step S19 A typical example of the processing of step S19 will be described later.
  • FIG. 16 is a flow chart showing the voice processing procedure and processing contents of the server 2 at the site R1 according to the first embodiment.
  • the event audio receiver 215 receives the RTP packet containing the audio A signal1 from the server 1 via the IP network (step S20). A typical example of the processing of step S20 will be described later.
  • the return audio transmission unit 216 transmits the RTP packet containing the audio A signal2 to the server 1 via the IP network (step S21). A typical example of the processing of step S21 will be described later.
  • FIG. 17 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing the audio A signal1 of the server 1 at the site O according to the first embodiment.
  • FIG. 17 shows a typical example of the processing of step S17 of the server 1.
  • the event audio transmission unit 115 acquires the audio A signal1 output from the event audio recording device 103 at regular intervals I audio (step S171).
  • the event audio transmission unit 115 generates an RTP packet containing the audio A signal1 (step S172).
  • step S172 for example, the event audio transmission unit 115 stores the acquired audio A signal1 in an RTP packet.
  • the event audio transmission unit 115 acquires the time T audio , which is the absolute time when the audio A signal1 is sampled, from the reference system clock managed by the time management unit 111 .
  • the event audio transmission unit 115 stores the acquired time T audio in the header extension area of the RTP packet.
  • the event audio transmission unit 115 transmits the RTP packet containing the generated audio A signal1 to the IP network (step S173).
  • FIG. 18 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing the voice A signal1 of the server 2 at the site R1 according to the first embodiment.
  • FIG. 18 shows a typical example of the processing of step S20 of the server 2.
  • the event audio reception unit 215 receives the RTP packet containing the audio A signal1 transmitted from the event audio transmission unit 115 via the IP network (step S201).
  • the event audio receiver 215 acquires the audio A signal1 stored in the RTP packet storing the received audio A signal1 (step S202).
  • the event sound reception unit 215 outputs the acquired sound A signal1 to the sound presentation device 204 (step S203).
  • the audio presentation device 204 reproduces and outputs the audio A signal1 .
  • the event audio receiver 215 acquires the time T audio stored in the header extension area of the RTP packet storing the received audio A signal1 (step S204).
  • the event audio reception unit 215 stores the acquired audio A signal1 and time T audio in the audio time management DB 232 (step S205).
  • step S ⁇ b>205 for example, the event audio reception unit 215 stores the acquired time T audio in the audio synchronization reference time column of the audio time management DB 232 .
  • the event audio reception unit 215 stores the acquired audio A signal1 in the audio data column of the audio time management DB 232 .
  • FIG. 19 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server 2 at the site R1 according to the first embodiment.
  • FIG. 19 shows a typical example of the processing of step S21 of the server 2.
  • the return audio transmission unit 216 acquires the audio A signal2 output from the return audio recording device 205 at regular intervals I audio (step S211).
  • the audio A signal2 is the audio acquired at the location R1 at the time when the audio presentation device 204 reproduces the audio A signal1 acquired at the location O at each time T audio at the location R1 .
  • the return audio transmission unit 216 refers to the audio time management DB 232 and extracts records having audio data including the acquired audio A signal2 (step S212).
  • the sound A signal2 acquired by the return sound transmission unit 216 includes the sound A signal1 reproduced by the sound presentation device 204 and the sound generated at the base R1 (such as the cheers of the audience at the base R1 ).
  • the return voice transmission unit 216 separates two voices by a known voice analysis technique.
  • the return audio transmission unit 216 identifies the audio A signal1 reproduced by the audio presentation device 204 by separating the audio.
  • the return audio transmission unit 216 refers to the audio time management DB 232 and searches for audio data that matches the audio A signal1 reproduced by the specified audio presentation device 204 .
  • the return audio transmission unit 216 refers to the audio time management DB 232 and extracts a record having audio data that matches the audio A signal1 reproduced by the specified audio presentation device 204 .
  • the return audio transmission unit 216 refers to the audio time management DB 232 and acquires the time T audio in the audio synchronization reference time column of the extracted record (step S213).
  • the return audio transmission unit 216 generates an RTP packet containing the audio A signal2 (step S214).
  • the return audio transmission unit 216 stores the acquired audio A signal2 in an RTP packet.
  • the return audio transmission unit 216 stores the acquired time T audio in the header extension area of the RTP packet.
  • the return audio transmission unit 216 transmits the RTP packet containing the generated audio A signal2 to the IP network (step S215).
  • FIG. 20 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server 1 at the site O according to the first embodiment.
  • FIG. 20 shows a typical example of the processing of step S18 of the server 1.
  • the return voice receiving unit 116 receives the RTP packet containing the voice A signal2 transmitted from the return voice transmitting unit 216 via the IP network (step S181).
  • the return audio receiving unit 116 acquires the audio A signal2 stored in the RTP packet storing the received audio A signal2 (step S182).
  • the return audio receiving unit 116 acquires the time T audio stored in the header extension area of the RTP packet storing the received audio A signal2 (step S183).
  • the return audio receiving unit 116 acquires the transmission source site R x from the information stored in the header of the RTP packet containing the received audio A signal2 (step S184).
  • the return audio receiving unit 116 refers to the audio synchronization control DB 132 , and the time T audio stored in the audio synchronization reference time column is the time T audio associated with the audio A signal2 obtained from the RTP packet storing the audio A signal2. (step S185).
  • the return voice receiving unit 116 stores the acquired voice A signal2 in the voice data x column related to the acquired transmission source site R x among the extracted records (step S186).
  • Storing the audio A signal2 in the record of the audio synchronization control DB 132 is an example of storing the audio A signal2 in association with the time T audio .
  • the return audio receiving unit 116 receives an RTP packet containing audio A signal2 from the server 2 of the location R1 , it stores the audio A signal2 in the audio data 1 column for the transmission source location R1.
  • FIG. 21 is a flowchart showing a synchronization processing procedure and processing contents of the audio A signal2 of the server 1 at the site O according to the first embodiment.
  • FIG. 21 shows a typical example of the process of step S19 of the server 1.
  • the return audio synchronization control unit 117 simultaneously outputs all the sounds A signal2 stored in the n audio data columns of the r-th record in the audio synchronization control DB 132 to the return audio presentation device 104 (step S191).
  • step S191 for example, the return audio synchronization control unit 117 starts processing from the 0th record.
  • Return audio synchronization control section 117 starts outputting audio A signal2 to return audio presentation device 104 after time t audio_start has elapsed from the timing at which event audio transmission section 115 starts sending the RTP packet containing audio A signal1 .
  • the time t audio_start is from the start timing of the transmission of the RTP packet containing the audio A signal1 by the event audio transmission unit 115 to all of the n audio data columns of the 0th record in the audio synchronization control DB 132. may be the time until is stored.
  • the time t audio_start may be calculated by the return audio synchronization control unit 117 .
  • the time t audio_start may be a predetermined value.
  • the return audio synchronization control unit 117 extracts one line from the r-th record.
  • the return audio synchronization control unit 117 simultaneously outputs all the sounds A signal2 stored in the n audio data columns of the r-th record to the return audio presentation device 104 .
  • the r-th record is a record of one time T audio . All the audio A signal2 stored in the n audio data columns of the r-th record are the audio A signal2 related to multiple locations R among the locations R 1 to R n associated with one time T audio . is an example.
  • the rth record may store audio A signal2 in all n audio data columns.
  • the r-th record stores audio A signal2 related to all sites R among sites R 1 to R n .
  • the return audio synchronization control unit 117 simultaneously outputs all the sounds A signal2 stored in all of the n audio data columns of the r-th record to the return audio presentation device 104 .
  • the rth record may also store the audio A signal2 in part of the n audio data columns.
  • the r-th record stores audio A signal2 for a plurality of sites R that are part of sites R 1 to R n .
  • the return audio synchronization control unit 117 simultaneously outputs all the sounds A signal2 stored in the plurality of audio data columns that are part of the n audio data columns of the r-th record to the return audio presentation device 104 .
  • the return audio synchronization control unit 117 outputs this output to the return audio presentation device 104 in the processing of the (r-1)th record in the audio data column related to the site R in which the r-th record audio A signal2 is not stored.
  • the audio A signal2 related to the site R may be repeatedly output to the audio presentation device 104 in return. Note that when r is 0, the loopback audio synchronization control unit 117 does not output the loopback audio signal2 to the loopback audio presentation device 104 in the audio data column related to the site R where the audio A signal2 of the 0th record is not stored. .
  • the return audio synchronization control unit 117 determines whether or not an unprocessed record exists in the audio synchronization control DB 132 (step S192). If there is no unprocessed record (step S192, NO), the process ends. If there is an unprocessed record (step S192, YES), the process transitions from step S192 to step S193. The return audio synchronization control unit 117 increments the line number r by 1 (step S193).
  • the return audio synchronization control unit 117 determines whether or not a certain interval I audio has passed after processing the (r-1)th record (step S194). If the interval I audio has not elapsed (step S194, NO), the return audio synchronization control unit 117 repeats the process of step S194. If the interval I audio has passed (step S194, YES), the process returns from step S194 to step S191.
  • the return audio synchronization control unit 117 extracts records line by line from the audio synchronization control DB 132 at regular intervals Iaudio . Each time a record is extracted, return audio synchronization control section 117 simultaneously outputs all sounds A signal2 stored in n audio data columns of the extracted record to return audio presentation device 104 . In other words, even if there is an RTP packet that has not arrived at the base O by the playback time, which is the processing time of the record, the loopback audio synchronization control unit 117 detects all the voices A that have arrived at the hub O by the playback time. At the same time, the signal2 is output to the audio presentation device 104 by returning. Even if the RTP packet arrives at the site O after the reproduction time, the return audio synchronization control unit 117 does not output the audio A signal2 stored in the RTP packet to the return audio presentation device 104 .
  • the timing of outputting all audio A signal2 of the record to the loopback audio presentation device 104 at the same time may be the same or may be different.
  • the server 1 stores the video V signal2 in the video synchronization control DB 131 based on the time T video stored in the RTP packet storing the video V signal2 .
  • the server 1 simultaneously outputs to the video presentation device 102 the video V signal2 related to the plurality of bases R associated with one time T video stored in the video synchronization control DB 131 .
  • the server 1 stores the audio A signal2 in the audio synchronization control DB 132 based on the time T audio stored in the RTP packet storing the audio A signal2.
  • the server 1 simultaneously outputs to the audio presentation device 104 the audio A signal2 related to the multiple sites R associated with one time T audio stored in the audio synchronization control DB 132 .
  • the server 1 can associate with each other the video V signal2 or the audio A signal2 related to the same acquisition time transmitted at different timings from the plurality of bases R based on the acquisition time of the video V signal1 or the audio A signal1 . .
  • the server 1 can simultaneously output video V signal2 or audio A signal2 for a plurality of locations R associated with one acquisition time.
  • the server 1 can appropriately synchronously reproduce a plurality of video/audio returned from a plurality of bases R through different transmission routes.
  • Video and audio will be explained as RTP packetized and sent and received, but it is not limited to this.
  • Video and audio may be processed and managed by the same functional unit/DB (database).
  • Video and audio may both be sent and received in one RTP packet.
  • 2nd Embodiment In 2nd Embodiment, the same code
  • the hardware configuration of each electronic device included in the media synchronization system S according to the second embodiment may be the same as that of the first embodiment, and the description thereof will be omitted.
  • FIG. 22 is a block diagram showing an example of the software configuration of each electronic device that constitutes the media synchronization system S according to the second embodiment.
  • the server 1 includes a time management unit 111, an event video transmission unit 112, a return video reception unit 113, a return video synchronization control unit 114, an event audio transmission unit 115, a return audio reception unit 116, a return It has an audio synchronization control unit 117 , a video synchronization control DB 131 and an audio synchronization control DB 132 .
  • the server 1 includes a video time correction notification unit 118 and an audio time correction notification unit 119 unlike the first embodiment.
  • Each functional unit is implemented by execution of a program by the control unit 11 . It can also be said that each functional unit is provided in the control unit 11 or the processor. Each functional unit can be read as the control unit 11 or a processor.
  • the video synchronization control DB 131 and the audio synchronization control DB 132 are implemented by the data storage unit 13 .
  • the video time correction notification unit 118 receives an RTCP packet containing the correction time information ⁇ t video from the server of each site R via the IP network.
  • the corrected time information ⁇ t video is the value of the difference between the time t2 and the time T video .
  • the time t2 is an example of the acquisition time of the video V signal2 acquired at the site R at the time when the video V signal1 acquired at the site O at the time T video is reproduced at the site R.
  • An RTCP packet is an example of a packet.
  • the RTCP packet storing the corrected time information ⁇ t video is an example of the third packet.
  • the video time correction notifier 118 is an example of a second receiver.
  • the audio time correction notification unit 119 receives an RTCP packet containing the correction time information ⁇ t audio from the server of each site R via the IP network.
  • the corrected time information ⁇ t audio is the value of the difference between the time t3 and the time T audio .
  • Time t3 is an example of the acquisition time of the audio A signal2 acquired at the site R at the time when the audio A signal1 acquired at the site O at the time T audio is reproduced at the site R.
  • the RTCP packet storing the corrected time information ⁇ t audio is an example of the third packet.
  • the voice time correction notifier 119 is an example of a second receiver.
  • the server 2 includes a time management unit 211, an event video reception unit 212, a video offset calculation unit 213, a return video transmission unit 214, an event audio reception unit 215, a return audio transmission unit 216, a video time It has a management DB 231 and an audio time management DB 232 .
  • the server 2 includes a video time correction transmission section 217 and an audio time correction transmission section 218 unlike the first embodiment.
  • Each functional unit is implemented by execution of a program by the control unit 21 . It can also be said that each functional unit is provided in the control unit 21 or the processor. Each functional unit can be read as the control unit 21 or the processor.
  • the video time management DB 231 and the audio time management DB 232 are realized by the data storage unit 23.
  • the video time correction transmission unit 217 transmits an RTCP packet containing the correction time information ⁇ t video to the server 1 via the IP network.
  • the audio time correction transmission unit 218 transmits an RTCP packet containing the correction time information ⁇ t audio to the server 1 via the IP network.
  • base O and the base R1 will be described as an example.
  • the operation of the bases R 2 to R n may be the same as the operation of the base R 1 , and the description thereof will be omitted.
  • the notation of base R 1 may be read as base R 2 to base R n .
  • FIG. 23 is a flowchart showing video processing procedures and processing details of the server 1 at the site O according to the second embodiment.
  • the event video transmission unit 112 transmits the RTP packet storing the video V signal1 to the server of each site R via the IP network (step S22).
  • a typical example of the processing of the event video transmission unit 112 in step S22 may be the same as the processing described in the first embodiment using FIG. 9, and the description thereof will be omitted.
  • the event video transmission unit 112 may store the time T video in the RTP timestamp of the RTP packet instead of the header extension area of the RTP packet.
  • the video time correction notification unit 118 receives the RTCP packet containing the correction time information ⁇ t video from the server of each site R via the IP network (step S23). A typical example of the processing of step S23 will be described later.
  • the return video receiving unit 113 receives the RTP packet containing the video V signal2 from the server of each site R via the IP network (step S24).
  • the return video reception unit 113 stores the video V signal2 in the video synchronization control DB 131 based on the time obtained by subtracting the correction time information ⁇ t video from the time T' stored in the RTP packet storing the video V signal2 .
  • the time T′ is an example of the acquisition time of the video V signal2 acquired at the site R at the time when the video V signal1 acquired at the site O at the time T video is reproduced at the site R. A typical example of the processing of step S24 will be described later.
  • the return video synchronization control unit 114 simultaneously returns the video V signal2 related to the plurality of sites R among the sites R 1 to R n associated with one time T video stored in the video synchronization control DB 131 . 102 (step S25).
  • a typical example of the processing of the turn-back video synchronization control unit 114 in step S25 may be the same as the processing described in the first embodiment using FIG. 14, so description thereof will be omitted.
  • FIG. 24 is a flowchart showing video processing procedures and processing details of the server 2 at the site R1 according to the second embodiment.
  • the event video reception unit 212 receives the RTP packet containing the video V signal1 from the server 1 via the IP network (step S26).
  • a typical example of the processing of the event video reception unit 212 in step S26 may be the same as the processing described in the first embodiment using FIG. 10, and the description thereof will be omitted.
  • the event video reception unit 212 may acquire the time T video stored in the RTP timestamp of the RTP packet instead of the header extension area of the RTP packet.
  • the video offset calculator 213 calculates the presentation time t1 at which the video V signal1 was reproduced by the video presentation device 201 (step S27).
  • a typical example of the processing of the video offset calculation unit 213 in step S27 may be the same as the processing described in the first embodiment using FIG. 11, and the description thereof will be omitted.
  • the return video transmission unit 214 transmits the RTP packet containing the video V signal2 to the server 1 via the IP network (step S28).
  • step S28 A typical example of the processing of step S28 will be described later.
  • the video time correction transmission unit 217 transmits the RTCP packet containing the correction time information ⁇ t video to the server 1 via the IP network (step S29). A typical example of the processing of step S29 will be described later.
  • FIG. 25 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet storing video V signal2 of the server 2 at the site R1 according to the second embodiment.
  • FIG. 25 shows a typical example of the processing of step S28 of the server 2.
  • the return video transmission unit 214 acquires the video V signal2 output from the return video camera 203 at regular intervals I video (step S281).
  • the video V signal2 is a video acquired at the site R1 at the time when the video presentation device 201 reproduces the video V signal1 acquired at each time T video at the site O at the site R1 .
  • the return video transmission unit 214 acquires the time t2, which is the absolute time at which the video V signal2 captured by the return video camera 203 is sampled.
  • time t2 is the time obtained by adding ⁇ (minimum) to the time t , which is the absolute time when the video V signal2 was shot.
  • ⁇ is a process in which an image (one still image) is shot, this image is sent from the return image shooting device 203 to the return image transmission unit 214, and the return image transmission unit 214 converts an analog signal into a digital signal. is the time until is started. Since ⁇ is infinitely close to 0 , time t2 may be regarded as the same as time t.
  • the return video transmission unit 214 calculates the time t, which is the absolute time when the acquired video V signal2 was captured (step S282).
  • the return video transmission unit 214 refers to the video time management DB 231 and extracts a record having time t1 that matches the acquired time t (step S283).
  • the return video transmission unit 214 refers to the video time management DB 231 and acquires the time T video in the video synchronization reference time column of the extracted record (step S284).
  • the return video transmission unit 214 generates an RTP packet containing the video V signal2 (step S285).
  • step S285 for example, the return video transmission unit 214 stores the acquired video V signal2 in the RTP packet.
  • the return video transmission unit 214 stores the time T' corresponding to the time t2 in the RTP timestamp of the RTP packet.
  • the time T' is the earliest time t2 in the set of times t2 regarding the video V signal2 stored in the RTP packet. Time T' may be regarded as the same as time t.
  • the RTP packet storing the video V signal2 includes the sequence number s of the RTP packet header. To simplify the processing flow, the sequence number s is assumed to continue to be incremented for each generated RTP packet without returning to 0.
  • the return video transmission unit 214 transfers the acquired time T video , time t 2 and sequence number s to the video time correction transmission unit 217 (step S286).
  • the return video transmission unit 214 transmits the RTP packet storing the generated video V signal2 to the IP network (step S287).
  • FIG. 26 is a flow chart showing a transmission processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t video of the server 2 at the site R1 according to the second embodiment.
  • FIG. 26 shows a typical example of the processing of step S29 of the server 2.
  • the video time correction transmission unit 217 acquires the time T video , the time t 2 and the sequence number s from the return video transmission unit 214 (step S291).
  • the video time correction transmission unit 217 calculates the time (t2 - Tvideo) by subtracting the time Tvideo from the time t2 based on the time Tvideo and the time t2 ( step S292).
  • the video time correction transmission unit 217 determines whether or not the time (t 2 -T video ) matches the current correction time information ⁇ t video (step S293).
  • the corrected time information ⁇ t video is the value of the difference between the time t2 and the time T video .
  • the current corrected time information ⁇ t video is the value of the time (t 2 ⁇ T video ) calculated before the time (t 2 ⁇ T video ) calculated this time. Note that the initial value of the corrected time information ⁇ t video is 0. If the time (t 2 -T video ) matches the current corrected time information ⁇ t video (step S293, YES), the process ends.
  • step S293 If the time (t 2 ⁇ T video ) does not match the current corrected time information ⁇ t video (step S293, NO), the process transitions from step S293 to step S294.
  • the fact that the time (t 2 -T video ) does not match the current corrected time information ⁇ t video corresponds to a change in the corrected time information ⁇ t video .
  • the video time correction transmission unit 217 generates an RTCP packet containing the correction time information ⁇ t video (step S295).
  • step S295 for example, the video time correction transmission unit 217 describes the updated correction time information ⁇ t video using APP in RTCP.
  • the video time correction transmission unit 217 generates an RTCP packet containing the correction time information ⁇ t video .
  • the video time correction transmission unit 217 describes the sequence number s regarding the updated correction time information ⁇ t video using APP in RTCP.
  • the RTCP packet storing the corrected time information ⁇ t video stores the sequence number s.
  • the video time correction transmission unit 217 transmits the RTCP packet storing the generated correction time information ⁇ t video to the IP network (step S296). Note that the video time correction transmission unit 217 starts the processing illustrated in FIG. 26 before the return video transmission unit 214 transmits the RTP packet storing the video V signal2 . Therefore, the timing at which the video time correction transmission unit 217 transmits the RTCP packet containing the corrected time information ⁇ t video is temporally earlier than the return video transmission unit 214 transmits the RTP packet containing the video V signal2 .
  • the timing at which the video time correction transmission unit 217 transmits the RTCP packet containing the corrected time information ⁇ t video is temporally earlier than the return video transmission unit 214 transmits the RTP packet containing the video V signal2 .
  • FIG. 27 is a diagram showing an example of processing by the video time correction transmission unit 217 of the server 2 at the site R1 according to the second embodiment.
  • FIG. 27 shows the time T video acquired by the video time correction transmission unit 217 from the return video transmission unit 214, the time t 2 and the sequence number s, and the time calculated by the video time correction transmission unit 217 (t 2 - T video ). is shown.
  • the time t2 is a time at regular intervals according to the sequence number s.
  • FIG. 28 is a flowchart showing a reception processing procedure and processing contents of an RTCP packet containing the corrected time information ⁇ t video of the server 1 at the site O according to the second embodiment.
  • FIG. 28 shows a typical example of the processing of step S23 of the server 1.
  • the video time correction notification unit 118 receives the RTCP packet containing the correction time information ⁇ t video from the server of each site R via the IP network (step S231). Note that, as described above, the video time correction transmission unit 217 transmits to the server 1 an RTCP packet containing the correction time information ⁇ t video based on the change in the correction time information ⁇ t video . Therefore, the video time correction notification unit 118 receives the RTCP packet containing the correction time information ⁇ t video based on the change of the correction time information ⁇ t video by the server of each base R.
  • the video time correction notification unit 118 acquires the correction time information ⁇ t video and the sequence number s stored in the RTCP packet containing the correction time information ⁇ t video (step S232).
  • the video time correction notification unit 118 updates (s video_old , ⁇ t video_old ) and (s video_new , ⁇ t video_new ) based on the acquired correction time information ⁇ t video and sequence number s (step S233).
  • s video_old and s video_new are values based on the acquisition history of the sequence number s.
  • ⁇ t video_old and ⁇ t video_new are values based on the acquisition history of the corrected time information ⁇ t video .
  • step S233 the video time correction notification unit 118 updates (s video_old , ⁇ t video_old ) and (s video_new , ⁇ t video_new ) as follows.
  • the video time correction notification unit 118 sets ⁇ t video_new before update processing to ⁇ t video_old .
  • the video time correction notification unit 118 changes the update mode of s video_old based on the result of comparison between the sequence number s and s video_new and the result of comparison between the correction time information ⁇ t video and ⁇ t video_new .
  • the video time correction notification unit 118 sets the acquired sequence number s and correction time information ⁇ t video to (s video_new , ⁇ t video_new ).
  • FIG. 29 is a diagram showing an example of processing by the image time correction notification unit 118 of the server 1 at the site R according to the second embodiment.
  • the video time correction notification unit 118 does not update s video_old .
  • the video time correction notification unit 118 sets ⁇ t video_new (0) before update processing to ⁇ t video_old .
  • the video time correction notification unit 118 sets the acquired sequence number s(1) to s video_new .
  • the video time correction notification unit 118 sets the acquired ⁇ t video (0:00:01.100) to ⁇ t video_new .
  • the video time correction notification unit 118 sets ⁇ t video_new (0:00:01.100) before update processing to ⁇ t video_old .
  • the video time correction notification unit 118 sets the acquired sequence number s(4) to s video_new .
  • the video time correction notification unit 118 sets the acquired ⁇ t video (0:00:01.120) to ⁇ t video_new .
  • the video time correction notification unit 118 sets s video_new (6) before update processing to s video_old .
  • the video time correction notification unit 118 sets ⁇ t video_new (0:00:01.160) before update processing to ⁇ t video_old .
  • the video time correction notification unit 118 sets the acquired sequence number s(7) to s video_new .
  • the video time correction notification unit 118 sets the acquired ⁇ t video (0:00:01.100) to ⁇ t video_new .
  • FIG. 30 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing video V signal2 of the server 1 at the site O according to the second embodiment.
  • FIG. 30 shows a typical example of the processing of step S24 of the server 1.
  • the return video reception unit 113 receives the RTP packet containing the video V signal2 transmitted from the return video transmission unit 214 via the IP network (step S241).
  • the return video reception unit 113 acquires the video V signal2 stored in the RTP packet storing the received video V signal2 (step S242).
  • the return video reception unit 113 acquires the time T' stored in the RTP time stamp of the RTP packet storing the received video V signal2 (step S243).
  • the return video receiving unit 113 acquires the transmission source base R x (x is any one of 1, 2, . S244).
  • the return video receiving unit 113 calculates the time (T' - ⁇ t video ) obtained by subtracting the corrected time information ⁇ t video from the time T' based on the time T' and the corrected time information ⁇ t video (step S245).
  • the return video receiving unit 113 refers to the video synchronization control DB 131 and determines whether the video data x column related to the acquired transmission source site R x is empty among the records whose time T video matches the time (T' - ⁇ t video ). (step S246). If the video data x column related to the transmission source site R x is empty (step S246, YES), the process transitions from step S246 to step S247. If the video data x column related to the transmission source site R x is not empty (step S246, NO), the process transitions from step S246 to step S248.
  • the return video receiving unit 113 refers to the video synchronization control DB 131, and stores the video V signal2 in the video data x column related to the transmission source site R x among the records whose time T video matches the time (T'- ⁇ t video ). (step S247).
  • the processing in step S247 is an example of storing the video V signal2 in the video synchronization control DB 131 in association with the time T video related to the video V signal2 based on the time (T′ ⁇ ⁇ t video ).
  • the return video receiving unit 113 refers to the video synchronization control DB 131 and finds a record whose time T video matches the time ⁇ (T' - ⁇ t video_new ) + ( ⁇ t video_new - ⁇ t video_old )*(s video_new - s video_old ) ⁇ .
  • the image V signal2 is stored in the image data x column related to the transmission source site R x (step S248).
  • the processing in step S248 is an example of storing the video V signal2 in the video synchronization control DB 131 in association with the time T video related to the video V signal2 based on the time (T′ ⁇ ⁇ t video ).
  • FIG. 31 is a flow chart showing the voice processing procedure and processing contents of the server 1 at the site O according to the second embodiment.
  • the event audio transmission unit 115 transmits the RTP packet storing the audio A signal1 to the server of each site R via the IP network (step S30).
  • a typical example of the processing of the event sound transmission unit 115 in step S30 may be the same as the processing described in the first embodiment using FIG. 17, so description thereof will be omitted.
  • the event audio transmission unit 115 may store the time T audio in the RTP timestamp of the RTP packet instead of the header extension area of the RTP packet.
  • the audio time correction notification unit 119 receives the RTCP packet containing the correction time information ⁇ t audio from the server of each site R via the IP network (step S31). A typical example of the processing of step S31 will be described later.
  • the return audio receiving unit 116 receives the RTP packet containing the audio A signal2 from the server of each site R via the IP network (step S32).
  • the return audio receiving unit 116 stores the audio A signal2 in the audio synchronization control DB 132 based on the time obtained by subtracting the correction time information ⁇ t audio from the time T' stored in the RTP packet storing the audio A signal2.
  • the time T′ is an example of the acquisition time of the audio A signal2 acquired at the site R at the time when the audio A signal1 acquired at the site O at the time T audio is reproduced at the site R. A typical example of the processing of step S32 will be described later.
  • the turn-back audio synchronization control unit 117 simultaneously turns back the audio A signal1 related to a plurality of locations R among the locations R 1 to R n associated with one time T audio stored in the audio synchronization control DB 132. 104 (step S33).
  • a typical example of the processing of the turn-back audio synchronization control unit 117 in step S33 may be the same as the processing described in the first embodiment using FIG. 21, so description thereof will be omitted.
  • FIG. 32 is a flow chart showing the voice processing procedure and processing contents of the server 2 at the site R1 according to the second embodiment.
  • the event audio receiver 215 receives the RTP packet containing the audio A signal1 from the server 1 via the IP network (step S34).
  • a typical example of the processing of the event sound receiving unit 215 in step S34 may be the same as the processing described in the first embodiment using FIG. 18, and the description thereof will be omitted.
  • the event audio reception unit 215 may acquire the time T audio stored in the RTP timestamp of the RTP packet instead of the header extension area of the RTP packet.
  • the return audio transmission unit 216 transmits the RTP packet containing the audio A signal2 to the server 1 via the IP network (step S35). A typical example of the processing of step S35 will be described later.
  • the audio time correction transmission unit 219 transmits the RTCP packet containing the correction time information ⁇ t audio to the server 1 via the IP network (step S36). A typical example of the processing of step S36 will be described later.
  • FIG. 33 is a flow chart showing a transmission processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server 2 at the site R1 according to the second embodiment.
  • FIG. 33 shows a typical example of the processing of step S35 of the server 2.
  • the return audio transmission unit 216 acquires the audio A signal2 output from the return audio recording device 205 at regular intervals I audio (step S351).
  • the audio A signal2 is the audio acquired at the location R1 at the time when the audio presentation device 204 reproduces the audio A signal1 acquired at the location O at each time T audio at the location R1 .
  • Return audio transmission section 216 acquires time t3 , which is the absolute time at which audio A signal2 recorded by return audio recording device 205 is sampled.
  • time t3 is the time obtained by adding ⁇ (minimum) to the absolute time when the audio A signal2 was recorded. After the audio A signal2 is recorded, the audio A signal2 is sent from the return audio recording device 205 to the return audio transmission unit 216, and the return audio transmission unit 216 starts conversion processing from an analog signal to a digital signal. It is the time until Since ⁇ is infinitely close to 0, time t3 may be regarded as the same as the absolute time when audio A signal2 was recorded.
  • the return audio transmission unit 216 refers to the audio time management DB 232 and extracts records having audio data including the acquired audio A signal2 (step S352).
  • the sound A signal2 acquired by the return sound transmission unit 216 includes the sound A signal1 reproduced by the sound presentation device 204 and the sound generated at the base R1 (such as the cheers of the audience at the base R1 ).
  • the return voice transmission unit 216 separates two voices by a known voice analysis technique.
  • the return audio transmission unit 216 identifies the audio A signal1 reproduced by the audio presentation device 204 by separating the audio.
  • the return audio transmission unit 216 refers to the audio time management DB 232 and searches for audio data that matches the audio A signal1 reproduced by the identified audio presentation device 204 .
  • the return audio transmission unit 216 refers to the audio time management DB 232 and extracts a record having audio data that matches the audio A signal1 reproduced by the specified audio presentation device 204 .
  • the return audio transmission unit 216 refers to the audio time management DB 232 and acquires the time T audio in the audio synchronization reference time column of the extracted record (step S353).
  • the return audio transmission unit 216 generates an RTP packet containing the audio A signal2 (step S354).
  • step S354 for example, the return audio transmission unit 216 stores the acquired audio A signal2 in an RTP packet.
  • the return voice transmission unit 216 stores the time T' corresponding to the time t3 in the RTP timestamp of the RTP packet.
  • the time T' is the earliest time t3 among the times t3 regarding the audio A signal2 stored in the RTP packet.
  • the time T' may be regarded as the same as the absolute time when the audio A signal2 was recorded.
  • the RTP packet containing the audio A signal2 includes the sequence number s of the RTP packet header. To simplify the processing flow, the sequence number s is assumed to continue to be incremented for each generated RTP packet without returning to 0.
  • the return audio transmission unit 216 passes the acquired time T audio , time t 3 and sequence number s to the audio time correction transmission unit 218 (step S355).
  • the return audio transmission unit 216 transmits the RTP packet containing the generated audio A signal2 to the IP network (step S356).
  • FIG. 34 is a flow chart showing a transmission processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t audio of the server 2 at the site R1 according to the second embodiment.
  • FIG. 34 shows a typical example of the processing of step S36 of the server 2.
  • the audio time correction transmission unit 218 acquires the time T audio , the time t 3 and the sequence number s from the return audio transmission unit 216 (step S361).
  • the audio time correction transmission unit 218 calculates the time (t3 - Taudio ) by subtracting the time Taudio from the time t3 based on the time Taudio and the time t3 ( step S362 ).
  • the audio time correction transmission unit 218 determines whether or not the time (t 3 ⁇ T audio ) matches the current corrected time information ⁇ t audio (step S363).
  • the corrected time information ⁇ t audio is the value of the difference between the time t3 and the time T audio .
  • the current corrected time information ⁇ t audio is the value of the time (t 3 ⁇ T audio ) calculated before the time (t 3 ⁇ T audio ) calculated this time. Note that the initial value of the corrected time information ⁇ t audio is 0. If the time (t3 - Taudio ) matches the current corrected time information ⁇ taudio (step S363, YES), the process ends.
  • step S363 If the time (t 3 ⁇ T audio ) does not match the current corrected time information ⁇ t audio (step S363, NO), the process transitions from step S363 to step S364.
  • the fact that the time (t 3 ⁇ T audio ) does not match the current corrected time information ⁇ t audio corresponds to a change in the corrected time information ⁇ t audio .
  • the audio time correction transmission unit 218 generates an RTCP packet containing the correction time information ⁇ t audio (step S365).
  • step S365 for example, the audio time correction transmission unit 218 describes the updated correction time information ⁇ t audio using APP in RTCP.
  • the audio time correction transmission unit 218 generates an RTCP packet containing the correction time information ⁇ t audio .
  • the audio time correction transmission unit 218 describes the sequence number s regarding the updated correction time information ⁇ t audio using APP in RTCP.
  • the RTCP packet storing the corrected time information ⁇ t audio stores the sequence number s.
  • the audio time correction transmission unit 218 transmits the RTCP packet containing the generated correction time information ⁇ t audio to the IP network (step S366). Note that the audio time correction transmission unit 218 starts the processing illustrated in FIG. 34 before the return audio transmission unit 216 transmits the RTP packet containing the audio A signal2 . Therefore, the timing at which the audio time correction transmission unit 218 transmits the RTCP packet containing the corrected time information ⁇ t audio is temporally earlier than the return audio transmission unit 216 transmits the RTP packet containing the audio A signal2 .
  • the timing at which the audio time correction transmission unit 218 transmits the RTCP packet containing the corrected time information ⁇ t audio is temporally earlier than the return audio transmission unit 216 transmits the RTP packet containing the audio A signal2 .
  • FIG. 35 is a flow chart showing a reception processing procedure and processing contents of an RTCP packet storing the corrected time information ⁇ t audio of the server 1 at the site O according to the second embodiment.
  • FIG. 35 shows a typical example of the processing of step S31 of the server 1.
  • the audio time correction notification unit 119 receives the RTCP packet containing the correction time information ⁇ t audio from the server of each site R via the IP network (step S311). Note that, as described above, the audio time correction transmission unit 218 transmits to the server 1 an RTCP packet containing the correction time information ⁇ t audio based on the change in the correction time information ⁇ t audio . Therefore, the video time correction notification unit 118 receives an RTCP packet containing the correction time information ⁇ t audio based on the change of the correction time information ⁇ t audio by the server of each base R.
  • the audio time correction notification unit 119 acquires the corrected time information ⁇ t audio and the sequence number s stored in the RTCP packet storing the corrected time information ⁇ t audio (step S312).
  • the audio time correction notification unit 119 updates ( saudio_old , ⁇ taudio_old ) and ( saudio_new , ⁇ taudio_new ) based on the acquired correction time information ⁇ taudio and sequence number s (step S313).
  • s audio_old and s audio_new are values based on the acquisition history of the sequence number s.
  • ⁇ t audio_old and ⁇ t audio_new are values based on the acquisition history of the corrected time information ⁇ t audio .
  • the audio time correction notification unit 119 updates ( saudio_old , ⁇ taudio_old ) and ( saudio_new , ⁇ taudio_new ) as follows.
  • the audio time correction notification unit 119 sets ⁇ t audio_new before update processing to ⁇ t audio_old .
  • the audio time correction notification unit 119 changes the update mode of s audio_old based on the comparison result between the sequence number s and s audio_new and the comparison result between the correction time information ⁇ t audio and ⁇ t audio_new .
  • the audio time correction notification unit 119 sets the acquired sequence number s and corrected time information ⁇ t audio to (s audio_new , ⁇ t audio_new ).
  • FIG. 36 is a flow chart showing a reception processing procedure and processing contents of an RTP packet containing the voice A signal2 of the server 1 at the site O according to the second embodiment.
  • FIG. 36 shows a typical example of the processing of step S32 of the server 1.
  • the return voice receiving unit 116 receives the RTP packet containing the voice A signal2 transmitted from the return voice transmitting unit 216 via the IP network (step S321).
  • the return audio receiving unit 116 acquires the audio A signal2 stored in the RTP packet storing the received audio A signal2 (step S322).
  • the return audio receiving unit 116 acquires the time T' stored in the RTP timestamp of the RTP packet storing the received audio A signal2 (step S323).
  • the return audio receiving unit 116 acquires the transmission source site R x (x is any of 1, 2, . . . , n) from the information stored in the header of the RTP packet storing the received audio A signal2 (step S324).
  • the return audio receiving unit 116 calculates the time (T' - ⁇ t audio ) obtained by subtracting the corrected time information ⁇ t audio from the time T' based on the time T' and the corrected time information ⁇ t audio (step S325).
  • the return audio receiving unit 116 refers to the audio synchronization control DB 132, and among the records where the time T audio matches the time (T' - ⁇ t audio ), whether or not the audio data x column related to the acquired transmission source site R x is empty. (Step S326). If the voice data x column related to the transmission source site R x is empty (step S326, YES), the process transitions from step S326 to step S327. If the voice data x column related to the transmission source site R x is not empty (step S326, NO), the process transitions from step S326 to step S328.
  • the return audio receiving unit 116 refers to the audio synchronization control DB 132 and stores the audio A signal2 in the audio data x column related to the transmission source site R x among the records where the time T audio matches the time (T' - ⁇ t audio ). (step S327).
  • the processing in step S327 is an example of storing the audio A signal2 in the audio synchronization control DB 132 in association with the time T audio related to the audio A signal2 based on the time (T' - ⁇ t audio ).
  • the return audio receiving unit 116 refers to the audio synchronization control DB 132, and finds records whose time T audio matches the time ⁇ (T' - ⁇ t audio_new ) + ( ⁇ t audio_new - ⁇ t audio_old )*(s audio_new - s audio_old ) ⁇ .
  • the voice A signal2 is stored in the voice data x column related to the transmission source site R x (step S328).
  • the processing in step S328 is an example of storing the audio A signal2 in the audio synchronization control DB 132 in association with the time T audio related to the audio A signal2 based on the time (T' - ⁇ t audio ).
  • the server 1 stores the video V signal2 in the video synchronization control DB 131 based on the time (T' - ⁇ t video ).
  • the server 1 simultaneously outputs to the video presentation device 102 the video V signal2 related to a plurality of locations R associated with one time T video stored in the video synchronization control DB 131 .
  • the server 1 stores the audio A signal2 in the audio synchronization control DB 132 based on the time (T' - ⁇ t audio ).
  • the server 1 simultaneously outputs to the audio presentation device 104 the audio A signal2 related to the multiple sites R associated with one time T audio stored in the audio synchronization control DB 132 .
  • the server 1 based on the time (T' - ⁇ t video ) or the time (T' - ⁇ t audio ), at the same acquisition time of the video V signal1 or the audio A signal1 transmitted at different timings from the plurality of bases R Associated video V signal2 or audio A signal2 can be associated with each other.
  • the server 1 can simultaneously output video V signal2 or audio A signal2 for a plurality of locations R associated with one acquisition time.
  • the server 1 can appropriately synchronously reproduce a plurality of video/audio returned from a plurality of bases R through different transmission routes.
  • the server 1 receives an RTCP packet containing the corrected time information ⁇ t video based on the change of the corrected time information ⁇ t video by the server at the base R.
  • the server 1 receives the RTCP packet storing the corrected time information ⁇ t audio based on the change of the corrected time information ⁇ t audio by the server at the base R.
  • the server 1 can reduce the frequency of receiving RTCP packets storing the corrected time information ⁇ t video or RTCP packets storing the corrected time information ⁇ t audio .
  • the media synchronization control device may be realized by one device as described in the above example, or may be realized by a plurality of devices with distributed functions.
  • the program may be transferred while stored in the electronic device, or may be transferred without being stored in the electronic device. In the latter case, the program may be transferred via a network, or may be transferred while being recorded on a recording medium.
  • a recording medium is a non-transitory tangible medium.
  • the recording medium is a computer-readable medium.
  • the recording medium may be a medium such as a CD-ROM, a memory card, etc., which can store a program and is readable by a computer, and its form is not limited.
  • the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the gist of the invention at the implementation stage.
  • various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments.
  • constituent elements of different embodiments may be combined as appropriate.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
PCT/JP2021/025651 2021-07-07 2021-07-07 メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム Ceased WO2023281665A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/574,334 US20240321319A1 (en) 2021-07-07 2021-07-07 Media synchronization control apparatus, media synchronization control method, and media synchronization control program
PCT/JP2021/025651 WO2023281665A1 (ja) 2021-07-07 2021-07-07 メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム
JP2023532954A JPWO2023281665A1 (https=) 2021-07-07 2021-07-07

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/025651 WO2023281665A1 (ja) 2021-07-07 2021-07-07 メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム

Publications (1)

Publication Number Publication Date
WO2023281665A1 true WO2023281665A1 (ja) 2023-01-12

Family

ID=84800506

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/025651 Ceased WO2023281665A1 (ja) 2021-07-07 2021-07-07 メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム

Country Status (3)

Country Link
US (1) US20240321319A1 (https=)
JP (1) JPWO2023281665A1 (https=)
WO (1) WO2023281665A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003018567A (ja) * 2001-06-29 2003-01-17 Matsushita Electric Ind Co Ltd データ再生装置及びデータ送信装置
JP2006526967A (ja) * 2003-06-02 2006-11-24 クゥアルコム・インコーポレイテッド より高いデータレートのための信号プロトコルおよびインターフェイスの生成および実行
WO2011099273A1 (ja) * 2010-02-15 2011-08-18 パナソニック株式会社 コンテンツ通信装置及びコンテンツ加工装置並びにコンテンツ通信システム
JP2012054693A (ja) * 2010-08-31 2012-03-15 Pioneer Electronic Corp 映像音声送受信装置、映像音声送受信システム、コンピュータプログラム及び記録媒体

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7782363B2 (en) * 2000-06-27 2010-08-24 Front Row Technologies, Llc Providing multiple video perspectives of activities through a data network to a remote multimedia server for selective display by remote viewing audiences

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003018567A (ja) * 2001-06-29 2003-01-17 Matsushita Electric Ind Co Ltd データ再生装置及びデータ送信装置
JP2006526967A (ja) * 2003-06-02 2006-11-24 クゥアルコム・インコーポレイテッド より高いデータレートのための信号プロトコルおよびインターフェイスの生成および実行
WO2011099273A1 (ja) * 2010-02-15 2011-08-18 パナソニック株式会社 コンテンツ通信装置及びコンテンツ加工装置並びにコンテンツ通信システム
JP2012054693A (ja) * 2010-08-31 2012-03-15 Pioneer Electronic Corp 映像音声送受信装置、映像音声送受信システム、コンピュータプログラム及び記録媒体

Also Published As

Publication number Publication date
JPWO2023281665A1 (https=) 2023-01-12
US20240321319A1 (en) 2024-09-26

Similar Documents

Publication Publication Date Title
CN107018466B (zh) 增强音频记录
US7434154B2 (en) Systems and methods for synchronizing media rendering
JP6662063B2 (ja) 収録データ処理方法
CN104023250B (zh) 基于流媒体的实时互动方法和系统
KR101934200B1 (ko) 미디어 동기화 방법 및 시스템
JP3675739B2 (ja) デジタルストリームコンテンツ作成方法、デジタルストリームコンテンツ作成システム、デジタルストリームコンテンツ作成プログラム、このプログラムを記録した記録媒体、デジタルストリームコンテンツ配信方法
WO2016147176A1 (en) A method of controlling mobile devices in concert during a mass spectators event
WO2023281665A1 (ja) メディア同期制御装置、メディア同期制御方法及びメディア同期制御プログラム
US20220377409A1 (en) Distributed network recording system with single user control
JP7622844B2 (ja) メディア加工装置、メディア加工方法及びメディア加工プログラム
JP6275906B1 (ja) 動画コンテンツを再生するためのプログラム及び方法、並びに、動画コンテンツを配信及び再生するためのシステム
JP7586326B2 (ja) メディア加工装置、メディア加工方法及びメディア加工プログラム
JP6909903B1 (ja) 画像管理装置、画像管理システム及び画像管理方法
KR101811366B1 (ko) 미디어 재생 프로그램
CN111885345B (zh) 远程会议实现方法、装置、终端设备和存储介质
JP2015033061A (ja) コンテンツ再生方法、装置及びプログラム
JP2018038058A (ja) 音声出力装置、撮像システム、音声出力方法及びプログラム
WO2024057399A1 (ja) メディア再生制御装置、メディア再生制御方法及びメディア再生制御プログラム
WO2024057400A1 (ja) メディア再生制御装置、メディア再生装置、メディア再生方法及びプログラム
JP2007074684A (ja) 動画配信システム
EP4525449A1 (en) A method, an application, a device and a system for processing and presenting streams of a sensory signal
JP6909902B1 (ja) 画像管理装置、画像管理システム及び画像管理方法
JP2022185640A (ja) 視聴端末及びプログラム
KR102273795B1 (ko) 영상 동기화 처리를 위한 시스템 및 그 제어방법
KR102320670B1 (ko) 데이터 동기화 시스템 및 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21949299

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023532954

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18574334

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21949299

Country of ref document: EP

Kind code of ref document: A1