REMOTE VIRTUAL CONCERT SYSTEM AND METHOD
COPYRIGHT NOTICE A portion of the disclosure of this patent document contains material which is
subject to copyright protection. The copyright owner has no objection to the facsimile
reproduction by anyone of the patent document or of the patent disclosure as it appears in the
Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights
whatsoever.
BACKGROUND OF THE INVENTION The present invention relates to digital audiovisual transmission systems for live
events. More particularly, the present invention relates to a system and method for capturing a live performance and transmitting and presenting a virtual performance at one or more remote concert venues.
Live concert performances provide spectators with an experience which far
exceeds the enjoyment of listening to audio recordings or watching music videos or even the
concerts themselves on television. Live sporting events and other live performances have similar advantages which live concert performances have over recorded sporting events and
performances. Artists, performers or athletes impart a high energy level to their audience and the excitement and response of the audience in turn boosts their performance levels. This interaction
between live participants and audience makes for a unique and highly desirable experience.
With respect to musicians, live concert performances are important, not only as a
source of revenue but also to promote sales of their recordings and merchandise. Thus, after a
new recording is released, promoters try to have the artists performance in as many live concert
venues as possible.
However, to produce the greatest number of high quality live performances in
multiple venues, artists have heretofore been required to tour from venue to venue. Aside from
the artists themselves, each concert performance at each venue requires the participation of many
professionals, including producers, sound engineers, technical and pyrotechnic personnel and
road crew. Further, each change of venue for the concert performance requires movement of a
large number of instruments, amplifiers, monitors, stage props, concert set ornamentation,
pyrotechnics, light and laser light show equipment, and audio production and presentation
equipment.
Due to the extensive equipment movement, travel time, and set construction time
required to move between live concert venues, traditional multiple city concert touring has proven to be expensive for promoters and taxing on the performers and crew involved. What's more, artists usually tour only after an album is released, which may be only every three years or so on average. Consequently, the number of top acts available to tour each year is highly
uncertain. Moreover, fans in some cities often must wait for months, or even years, for the
performers to appear after the latest release of an album. Some cities are never reached because
the record company and performers are forced to make practical decisions and limit
performances to the most important cities for a particular tour. The fans in those venues would
be forced to travel long distances to the nearest venue; alternatively, the fans for those skipped
venues would wait for a lower quality video release sometimes produced after the concert tour to
be viewed in a standard movie theater or on a television screen providing a limited concert
experience, with no interaction between the performers and the viewing audience.
In a live concert performance, the experience created by the production and sound
engineers is crucial for audience enjoyment. Although attempts have been made to present a live
concert at a remote location using satellite feeds of audio and video signals captured at the live
concert, the resulting experience of the remote audience is not substantially better than simply
watching the concert on television. Also, satellite transponder time is scarce and very expensive
for these types of satellite feeds. Further, concert performers have heretofore not been able to
experience audience reactions from the remote venue, which, as explained above, is important to
the presentation of an effective concert experience.
There is thus a need for a system which provides the full experience of a live
performance to remote locations.
BRIEF SUMMARY OF THE INVENTION Accordingly, it is an object of this invention to solve the problems with existing
systems described above.
It is another object of this invention to provide a system and method for two-way
transmission and presentation of a live concert, sporting event or other performance.
It is another object of this invention to alleviate the need for musical performance
artists to travel to every venue to perform and promote their music.
It is another object of this invention to reduce the expense and delays associated
with artists performing in multiple venues.
It is another object of this invention to alleviate the need for fans from relatively
remote areas to travel long distances to experience concert performances because artists and
concert promoters do not find it economical or practical to set concert dates in those remote
areas.
It is another object of this invention to transmit and present a live audio and video
concert presentation at venues remote from the live performance with simulated live concert
audio, visual, and special effects so that audiences at remote venues will have a concert
experience similar to that presented at the live venue. It is another object of this invention to provide a system for collaboration between
audio and video capture of a live performance, and production, transmission, and presentation of
the performance at a remote venue.
It is another object of this invention to provide a quality presentation of a concert experience at remote venues.
It is another object of this invention to provide performing artists with interaction and instantaneous audio and visual feed back from audiences from remote venues.
It is another object of this invention to provide satellite or terrestrial broad-band alternative transmission mediums for transmission of a live performance.
The above and other objects are achieved by a system and related method
comprising capturing a live performance and transmitting and presenting a virtual performance at one or more virtual venues. The system includes a subsystem for capture and transmission of
high-quality audio, video and production signals from a live performance to one or more remote
venues for simulated presentation. The live venue where the artists or performers perform is
interconnected to a high-bandwidth two-way transmission system which distributes the signals to
the remote venues. The remote venues have high quality projectors, speakers and concert audio
and visual effect implements to enhance the virtual experience for the remote audiences.
For example, with respect to musical concerts, artists perform their musical works
on stage in front of a live audience or in a studio. Production personnel, stage crew and sound
engineers control lighting, laser light show effects, amplifier volumes, stage props, and audio
mixing for distributed speakers located in the live venue if the performance is in front of a live
audience. Pyrotechnic personnel may control fireworks and special explosive effects at the live
venue. The inputs used to control the audio, video and special effects are digitized, if not already
in digital format, and captured by the system. Alternatively, if the live performance is not in
front of a live audience, the system merely captures the inputs from the personnel at the live
venue for transmission to the remote venues.
High definition television (HDTV) video cameras capture video signals of the live performing artists at the live venue. HDTV video tape recorders (VTR) are attached to the system to archive the video signals from the HDTV cameras. Video production personnel
control the video mix of signals captured from the video cameras to be taped or transmitted to remote venues.
Audio signals from the artists' microphones and instruments are mixed by sound engineers or production personnel to create the concert's bowl audio effect for concert audiences.
The mixed audio signals are captured by the system.
The captured audio, video and production signals are multiplexed together for
transmission. The system then routes the multiplexed transmission signal to a two-way large
bandwidth digital transmission system to carry the signal to the remote venues. The transmission
system comprises a two-way long or short haul transmission system and may comprise either a
fiber optic, terrestrial microwave satellite connection or a combination thereof.
At each remote venue, transmitted signals are de-multiplexed back to their
separate components. The video signals are routed to HDTV video projectors for projection of
the video onto a large screen so as to simulate the actual live performers. The audio signals are
routed to various speakers so as to recreate the live venue's concert bowl audio effect for remote
audiences. Production data is processed and routed to machines wired to the system to control
concert effects similar to those at the live venue, such as lighting, laser light show effects, stage
props, and pyrotechnic effects. Production personnel and crew at the live venue may
communicate with personnel and crew at the remote venues who are aiding in simulating the
concert environment through talk back channels provided by the system.
Microphones and video cameras at the remote venue capture audio and video
signals of the remote audiences. The audio and video signals are routed through the transmission
system back to the live venue. At the live venue, the audio and video signals from remote venues are routed to audio and video monitors on stage to be presented to the performing artists.
Consequently, the artists performing at the live venue can see and hear real-time reaction to their performance from audiences at the remote venues and interact with them live.
Throughout the system, redundant hardened television production broadcast
components are used. Spare units for all critical path components are included at the live venue
and the remote venues. For example, spare cameras are used for capturing video signals at the live venue. Also, in case there is a loss of transmission between the live and a remote venue,
each remote venue has an HDTV master video of one of the previous recent performances of
artists which can be presented to audiences at the remote venue. Further, projectors at the remote
venues have cumulative presentation. The brightness of the normal image projected onto the
screen is the product two of HDTV projectors. If one of the cumulative HDTV projectors fails,
the second cumulative projector would already be calibrated in place projecting the show so that
the only change in the concert video experience is that the image becomes less bright.
BRIEF DESCRIPTION OF THE DRAWINGS The invention is illustrated in the figures of the accompanying drawings which are
meant to be exemplary and not limiting, in which like references refer to like corresponding
parts, and in which: Fig. 1 contains a block diagram showing primary components of a system of the
present invention with a live venue, a transmission system, and remote venues;
Fig. 2 contains a block diagram showing the system of Fig. 1 in greater detail;
Fig. 3 contains a flowchart representing the signal flow of the system of Figs. 1
and 2; and Fig. 4 contains a block diagram showing components of a virtual ad insertion system.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
With reference to Fig. 1, one preferred embodiment of the invention includes live venue 100, wherein artists 128 perform live in concert, with or without a live audience, and
includes one or more two-way transmission systems 200 for transmitting audio and video signals
from the live venue to one or more remote venues 300 where remote audiences may view the live
concert remotely. In accordance with the present invention, a live concert performance is
performed, produced and mixed by video and sound engineers, and producers use production
equipment located at the live venue 100. Audio, video and production signals are then routed
through transmission system 200 to one or more remote venues 300 for presentation to remote
audience 320. Production personnel and sound engineers can provide the proper audio mix and
video presentation to closely simulate the live concert experience to remote audience 320
through use of the system of the present invention, as described below.
At live venue 100 artists 128 perform their musical works on a stage 110, which
may be accessorized with advertising sinage 338, in view of a live audience 104 or in a studio.
At live venue 100, production personnel, stage crew and sound engineers control lighting, laser
light show effects, amplifier volumes, stage props, and audio mixing for distributed speakers 106
located in the live venue. Pyrotechnic personnel may also be needed to control fireworks and
special explosive effects at the live venue.
With reference to Fig. 2, four high definition television (HDTV) video cameras 112 are used at live venue 100 for capturing video signals of the live performance. One of the
HDTV cameras 112 is used as a primary HDTV camera for establishing the basic center stage view of the live performance. Other HDTV cameras 1 12 are mounted on dollies, booms and in point of view areas to capture a full video experience. The video signals from the cameras are routed to an HDTV console/switch 114. Three standard D5 HDTV video tape recorders (VTR)
116 are attached to the video console/switch 114 for capturing simultaneously an archive copy of the video signals from three of the HDTV cameras 1 12. Video production personnel control the
video mix of signals captured from video cameras 112 to be taped or transmitted to remote venue
300.
In a concert performance, the artists sing into microphones 130 and perform on
electric instruments with pick-ups which produce audio signals. The system of the present
invention captures the signals which are mixed into 24 to 64 audio channels using a primary
audio mixing board 118. Sound engineers or production personnel mix the audio signals to
create the concert's bowl or surround sound audio effect for concert audiences with primary
audio mixing board 118. The mixed audio signals are routed to a multi-channel audio mixing
board 120 connected in slave relationship to primary audio mixing board 118. In multi-channel
audio mixing board 120, the 24 to 64 channels fed from primary mixing board 118 are mixed
down to either six multi-channel plus 2 stereo outputs, or 8 multi-channel close to stereo outputs,
or 10 multi-channel plus two stereo audio outputs.
An HDTV audio and video transmitter/receiver 122 is linked to a two-way large
bandwidth digital transmission system 200 comprising either a fiber optic carrier 202, terrestrial
microwave carrier 204, or a satellite transmission system 306. Transmitter/receiver 122 may also comprise a digital compressor-decompressor (codec) for compressing outgoing signals, and
decompressing incoming signals. The transmitter/receiver 122 digitizes any analog signals
routed to transmitter/receiver 122, and converts some incoming signals to analog as required by some of the system's components. The audio and video signals are multiplexed together for transmission.
RS 422 data and talk back channels, and other assorted data channels are multiplexed in with the audio and video signals. All of the inputs from sound engineers, light show production crew, laser light production crew, pyrotechnic crew, and other production
personnel used to control lighting, laser light show effects, amplifier volumes, stage props, audio
mixing, video control commands, and pyrotechnic effects are captured from the respective
production equipment. An RS 422 talk back channel may be dedicated to provide for
communications between production personnel at remote venue 300 and live venue 100. The
production data and RS 422 talk back channels are combined into one digital signal by
transmitter/receiver 122, which multiplexes the signals in with the audio and video signals to
form one 270 Mb serial digital interface (SDI) signal to be routed through two-way transmission
system 200.
Two-way transmission system 200 is typically digital, and is either long or short
haul, and uses multiplexing or statistical multiplexing to maximize bandwidth. The system is
flexible to allow the audio and video signals to be passed over multiple wide area network
topologies. In many cases, the live venue 100 and a remote venue 300 are located in the same
metropolitan area. For those instances, dark fiber 202 is employed to create a point-to-point fiber
connection between the two venues. In other cases, where the distances are greater, such as from city to city, the transmission system comprises a public fiber network 202 such as a Sonet
standard point-to-point fiber network provided by Metromedia Fiber Network, ICG Telecom
Group, or other local regional bell operating companies (RBOC). Long-haul fiber carriers, such as QWest and Williams CommunicationsNyvx can provide fiber service for longer distances where, for example, a remote venue 300 is located in a different country than live venue 100. Where fiber installations are not available, a terrestrial microwave carrier 204 is used between venues to reach the closest available fiber head ends. In other cases, especially where the concert
is transmitted concurrently to multiple remote venues, transponder time slots will be rented and local uplink and downlink trucks will connect directly to HDTV audio and video
transmitter/receivers 122 and 310 to implement satellite transmission system 206.
At each remote venue 300, HDTV audio and video transmitter/receiver 310 is
coupled to two-way transmission system 200. The signal is de-multiplexed into audio, video
production signals, and RS 422 data and talk back channels. The video signals are routed to an
HDTV video console/switch 312. Video console/switch 312 routs the video signals to a line
quadrupler/video de-interlacer 334 which enhances the quality of the picture signal The signals
are then routed to native resolution HDTV projectors 304 for presentation of the live video
signals onto a large screen 314 so as to simulate the actual size of the performers. Preferably,
projectors 304 are capable of native HDTV resolutions to project images of at least 480 by 1920
lines (480P), 720 by 1920 lines (7202P) progressively scanned, or 1080 by 1920 lines interlaced
(1080i). The video signals remain in a standard digital format until they reach the HDTV
projectors, where they are converted to a red-green-blue (RGB) signal for the projectors.
At each remote venue 300, the de-multiplexed audio signals and any sound
production signals from the RS 422 channels or data channels are routed from
transmitter/receiver 310 to a multi-channel audio mixing board 316. The de-multiplexed audio
signals comprise a minimum of eight digital audio channels. In audio mixing board 316, the
sound production control signals are processed in order to steer the audio signals to various speakers 318 so as to create the concert's bowl or surround sound audio effect for concert audience 320 at remote venue 300 similar to the effect created for audience 104 at live venue 100. Before being routed to the final power amplifiers, sophisticated digital signal processors, such as OMNIDRIVE processors by BSS, are used to process the audio signals to provide a
synchronized and consistent audio experience for every member of remote venue audience 320.
A sub-woofer 332. may also provide base tone enhancement for remote venue audience 320.
State-of-the-art speaker systems such as those available from MYERS Sound Laboratories, EASTERN Acoustic Works, and JBL are used to create a premium audio experience for remote
venue audience 320. State-of-the-art surround sound systems such as systems offered by DOLBY Laboratories may also provide surround sound effects at remote venue 300.
At remote venue 300. data from the de-multiplexed RS 422 channel and other data
channels are processed and routed to machines wired to the system to control concert effects
similar to those at live venue 100 such as lighting, laser light show effects, stage props, and
pyrotechnic effects. The RS 422 channels or data channels may also comprise communication
channels used so that production personnel and crew at live venue 100 may communicate with
personnel and crew helping stage the concert effects and production at remote venue 300 through
the RS 422 talk back channels. The outgoing signals for the talk back channels are digitized in
transmitter/receiver 310 and multiplexed for transmission. Signals received through the talk-
back channels are de-multiplexed and de-digitized in transmitter/receiver 310.
The inputs from the light production crew may also include signals for
synchronizing lights at each remote venue 300 with lights and effects at live venue 100. Personal computer (PC) controllers for controlling movable onstage background, spotlight and arena
lights at live venue 100 are linked through the RS 422 data channels to PC controllers at remote venue 300 for controlling movable arena lights at live venue 300. The software and firmware of the PC controllers at live venue 100 and each remote venue 300 synchronize movement of lights at live venue 100 with lights at each remote venue 300, while preventing lights at each remote
venue 300 from conflicting with projection by HDTV projectors 304.
The advertising signage 338 at live venue 100 may display a different sponsor
than the sponsor for one or more remote venues 300. The signage at live venue 100 is inevitably captured into the live venue video signals which are transmitted to each remote venue 100. The
sponsor for a remote venue 300 may wish to have their own signage 342 displayed at remote
venue 300 instead of the signage 338 captured into the live venue video signals. With reference
to Fig. 4, a virtual ad insertion system 402 located at live venue 100 can electronically replace
the captured images of the live venue sponsorship signage 338 with a remote venue sponsor's
signage 342, or seamlessly insert synthesized messages or logos onto the speaker grills or other
empty spaces on screen.
Virtual ad insertion systems 402, such as CYBERSET O offered by ORAD Hi- Tec Systems of New York, New York, are available to insert advertising signage into video
signals. For systems which operate with National Television System Committee (NTSC) standard video signals, the live venue HDTV video signals captured at live venue 100 by HDTV
video cameras 1 12 are down-converted to NTSC by an HDTV to NTSC down-converter 404 for
ad insertion by virtual add insertion system 402. The live venue video signals are then up- converted back to HDTV by an NTSC to HDTV up-converter 406. The live venue video signals
are then routed through transmission system 200 to each remote venue 300 for projection by HDTV projectors 304 (Fig. 2) on to screen 314. The ad insertion system works similarly with systems which operate with Phase Alteration by Line (PAL) standard video signals. For ad
insertion systems which are HDTV compliant, the systems are not converted, and ad insertion is performed directly on the live venue HDTV video signals.
At each remote venue 300, a DI video camera 322 captures a video signal from
audience 320. The video signal from video camera 322 is routed through video console/switch
312. and then routed to audio and video transmitter/receiver 310. A microphone 324 at remote
venue 300 captures audio signals from audience 320. The audio signal from microphone 324 is routed to a mixer 326 to process the audio signal. The audio signal is then routed to
transmitter/receiver 310. The video signal from video camera 322 and the audio signal from
microphone 324 are then multiplexed and routed through transmission system 200 back to live
venue 100. At live venue 100, the audio and video signals from remote venue 300 are de¬
multiplexed. The video signal is routed to an on-stage video monitor 124 so that artists 128
performing at live venue 100 can see reaction to their performance from audience 320. The
audio signal is routed to an on-stage audio monitor 126 so that artists 128 can hear reaction to
their performance from audience 320. If multiple remote locations are involved, separate audio
and video monitors are provided, or a single set of audio and video monitors is shared by
displaying the remote venue video signals in frames and switching between the various audio
signals.
Throughout the system television production broadcast components are used
which are hardened, or designed to be fault tolerant to the best commercial practice standards of
the television broadcast industry. In addition, redundancy in components is used at the live and remote venues. Spare units for all critical path components are included. For instance, spare
cameras 112 are used for capturing video signals at live venue 100. Another redundant feature of the system comprises providing a remote D5 master
HDTV video 330 recorder at each remote venue. In the event of transmission loss between the two sites, a high-definition D5 master HDTV video recorder 330 of one of the previous recent performances of artists 128 can be shown to audience 320 at remote venue 300 despite a catastrophic failure in transmission system 200. While the live performance is being presented at
remote venue 300, personnel at remote venue 300 track the live performance against the previous
performance video which contains the same order of songs in the artists' live musical set. As
soon as the transmission loss is detected, input into projectors 304 is provided from HDTV video 330. The audience merely notices a short pause in the video presentation on screen 314 while the
system is switched over to accept signals from HDTV video 330 instead transmission system
200.
Another redundant feature of the system comprises providing projectors 304 at
remote venue 300 with a fail safe mode. In some cases, the HDTV projectors have cumulative
presentation brightness of the normal image projected onto screen 314. Thus the image on
screen 314 is the product of two HDTV projectors 304. If one of the cumulative HDTV projectors 304 fails, the second cumulative projector would already be calibrated in place
projecting the show so that the only change in the concert video experience presented on screen
314 is that the image becomes less bright. With reference to Fig. 3, a flow chart provides an illustration of the flow of audio,
video and production data through the system. The system captures live venue audio signals of a live performance at live venue 100, step 500, and allows the audio signals to be mixed, step 502.
Live venue video signals of a live performance at live venue 100 are also captured, step 504. The live venue audio and video signals are stored at live venue 100 for later presentation of the live performance, step 506. The system accepts further production and special effects signals, step 508, at live venue 100.
The live venue audio, video, and extra production and special effects signals are then multiplexed, as incoming signals from remote locations are de-multiplexed, step 510. The
multiplexed live venue audio, video, and extra production and special effects signals are
transmitted to remote venue 300, step 512. The live venue audio, video, and extra production
and special effects signals are de-multiplexed at remote venue 300, step 514.
At remote venue 300, the live venue audio signals are processed, step 516. The
processed live venue audio signals are then presented to remote audience 320, step 518. The
special effects signals are processed, step 520, and the special effects are presented, step 522, at
remote venue 300. The live venue video signals are presented, step 524, at remote venue 300.
At remote venue 300, remote venue audio and video signals of audience 320 are
captured, step 526. The remote venue audio and video signals are multiplexed, as incoming
signals (from the live location) are de-multiplexed, step 514. The remote venue audio and video
signals are transmitted from remote venue 300 to the live venue 100 as signals from live venue
100 are transmitted to remote venue 300. step 512. The remote venue audio and video signals
are de-multiplexed as live venue signals are multiplexed, step 510. The remote venue audio and video signals are presented to the artists performing live at remote venue 100, step 528.
As a result, use of the system described above provides a near realistic, interactive
concert experience for audience and performer alike. The audience receives a complete concert experience that almost precisely duplicates the high quality sound presented to the live venue
audience and the various visual and audio effects which enhance the concert. The audience and performers interact on a real time basis, and the performer receives the benefit of the audience reaction. The performers also need not travel to as many remote locations to provide multiple
concerts.
The above example describes the system of the present invention in relation to transmitting a live concert performance. However, one skilled in the art would recognize that the
system could be applied to transmit other types of performances such as live sporting events or
plays.
While the invention has been described and illustrated in connection with
preferred embodiments, many variations and modifications as will be evident to those skilled in
this art may be made without departing from the spirit and scope of the invention, and the
invention is thus not to be limited to the precise details of methodology or construction set forth
above as such variations and modification are intended to be included within the scope of the
invention.