KR100641654B1 - Apparatus and Method for Transmitting and Recieving Audio Visual Tele-presence - Google Patents

Abstract

1. TECHNICAL FIELD OF THE INVENTION

The present invention relates to a remote virtual reality sound / video transmission and reception apparatus and a method thereof.

2. The technical problem to be solved by the invention

The present invention provides a remote virtual reality sound / video transmission and reception apparatus and method for acquiring and transmitting an image and a surround sound which is spatially synchronized with the image according to the user's control by setting a predetermined reference viewpoint. Its purpose is to.

3. Summary of Solution to Invention

The present invention includes a omnidirectional image acquisition means for acquiring omnidirectional images on a horizontal plane and a vertical plane by setting a specific direction in space as a reference viewpoint; Omnidirectional sound acquiring means for acquiring an omnidirectional sound signal spatially synchronized with the omnidirectional image acquired by the omnidirectional image acquiring means based on the reference viewpoint; And encoding transmission means for encoding and transmitting the obtained omnidirectional image and sound.

4. Important uses of the invention

The present invention is used in a virtual reality system and the like.

Virtual Reality, Tele-presence, Sync, Surround Sound

Description

Apparatus and Method for Transmitting and Recieving Audio Visual Tele-presence             

1 is a block diagram of an embodiment of a remote virtual reality sound / video transmission and reception apparatus according to the present invention,

2 is a detailed configuration diagram of the remote virtual reality sound / video transmission apparatus of FIG.

FIG. 3 is a diagram for describing a method of generating a sound track in the omnidirectional sound image generation unit of FIG. 2; FIG.

4 is a conceptual diagram of a service provided by a remote virtual reality sound / video receiving device according to the present invention;

5 is a flowchart illustrating an embodiment of a method for transmitting a remote virtual reality sound / video according to the present invention;

6 is a flowchart illustrating an embodiment of a method for generating surround sound images according to the present invention;

7 is a flowchart illustrating a remote virtual reality sound / video receiving method according to the present invention.

* Explanation of symbols for the main parts of the drawings

110: omnidirectional sound acquisition unit 120: omnidirectional image acquisition unit

130: encoding transmission unit 140: receiving decoding unit

150: surround image control unit 160: view control unit

170: omnidirectional sound playback unit 180: omnidirectional video playback unit

The present invention relates to a remote virtual reality audio / video transmission and reception apparatus and a method thereof, and more particularly to a remote virtual for transmitting and receiving a video that can be changed by the user control and surround sound capable of user control spatially synchronized with it The present invention relates to a realistic audio / video transmission and reception apparatus and a method thereof.

Recently, when a user selects a desired view point from a panorama image of an arbitrary place, a technology for allowing a user to view an image of the selected view point has been developed and used.

In general, a method for acquiring a panoramic image includes a method in which a plurality of cameras are radially arranged and properly synthesized for each camera to produce a single connected panoramic image. There is a way to get a panoramic view of the surroundings at once.

However, such remote virtual reality technology does not include an audio signal, and when the audio signal includes an audio signal, spatial synchronization is required as well as temporal synchronization with an image scene whose viewpoint is changed by user control.

At this time, in order to solve the spatial synchronization with the video scene, it is necessary to obtain a high-resolution spatial sound using as many microphones as possible. However, there is a problem in that acquisition and processing of spatial sound by many microphones is not easy in reality.

The present invention has been proposed to solve the above problems, and by setting a predetermined reference time point, a remote virtual reality for acquiring and transmitting and receiving the image and the surround sound that is spatially synchronized with the image that can be changed according to the user's control It is an object of the present invention to provide an audio / video transmitting and receiving device and a method thereof.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

According to an aspect of the present invention, there is provided a remote virtual reality sound / video transmission apparatus comprising: omnidirectional image acquisition means for acquiring omnidirectional images on a horizontal plane and a vertical plane by setting a specific direction in space as a reference viewpoint; Omnidirectional sound acquiring means for acquiring an omnidirectional sound signal spatially synchronized with the omnidirectional image acquired by the omnidirectional image acquiring means based on the reference viewpoint; And encoding transmission means for encoding and transmitting the obtained omnidirectional image and sound.

In addition, the apparatus of the present invention, a receiving apparatus for receiving a omnidirectional sound / video signal transmitted by setting a reference time point, receiving apparatus for receiving and demultiplexing and decoding the omnidirectional sound / video signal; Viewpoint control means for changing a viewpoint of an image decoded by the reception decoding means according to a user's control; Image reproducing means for reproducing an image corresponding to the changed time point; Surround sound image control means for synchronizing sound spatially with the image of the changed viewpoint based on the reference viewpoint; And sound reproducing means for reproducing the synchronized sound.

On the other hand, the method of the present invention, remote virtual reality sound / image transmission method, the sound / image acquisition step of acquiring omnidirectional sound and image from the outside according to a predetermined reference time; A surround sound image generating step of generating a surround sound image by generating sound tracks at equal intervals in the horizontal / vertical direction from the obtained omnidirectional sound; An encoding step of encoding an image acquired in the sound / image acquisition step and the generated surround sound image; And a multiplexing transmission step of multiplexing and transmitting the encoded image and the surround sound image.

In addition, the method of the present invention comprises a demultiplexed decoding step of demultiplexing and decoding the received sound / video signal, wherein the method receives a virtual sound / video signal to which a reference viewpoint is set and transmitted; A viewpoint control step of changing a viewpoint of an image according to a user's control; An image reproducing step of reproducing an image corresponding to the changed time point; And a sound reproducing step of reproducing a sound spatially synchronized with the image of the changed time point based on the reference time point.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an embodiment of a remote virtual reality sound / video transmission and reception apparatus according to the present invention.

As shown in FIG. 1, the apparatus for transmitting a remote virtual reality sound / video according to the present invention includes an omnidirectional sound acquisition unit 110, an omnidirectional image acquisition unit 120, and an encoding transmission unit 130. The audio / video receiving apparatus includes a reception decoder 140, a surround sound controller 150, a viewpoint controller 160, an omnidirectional sound reproducing unit 170, and an omnidirectional image reproducing unit 180.

The omnidirectional sound acquisition unit 110 acquires omnidirectional sound signals on the horizontal plane and the vertical plane through the microphone array, and the omnidirectional image acquisition unit 120 acquires the omnidirectional image on the horizontal plane and the vertical plane through a plurality of cameras or optical lenses. At this time, the omnidirectional sound and the omnidirectional image should be spatially synchronized, which can be solved by predetermining a specific direction in the reference direction and corresponding to the difference in relative angles.

In addition, the encoding transmitter 130 encodes and multiplexes the omnidirectional sound / image obtained by the omnidirectional sound acquirer 110 and the omnidirectional image acquirer 120.

Meanwhile, the reception decoder 140 receives and demultiplexes and decodes the encoded sound / image transmitted from the remote virtual reality sound / video transmission apparatus, and the viewpoint controller 160 changes the viewpoint of the image under the control of the user. do.

The omnidirectional image reproducing unit 180 reproduces an image corresponding to the changed viewpoint by the viewpoint controller 160, and the surround sound controller 150 spatially changes the image viewpoint when the viewpoint of the image is changed by an arbitrary angle under the control of the user. Synchronize the sound.

In addition, the omnidirectional sound reproducing unit 170 reproduces the surround sound image synchronized with the image reproduced by the omnidirectional image reproducing unit 180 under the control of the surround sound image control unit 150. In this case, as a method for reproducing the surround sound image, a virtual speaker by a head related transfer function (HRTF) is generated at the position of the sound image, or a phantom image is generated by the energy combination of the multichannel speakers, or a large number of speaker arrays are used. Wavefront synthesis method can be used.

FIG. 2 is a detailed block diagram of the remote virtual reality sound / video transmission device of FIG. 1.

As shown in FIG. 2, the apparatus for remote virtual reality sound / video transmission according to the present invention includes a multichannel sound acquirer 210, an image generator 220, an audio encoder 230, and an image acquirer 240. , An image corrector 250, an image encoder 260, and a multiplexer 270.

First, the multichannel sound acquisition unit 210 converts a sound input through the microphone array 280 into a multichannel sound, and the image generator 220 generates a multichannel sound generated by the multichannel sound acquisition unit 210. Is converted into sound tracks evenly spaced in the horizontal plane or sound tracks extended to the vertical plane to create a surround sound image that is easily controlled by the user.

At this time, the microphone array 280 is disposed on a sphere of a solid state in order to reduce the cross-correlation between the input sound, the number of the microphone is determined to be two or more according to the purpose, in consideration of the 5.1-channel audio reproduction environment The microphone can be placed at the corresponding position.

In addition, when the number of mitrophones is greater than the number of surround sound images to be generated, the sound image generator 220 generates surround sound images by a combination of signals obtained through the microphones, and the number of microphones is to be generated. If the number is the same as the number of surround sound images and the direction is the same, it is used as it is obtained through the microphone, and if the number of microphones is smaller than the number of surround sound images to be generated, energy and time between channels are utilized by using the position information of the microphone. The difference produces an acoustic signal corresponding to each surround sound image.

In addition, the sound image generator 220 compares the sound signals obtained through the microphone to generate the omnidirectional sound image, calculates the cross-correlation, the level difference between the channels and the delay difference between the channels, and calculates each calculated value of each microphone. Based on the mutual position analysis to extract the positions of the sound source signals in the space, based on the extracted position of the sound source signals in the spatial mix signals of the transmission track as shown in Figure 3 to be described later to each sound source signals.

At this time, the omnidirectional sound must be spatially synchronized with the image signal, which can be solved by setting a specific viewpoint as a reference viewpoint and corresponding to the relative angle.

The acoustic encoder 230 encodes a signal generated by the image generator 220. In this case, as a coding method, conventional multichannel audio encoding methods such as MPEG-2, MPEG-4, and AC-3 can be used.

The image acquirer 240 acquires an omnidirectional image through the omnidirectional camera 290 equipped with a plurality of cameras or optical lenses. In this case, since the omnidirectional sound must be spatially synchronized with the omnidirectional image signal, the image acquisition unit 240 acquires the omnidirectional image by setting a reference viewpoint in the same direction as the sound generation unit 220.

The image corrector 250 corrects the obtained omnidirectional image. That is, the image correcting unit 250 removes the discontinuous surface or corrects the distorted image so that the image can be naturally reproduced on the receiving side.

Also, the image encoder 260 encodes the corrected omnidirectional image. In this case, as a coding method, conventional coding methods such as MPEG-2 and MPEG-4 are used.

The multiplexer 270 multiplexes and transmits the signals encoded by the audio encoder 230 and the image encoder 260. At this time, multiplexing methods such as MPEG-2 and MPEG-4 are used as the multiplexing methods.

FIG. 3 is a diagram for describing a method of generating a sound track in the omnidirectional sound image generating unit of FIG. 2.

As shown in FIG. 3, the sound image generating unit 220 converts the multi-channel sounds obtained through the microphone array 280 to generate sound tracks on the horizontal and vertical surfaces at equal intervals.

4 is a conceptual diagram of a service provided in a remote virtual reality sound / video receiving device according to the present invention.

As shown in FIG. 4, the user 410 may select a viewpoint of a desired image, the viewpoint controller 160 of the receiving apparatus changes the viewpoint of the image under the control of the user, and the omnidirectional image playback unit 180 Plays the video corresponding to the changed time.

At this time, the sound is also synchronized with the image, and if the viewpoint of the image is changed, the sound is also changed. That is, the surround sound controller 150 synchronizes the sound spatially to the changed image viewpoint when the viewpoint of the image is changed by an arbitrary angle by the user's control, and the omnidirectional sound reproduction unit 170 controls the surround sound controller 150. As a result, the surround sound image synchronized with the image reproduced by the omnidirectional image reproducing unit 180 is reproduced.

5 is a flowchart illustrating a remote virtual reality sound / video transmission method according to the present invention.

As shown in FIG. 5, first, an omnidirectional sound and an image are acquired from the outside according to a predetermined reference time point (510). That is, the omnidirectional sound and the image are acquired based on the reference viewpoint to obtain the spatially synchronized sound and the image.

Subsequently, an equally spaced sound track is generated in the horizontal / vertical direction from the obtained omnidirectional sound to generate a surround sound image (520).

Next, the obtained image and the generated surround image are encoded (530), and the encoded surround image and the image are multiplexed and transmitted (540).

6 is a flowchart illustrating a method of generating a surround sound image according to the present invention.

As shown in FIG. 6, first, a multichannel sound is generated from the obtained omnidirectional sound (610).

Subsequently, the positions of the sound source signals in the space are extracted by comparing the generated multichannel sounds to calculate cross-correlation, level differences between channels, and delay differences between channels.

A surround sound image is generated by mixing the signals of the transport track with the respective sound source signals based on the positions of the extracted sound source signals in the spatial space (630).

7 is a flowchart illustrating a remote virtual reality sound / video receiving method according to the present invention.

As shown in FIG. 7, first, a reference view is set, and the received audio / video signal is demultiplexed and decoded (710).

Subsequently, the viewpoint of the image is changed (720) under the control of the user, and the image corresponding to the changed viewpoint is reproduced (730).

Subsequently, based on the reference viewpoint, a sound spatially synchronized with the image of the changed viewpoint is reproduced (740). That is, when the image viewpoint is changed by an angle from the reference viewpoint by the user's control, the sound of the viewpoint corresponding to the changed image viewpoint, that is, the angle changed from the reference viewpoint is reproduced.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

The present invention as described above, by controlling the direction of the omnidirectional sound and image, it is possible to experience the current situation of the tourist destination or remote place in a realistic sense, there is an effect to enable the search and monitoring of any place by the omnidirectional sound and image.

Claims (11)

Omnidirectional image acquisition means for acquiring omnidirectional images on a horizontal plane and a vertical plane by setting a specific direction in space as a reference viewpoint; Omnidirectional sound acquiring means for acquiring an omnidirectional sound signal spatially synchronized with the omnidirectional image acquired by the omnidirectional image acquiring means based on the reference viewpoint; And Encoding transmission means for encoding and transmitting the obtained omnidirectional image and sound Remote virtual reality sound / video transmission device comprising a. The method of claim 1, The omnidirectional sound acquisition means, A multichannel sound acquisition unit for generating a multichannel sound from the omnidirectional sound input from the outside; And Set a specific direction in space as a reference viewpoint and convert the multi-channel sound generated from the multi-channel sound acquisition unit to equally space in the horizontal / vertical direction spatially synchronized with the omnidirectional image obtained by the omnidirectional image acquisition means. Omni-directional sound generator for generating sound tracks Remote virtual reality sound / video transmission device comprising a. The method of claim 2, The omnidirectional sound image generation unit, Comparing sound signals to calculate cross-correlation, level difference between channels, and delay difference between channels to extract positions of sound source signals in space, and based on the positions of the extracted sound source signals, the signals of the transmission tracks are assigned to the respective sound source signals. A remote virtual reality sound / video transmission device, characterized in that for mixing. The method according to any one of claims 1 to 3, Omnidirectional image correction means for correcting an omnidirectional image obtained through the omnidirectional image acquisition means; Remote virtual reality sound / video transmission device further comprising. A receiving device for receiving an omnidirectional sound / video signal transmitted by setting a reference time point, A reception box lake end for receiving, demultiplexing, and decoding the omnidirectional audio / video signal; Viewpoint control means for changing a viewpoint of an image decoded by the reception decoding means according to a user's control; Image reproducing means for reproducing an image corresponding to the changed time point; Surround sound image control means for synchronizing sound spatially with the image of the changed viewpoint based on the reference viewpoint; And Sound reproducing means for reproducing synchronized sounds Remote virtual reality sound / video receiving device comprising a. The method of claim 5, The sound reproducing means, Remote virtual reality sound / video receiving apparatus characterized in that to reproduce the sound by generating a virtual seeker by the HRTF (Head Related Transfer Function) in the position of the sound image. A sound / image acquisition step of acquiring omnidirectional sounds and images from outside according to a predetermined reference time point; A surround sound image generating step of generating a surround sound image by generating sound tracks at equal intervals in the horizontal / vertical direction from the obtained omnidirectional sound; An encoding step of encoding an image acquired in the sound / image acquisition step and the generated surround sound image; And Multiplexed transmission step of multiplexing and transmitting the encoded video and surround sound image Remote virtual reality sound / video transmission method comprising a. The method of claim 7, wherein The surround sound image generation step, Generating a multichannel sound from the obtained omnidirectional sound; Comparing the generated multichannel sounds to calculate cross-correlation, level differences between channels, and delay differences between channels to extract positions of sound source signals in space; And Generating a surround sound image by mixing the signals of the transport track with the respective sound source signals based on the positions of the extracted sound source signals in the space; Remote virtual reality sound / video transmission method comprising a. The method according to claim 7 or 8, Image correction step of correcting the omnidirectional image obtained in the sound / image acquisition step Remote virtual reality sound / video transmission method further comprising. In the method for receiving a virtual sound / video signal transmitted by setting a reference time point, A demultiplexing decoding step of demultiplexing and decoding the received audio / video signal; A viewpoint control step of changing a viewpoint of an image according to a user's control; An image reproducing step of reproducing an image corresponding to the changed time point; And A sound reproducing step of reproducing a sound spatially synchronized with the image of the changed time point based on the reference time point Remote virtual reality sound / video receiving method comprising a. The method of claim 10, The sound reproduction step, And calculating an angle between the changed viewpoint and the changed viewpoint, and reproducing a sound synchronized with the image of the changed viewpoint corresponding to the calculated angle.

Images