KR100874226B1 - Multi-view Image and 3D Audio Transceiver and Transmit and Receive Method Using the Same - Google Patents

Abstract

The present invention relates to a multi-view image and a three-dimensional audio transmission and reception apparatus and a transmission and reception method using the same. More specifically, by combining a multi-view image and three-dimensional audio data, by playing different audio according to the viewer's view image, the viewer can supply more realistic media, and the receiver can selectively transmit only the image of the desired viewpoint. The present invention relates to a multi-view image and a three-dimensional audio transmission apparatus and a transmission / reception method using the same, which can be used as a bandwidth and can generate an image of an intermediate view to provide an image more suitable for monitor parallax.

Multiview Images, 3D Audio, Viewpoint Conversion, Multiview Adaptation, Midview Images

Description

Device for tranceiving multi-view video and 3D audio, Method for tranceiving the same}

1 is a block diagram of a multi-view image and 3D audio transmission and reception apparatus according to a preferred embodiment of the present invention;

FIG. 2 is a block diagram showing a detailed configuration of a multi-view image encoder in FIG. 1;

3 is a block diagram showing a detailed configuration of a multi-view broadcast server unit in FIG.

4 is a block diagram showing a detailed configuration of a multi-view adaptive server unit in FIG.

5 is a block diagram illustrating a detailed configuration of a multi-view data transmission unit of FIG. 1;

6 is a block diagram illustrating a detailed configuration of a viewpoint converting unit in FIG. 1;

7 is a block diagram illustrating a detailed configuration of a multi-view data decoding unit of FIG. 1;

FIG. 8 is a block diagram illustrating a detailed configuration of an intermediate view image generation unit in FIG. 1;

FIG. 9 is a block diagram illustrating a detailed configuration of a 3D audio synthesizer in FIG. 1.

<Explanation of symbols for main parts of the drawings>

100-Multiview Image Acquisition Unit 200-Multiview Image Coding Unit

300-3D audio encoder 400-3D audio encoder

500-Multiview Broadcast Server Unit 600-Multiview Adaptive Server Unit

700-Multiview Data Transmitter 800-Viewpoint Converter

900-Multiview Data Decoding Unit 1000-Multiview Image Storage

1100-Mid-view image generator 1200-Multi-view image playback unit

1300-3D Audio Storage 1400-3D Audio Synthesis

1500-3D Audio Playback

The present invention relates to a multi-view image and a three-dimensional (hereinafter referred to as "3D") audio transceiver and a transmission and reception method using the same. More specifically, by combining multi-view images and 3D audio data, and playing different audio according to the viewpoint image viewed by the viewer, more realistic media can be supplied, and only the image of the desired viewpoint can be selectively transmitted from the receiving side, thereby narrowing bandwidth. Also, the present invention relates to a multi-view image and a 3D audio transmitting apparatus and a transmitting / receiving method using the same, which can generate an image of an intermediate view and provide a more suitable image for monitor parallax.

Multi-view video is a collection of view images obtained by photographing the same object with a large number of synchronized cameras (e.g. eight) arranged in parallel or arc shapes. Such multi-view video can be applied not only to stereoscopic display devices but also to stereoscopic viewing, realistic broadcasting, 3D DMB broadcasting, free-view TV (FTV), etc., when the user wants to view a content or view content in 3D stereoscopic images. Is a broad technology.

Element technologies related to such multi-view image processing include acquisition techniques, modeling / rendering techniques, encoding / decoding techniques, and transmission techniques.

Modeling / rendering technology is a technique of modeling a specific object through a plurality of cameras, the user can look at the object at any point in time. For example, "human body modeling and random view image generation technology using HD camera" by NHK R & D Center in Japan, and "free-viewpoint video technology" by MPI-infomatik of Germany have been proposed. Modeling / rendering technology is developed separately from the concept of transmission and reception, and is used in the field of pure image processing.

Meanwhile, a multi-view video-based encoding and transmission technology includes "Eye Vision." Eye Vision, for example, installs about 50 cameras in a sports arena and creates a 360-degree view to a user for a specific object at any time. In this technology, since the operator selects a random point of view and transmits it through a broadcast, there is a problem that the viewer has no choice but to watch as the operator transmits.

In addition, conventional multi-view image processing systems do not consider the combination of multi-view image data and audio data or multiplex it with a single audio data even if this is considered, so that even if a viewer watches an image at any point in time, There is a problem that drop the sense of reality. Moreover, when the user can select a viewpoint, there is a problem that the audio data does not adapt in real time to a desired specific viewpoint image.

In addition, the conventional multi-view image processing system transmits all of the various viewpoint images acquired from several cameras, and the viewer selects a desired viewpoint image on the receiving side, thus inevitably taking up tremendous bandwidth in data transmission. There is a problem.

In addition, even if the existing multi-viewpoint image processing system transmits all of the obtained viewpoint images, the receiving side can view only as many viewpoint images as the number of cameras. Therefore, there is a problem in that there is not enough viewpoint image to match the viewing interval of the receiving 3D monitor, so that the difference between the viewpoint images is large and a smooth change between viewpoints cannot be provided.

The present invention has been made to solve the above problems, in particular, by combining a multi-view image and 3D audio data, it is possible to supply a more realistic media by playing different audio according to the viewing image viewed by the viewer, at the receiving side Provides a multi-view image and 3D audio transmitting and receiving device and a transmission and reception method using the same that can selectively transmit only the image of the desired point of view, even with a narrow bandwidth, and can provide an image that meets the monitor parallax by generating an image of the intermediate view Its purpose is to.

In order to achieve the above object, a multi-view image and a 3D audio transmitting and receiving device according to the present invention are apparatuses for transmitting and receiving multi-view image data and 3D audio data, and the multi-view image for multiplexing a part of encoded multi-view image data. A multi-view broadcasting server unit including a multiplexing unit and a video audio multiplexing unit for multiplexing the encoded remaining multiview image data and the encoded 3D audio data; A multi-view adaptive server unit for generating a viewpoint image requested by the receiving side and 3D audio corresponding thereto; A viewpoint conversion unit for generating a descriptor of a viewpoint image and a descriptor of 3D audio according to a viewpoint requested by the viewer; And an intermediate viewpoint image generating unit generating an intermediate viewpoint image through the decoded multi-view image data.

The multi-view adaptive server unit may further include: a view descriptor parser configured to parse a view descriptor transmitted from the view transform unit; An image resource adaptor adapted to adapt a multi-view resource from the viewpoint descriptor parser; An image descriptor adaptor adapted to adapt a multi-view image descriptor from the image resource adaptor; An audio resource adaptor adapted to adapt an audio resource from the viewpoint descriptor parser; And an audio descriptor adaptor adapted to adapt an audio descriptor from the audio resource adaptor.

The view converting unit may further include: a view determining unit selecting at least one view desired by a viewer from a plurality of received view images; And a viewpoint descriptor generator for generating a descriptor of a viewpoint selected from the viewpoint determiner and transmitting the descriptor to the multi-view adaptive server unit.

The intermediate view image generation unit may include a camera parameter calculator configured to calculate external and internal parameters of the camera; A depth information calculator for finding initial depth information by finding matching points between neighboring viewpoint images; A region dividing unit for determining a reference image between neighboring viewpoint images and dividing the reference image into regions having different color information based on the color information of the reference image; A depth layer predictor for generating a depth layer by reflecting initial depth information calculated by the depth information calculator in a region divided by colors through the region divider; A depth information improver which repeatedly allocates depth to the depth layer generated from the depth layer predictor; And a mid-view production unit configured to generate an image at any point in time using depth information optimally generated by the depth information improver and camera parameters calculated by the camera parameter calculator.

In addition, the multi-view image and 3D audio transmission and reception apparatus includes a multi-view image acquisition unit for obtaining a multi-view image from a plurality of cameras; A multi-view image encoder for encoding multi-view image data obtained through the multi-view image acquisition unit; A 3D audio acquisition unit for acquiring multi-channel 3D audio from the plurality of microphones; The apparatus may further include a 3D audio encoder configured to encode 3D audio data obtained through the 3D audio acquirer.

In addition, the multi-view image and 3D audio transmission and reception apparatus includes an acquisition server unit for importing data of each viewpoint image from the camera; A control server unit for giving a control command to selectively transmit a requested time point at the receiving side; And an IP (Internet Protocol) multicast processing unit for obtaining data of the viewpoint image commanded from the control server unit from the acquisition server unit and transmitting the data of the viewpoint image in the form of uncompressed data. It may further include.

The multi-view data transmission unit may further include: a viewpoint data collection unit receiving all viewpoint data including each viewpoint image and generation time information of 3D audio from the acquisition server unit; And a multi-view multiplexer configured to perform synchronous multiplexing of the view data using the generation time information.

The multi-view image and 3D audio transmission and reception apparatus may further include: a j-view decoder configured to decode j views from among n multi-view coded data; A viewpoint data collection unit for collecting the data decoded through the j viewpoint decoder; And a multi-view data decoder having a 3D audio decoder for decoding the received multi-channel audio data (where n ≧ j).

In addition, the multi-view image and the 3D audio transmission and reception apparatus includes a multi-view image storage unit for storing the data collected through the viewpoint data collection unit for generating a viewpoint image of each frame unit for each viewpoint; A 3D audio storage unit for synchronizing with a multi-view image and storing decoded 3D audio data for 3D audio synthesis; A 3D audio synthesizing unit for synthesizing 3D audio corresponding to the viewpoint image selected by the viewer from the 3D audio storage unit; And a 3D audio reproducing unit for reproducing 3D audio data synthesized through the 3D audio synthesizing unit through a multichannel speaker or a stereo speaker.

The 3D audio synthesizing unit may further include: a surround panning unit configured to synthesize 3D audio corresponding to the selected viewpoint image through surround panning according to a viewer's viewpoint selection; 3D parameter extraction unit for extracting 3D parameters from the multi-channel audio data to give a 3D effect in stereo playback; And a down mixing unit generating stereo audio data using the 3D parameter extracted through the 3D parameter extracting unit.

In addition, a multi-view image and 3D audio transmission and reception method according to the present invention comprises: (a) a multi-view image multiplexing step of multiplexing a part of encoded multi-view image data; (b) a video audio multiplexing step of multiplexing the remaining coded multiview image data and the encoded 3D audio data; (c) generating a descriptor of a viewpoint image and a descriptor of 3D audio according to the viewpoint requested by the viewer; (d) generating a viewpoint image required by the receiver and 3D audio corresponding thereto; And (e) generating an image of an intermediate view through the decoded multi-view image data.

The step (d) may include parsing a view descriptor transmitted through the step (c); Adapting a multi-view resource; Adapting a multiview image descriptor; Adapting the audio resource; And adapting the audio descriptor.

Also, the step (c) may include selecting at least one viewpoint desired by the viewer from the plurality of viewpoint images received; And generating and transmitting the descriptor of the selected time point.

In addition, step (e) may include calculating external and internal parameters of the camera; Finding initial depth information by finding a matching point between neighboring viewpoint images; Determining a reference picture between neighboring viewpoint images, and dividing the reference picture into regions having different color information based on color information of the reference picture; Generating a depth layer by reflecting the initial depth information in a region divided by colors; Iteratively performing depth allocation to the depth hierarchy; And generating an image of an arbitrary view point using depth information optimally generated through the depth assignment and calculated camera parameters.

The multi-view image and the 3D audio transmission and reception method may further include: an uncompressed data transmission step of transmitting a viewpoint image requested by a receiver in the form of uncompressed data; And a compressed data transmission step of multiplexing all viewpoint images and 3D audio corresponding thereto into one data stream and transmitting the compressed data in the form of compressed data.

In addition, the multiplexing of the compressed data transmission step may be performed in a synchronized form by using generation time information of each viewpoint image and 3D audio.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible, even if shown on different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the following will describe a preferred embodiment of the present invention, but the technical idea of the present invention is not limited thereto and may be variously modified and modified by those skilled in the art.

First, a multi-view image and a 3D audio transmission and reception apparatus according to a preferred embodiment of the present invention will be described.

1 is a block diagram of a multi-view image and 3D audio transmission and reception apparatus according to a preferred embodiment of the present invention. FIG. 2 is a multi-view image encoder in FIG. 1, FIG. 3 is a multi-view broadcast server in FIG. 1, FIG. 4 is a multi-view adaptive server in FIG. 1, FIG. 5 is a multi-view data transmitter in FIG. FIG. 1 is a block diagram illustrating in detail the view converting unit of FIG. 1, FIG. 7 is a multiview data decoding unit of FIG. 1, FIG. 8 is an intermediate view image generating unit of FIG. 1, and FIG. 9 is a 3D audio synthesizing unit of FIG. 1. .

In the multi-view image and 3D audio transmission and reception apparatus according to the preferred embodiment of the present invention, referring to FIG. 1, the multi-view image acquisition unit 100, the multi-view image encoding unit 200, the 3D audio acquisition unit 300, 3D audio encoder 400, multi-view broadcast server 500, multi-view adaptive server 600, multi-view data transmission unit 700, a viewpoint converting unit 800, a multi-view data decoding unit 900 , The multi-view image storage unit 1000, the mid-view image generation unit 1100, the multi-view image playback unit 1200, the 3D audio storage unit 1300, the 3D audio synthesis unit 1400, and the 3D audio playback unit ( 1500).

The multi-view image acquisition unit 100 is a part of obtaining a multi-view image from a plurality of cameras. For example, if the multi-viewpoint image acquisition unit 100 includes n cameras, each camera captures the same photographing target and stores a digital or analog signal in a corresponding buffer through a transmission line, It is transmitted to the image encoder 200.

The multi-view image encoder 200 is a portion for encoding multi-view image data obtained through the multi-view image acquisition unit 100. 2, the multiview image encoder 200 includes a multiview image rearranger 210 and a rearrangement image encoder 220.

The multi-view image rearranger 210 divides a plurality of multi-view images V1, V2,..., Vn input into an encoder into GOP units, and rearranges them sequentially. Part. The frame rearrangement method of the multi-view image rearranger 210 will be described below. For example, when eight viewpoint images S0 to S7 are inputted through eight cameras, the screens of all viewpoints S0 to S7 are arranged in a line at time T0, and then a predetermined time (T1 to T8). While outputting the list of the first time screen (S0) in a row. Subsequently, the process of listing the screen S1 of the next time point for a predetermined time T1 to T8 is repeated, and when listing of the screen of the last time point S7 is completed in a line, the predetermined time T9 to T16 is again performed. ), The list starts from the screen S0 of the first time point and the screen from the last time point S7. Such a screen rearrangement method is currently a method for reviewing a multiview image in the H.264 / AVC standard. Of course, the screen may be rearranged in a different manner.

The rearranged image encoder 220 encodes the rearranged image according to a standard encoding scheme such as MPEG-2 and H.264 / AVC. Each viewpoint image may be simultaneously encoded (simulcast encoding), but is preferably multi-view encoding using inter-view correlation. In multi-view images, there are not only temporal redundancies between temporally adjacent images, but also spatial redundancy between images obtained by spatially adjacent cameras. The simultaneous encoding method has a disadvantage in that the spatial redundancy existing between each viewpoint image cannot be removed because the images of each viewpoint are independently encoded. In contrast, in the multi-view image encoding method, spatial redundancy can also be eliminated by rearranging the view images and encoding them with a single encoder, so that multi-view images can be encoded more efficiently.

The 3D audio acquisition unit 300 is a part for obtaining a multi-channel 3D audio sound source from a plurality of microphones. The 3D audio acquisition unit 300 may acquire a stereo sound source by using two microphones, but preferably obtains a multichannel sound source such as 4 channel or 5.1 channel using a plurality of microphones. Acquisition of such a sound source is preferably made for each viewpoint image. The viewer can watch 3D audio reflecting the viewpoint image together with the viewpoint image selected by the viewer through the viewpoint converting unit 800, so that the viewer can experience more realistic three-dimensional sound.

The 3D audio encoder 400 is a part for encoding 3D audio data obtained through the 3D audio acquirer. In this case, the 3D audio encoder 400 encodes a sound source using a multi-channel audio compression method such as AAC (Advanced Audio Coding) or AC-3 (Audio Coding-3).

The multi-view broadcasting server 500 manages a multi-view system and processes multi-view images and 3D audio data. Referring to FIG. 3, the multi-view broadcast server 500 includes a multi-view image multiplexer 510 which multiplexes a part of multi-view image data and 3D audio data, and the remaining coded multi-view image data and the encoded 3D audio. And an image audio multiplexer 520 for multiplexing data. For example, if the multi-view image coded data consists of V1, V2, ..., Vn, and the audio-coded data consists of A single, the multi-view image multiplexer 510 includes the V1, V2, ... , Multiplexing the image data of Vn-1 and the image audio multiplexer 520 may multiplex the viewpoint image of Vn and the audio of A. On the other hand, when audio coded data exists for each view image, a plurality of image audio multiplexers 520 may be provided or multiplexed through a single image audio multiplexer 520 by appropriately grouping a plurality of data.

The multi-view adaptive server unit 600 transmits a multi-view image to the receiving side (client), exchanges messages, and generates a viewpoint image and 3D audio data according to the viewpoint requested by the receiving side. That is, the multi-view adaptive server unit 600 uses a single content differently using information on a usage environment that consumes a multi-view image and 3D audio, that is, information on a receiving terminal characteristic and / or a viewer's reproduction preference characteristic. It provides a single-source multi-use environment that adaptively transforms to match the environment. Referring to FIG. 4, the multi-view adaptation server unit 600 includes: a view descriptor parser 610, an image resource adaptor 620, an image descriptor adaptor 630, an audio resource adaptor 640, and Audio descriptor adaptation unit 650.

The view descriptor parser 610 parses a view descriptor transmitted from the view converter 800. Here, a descriptor means information related to an item or component in a digital item (DI), which is a structured digital object having a standardized representation, identification, and metadata. That is, the viewpoint descriptor parser 610 parses the related information by receiving the viewpoint descriptor generated by the viewpoint descriptor generator 820 according to the viewpoint selected by the viewer in the viewpoint determiner 810 of the viewpoint transformer 800. do.

The image resource adaptor 620 adaptively transforms the multi-view resource parsed by the view descriptor parser 610. Here, the resource means an individually identifiable item such as a video, audio, image or text item. That is, the image resource adaptation unit 620 adaptively converts the image content by reflecting the playback taste of the viewer in the multi-view resource. Through this, an adaptive resource Rv is generated, and after a predetermined conversion process, is transmitted to the client.

The image descriptor adaptation unit 630 is a part for adaptively transforming a multiview image descriptor. In general, the image descriptor describes characteristics such as the image format of the receiving terminal, the maximum vertices processed per second, the maximum pixels per second, and the maximum rate, or prefers a specific viewpoint image. The reproduction taste characteristics of the viewer are described. The image descriptor adaptor 630 adaptively converts the image content by reflecting the multi-view image descriptor received from the viewpoint descriptor generator 820. Through this, an adaptive descriptor Dv is generated, and after a predetermined conversion process, is transmitted to the client.

The audio resource adaptor 640 adaptively converts the audio resource parsed through the view descriptor parser 610. Through this, an adaptive resource Ra is generated, and after being converted into a predetermined process, is transmitted to the client.

The audio descriptor adaptor 650 adaptively converts the audio content by reflecting the audio descriptor received from the viewpoint descriptor generator 820. Through this, the adaptive descriptor Da is generated and transmitted to the client through a predetermined conversion process.

The multi-view data transmitter 700 provides a transmission protocol and a mechanism for efficiently transmitting data to the client. In this case, the multi-view data transmitter 700 has a structure for transmitting two types of data for uncompressed data transmission and compressed data transmission. The uncompressed data transmission is transmitted in the form of uncompressed data in order to prevent the client from requesting a delayed image or deteriorated picture quality. On the other hand, compressed data transmission compresses all viewpoint images including a viewpoint image requested by a client, and multiplexes and transmits 3D audio data.

First, the multi-view data transmitting unit 700, for obtaining uncompressed data, includes an acquisition server unit 710, an IP multicast processing unit 720, and a control server unit 740. Referring to FIG. The acquiring server unit 710 is provided in plural, preferably provided by the number of multi-view cameras to obtain viewpoint data from each multi-view camera. The control server unit 740 issues a control command to selectively transmit a time point requested by the client. The control server unit 740 obtains data from the acquiring server unit 710 corresponding to the time point desired by the client, and forms each of the uncompressed data. The IP multicast processor 720 transmits the data. That is, only the time point desired by the client is selectively transmitted through the control data of the control server 740.

Second, the multiview data transmitter 700 includes an acquisition server unit 710, a viewpoint data collection unit 730, a synchronous multiplexer 750, and an IP multicast processor 760 for compressed data transmission. . The viewpoint data collection unit 730 receives data of all viewpoints, including generation time information of each viewpoint image and 3D audio, from the acquisition server unit 710. The collected data are multiplexed into one data stream through the synchronous multiplexer 750. In this case, the synchronous multiplexer 750 performs synchronized multiplexing by using generation time information of each viewpoint image and 3D audio. This multiplexed stream is transmitted to the client through the IP multicast processor 760.

The viewpoint converting unit 800 generates a descriptor of a viewpoint image and a descriptor of 3D audio according to a viewpoint requested by the viewer. Referring to FIG. 6, the viewpoint converter 800 includes a viewpoint determiner 810 and a viewpoint descriptor generator 820.

The viewpoint determination unit 810 is a part for selecting a viewpoint desired by the viewer from the plurality of viewpoint images received. In this case, one or more viewpoints may be desired by the viewer. That is, the viewpoint determination unit 810 selects the viewpoint Vk (k = 1 to n) desired by the user from the viewpoint images V1, V2, ..., Vn of the multiview camera.

The view descriptor generator 820 generates a view descriptor according to the view determiner through the view determiner 810. The viewpoint descriptor may be expressed in a machine readable language in an XML (eXtensible Markup Language) format, and there are a multiview image descriptor and an audio descriptor. The view descriptor generated through the view descriptor generator 820 is transmitted to the multiview adaptation server 600 to adaptively convert the view image and the 3D audio.

The multi-view data decoding unit 900 decodes the received multi-view image and the 3D audio data, and transmits the multi-view data storage unit 1000 and the 3D audio storage unit 1400. Referring to FIG. 7, the multi-view data decoder 900 includes a j-view decoder 910, a view data collection unit 920, and a 3D audio decoder 930.

The j-view decoder 910 is a part for decoding j-views from among the n multi-view coded data. At this time, j value is preferably less than or equal to n value. That is, the j-view decoder 910 decodes j views which are a part of n views or decodes all the views. When decoding for all views (j = n), although not shown, n multi-view coded data may be decoded using only one decoder. This is because the client may use a decoder having a lower complexity than the encoder according to the quality desired by the client. In the case of decoding using only one decoder as described above, it is possible to efficiently control while securing a certain quality or more, and can flexibly cope with the increase or decrease of the number of viewpoints. As shown in FIG. 7, the number of decoders may be increased as necessary to ensure more improved performance. The j value is determined by several variables, including the performance of the decoder and the usage environment. The j-time decoder 910 is formed by partially accepting a decoding rule of a standard system such as MPEG-2 and H.264 / AVC.

The viewpoint data collection unit 920 collects the data decoded by the j viewpoint decoder 910 into one. The viewpoint data collection unit 920 transmits the collected data to the multi-viewpoint image storage unit 1000. The multi-viewpoint image storage unit 1000 temporarily stores the received data to generate a viewpoint image in a frame unit for each viewpoint.

The 3D audio decoder 930 is a part for decoding the received audio data. In this case, the audio data may be a multi-channel or a stereo channel and is transmitted to the 3D audio storage 1300. The 3D audio storage unit 1300 temporarily stores decoded 3D audio data in order to synthesize 3D audio in synchronization with a multi-view image.

The intermediate view image generator 1100 generates an intermediate view image of the multiview image and transmits the intermediate view image to the multiview image reproducing unit 1200. That is, the mid-view image generator 1100 generates a mid-view image between the decoded viewpoint images, and provides a smooth viewpoint image according to the viewpoint interval of the 3D monitor. Referring to FIG. 8, the intermediate view image generator 1100 may include a camera parameter calculator 1110, a depth information calculator 1120, an area divider 1130, a depth layer predictor 1140, and depth information. An improvement unit 1150 and an intermediate view image production unit 1160 are provided.

The camera parameter calculator 1110 calculates external and internal parameters of the camera. The external parameters of the camera are parameters related to the camera settings, that is, the position, height, angle, etc. of the camera, and the internal parameters are parameters related to the characteristics of the camera itself, that is, the optical center, focal length, and resolution.

The depth information calculator 1120 finds a matching point between neighboring viewpoint images to find initial depth information. That is, the depth information calculator 1120 measures similarity between each given viewpoint images in block units, and finds the most similar matching point and calculates initial depth information.

The region dividing unit 1130 determines a reference image between neighboring viewpoint images, and divides the region into regions having different color information based on the color information of the reference image.

The depth layer predictor 1140 is a portion that generates the depth layer by reflecting initial depth information calculated by the depth information calculator 1120 in a region divided by colors through the region divider 1130. In this case, the depth layer predictor 1140 allocates representative depth information to the same region. This operation is repeated, grouping similar representative depth information for all regions into groups, and extracting one representative depth value from the group to create a depth hierarchy.

The depth information improving unit 1150 is a part which repeatedly allocates depth to the depth layer generated from the depth layer predicting unit 1140. In more detail, the depth information improving unit 1150 allocates the depth layer generated by the depth layer predicting unit 140 using representative depth information for each region, and the color region and color adjacent to the representative depth information. Define cost functions that reflect similarities. At this time, the depth allocation is preferably performed repeatedly to find the point where the cost function value is the minimum.

The intermediate view maker 1160 is a part that generates an image at any point in time using depth information optimally generated by the depth information improver 1150 and camera parameters calculated by the camera parameter calculator 1110. . Blank areas due to occlusion or the like may occur in any viewpoint image generated in this way. In this case, the empty area is restored based on the neighboring pixels, and a natural mid-view image is generated by using interpolation.

Referring to FIG. 9, the 3D audio synthesizer 1400 includes a surround panning unit 1410, a 3D parameter extracting unit 1420, and a down mixing unit 1430. The surround panning unit 1410 synthesizes 3D audio corresponding to the selected viewpoint image through surround panning according to the viewpoint selected by the user. The synthesized 3D audio data is reproduced through the multichannel speakers. The 3D parameter extractor 1420 extracts a 3D parameter from multi-channel audio data to give a 3D effect in stereo reproduction. The down mixing unit 1430 generates stereo audio data using the extracted 3D parameter and the selected viewpoint. The generated stereo audio data is reproduced through the stereo speakers.

The 3D audio player 1500 reproduces the synthesized 3D audio data through a multichannel speaker or a stereo speaker. In this case, as described above, the data transmitted through the surround panning unit 1410 is reproduced by the multi-channel speaker, and the data transmitted by the down mixing unit 1430 is reproduced by the stereo speaker.

Next, a multi-view image and a 3D audio transmission / reception method according to a preferred embodiment of the present invention will be described.

A multi-view image and a 3D audio transmission / reception method according to a preferred embodiment of the present invention may include a multi-view image acquisition step, a multi-view image encoding step, a 3D audio acquisition step, a 3D audio encoding step, and (a) a part of encoded multi-view image data. A multi-view image multiplexing step of multiplexing, (b) a video-audio-multiplexing step of multiplexing the remaining encoded multi-view image data and the encoded 3D audio data, and (c) the description of the viewpoint image and the 3D audio according to the viewpoint requested by the viewer. Generating a descriptor; transmitting uncompressed data in the form of uncompressed data requested by the receiving side; multiplexing all the view images and the corresponding 3D audio into a single data stream and transmitting them in compressed data form (D) generating a view image requested by the receiving side and 3D audio corresponding thereto; A multi-view data decoding step, a multi-view image storing step, (e) generating an image of an intermediate view through the decoded multi-view image data, a multi-view image reproducing step, a 3D audio storing step, a 3D audio synthesis step, and 3D audio reproduction step.

(C) generating the descriptor of the viewpoint image and the descriptor of 3D audio according to the viewpoint requested by the viewer, selecting at least one viewpoint desired by the viewer from the plurality of viewpoint images received and generating the descriptor of the selected viewpoint. It comprises the step of transmitting.
The step (d) of generating a viewpoint image required by the receiver and 3D audio corresponding thereto may include parsing the transmitted viewpoint descriptor, adapting a multiview resource, adapting a multiview image descriptor, and audio resources. And adapting the audio descriptor.

delete

In addition, the step (e) generating the image of the intermediate view through the decoded multi-view image data, the step of calculating the external and internal parameters of the camera, the step of finding the initial depth information by finding a matching point between neighboring viewpoint images, Determining a reference image between neighboring viewpoint images, dividing the reference image into regions having different color information based on the color information of the reference image, generating a depth hierarchy by reflecting initial depth information in the region divided by colors, Repeatedly performing depth assignment to the depth hierarchy and performing depth assignment to generate an image at any point of time using the optimally generated depth information and the calculated camera parameters. Since each of the above steps has been described above in the structure of the multi-view image and 3D audio transceiver, the description thereof will be omitted.

The general order of performing the multi-view image and 3D audio transmission / reception method according to the preferred embodiment of the present invention is as follows.

First, the multi-view image input to the multi-view image acquisition unit 100 is encoded through the multi-view image encoder 200 and then transmitted to the multi-view broadcast server unit 500. Similarly, the sound source input to the 3D audio acquirer 300 is encoded through the 3D audio encoder 400 and then transmitted to the multi-view broadcast server 500.

The multi-view broadcasting server 500 multiplexes a part of the encoded multi-view image data (step a), and multiplexes the remaining multi-view image data and 3D audio data (step b). The viewpoint converting unit 800 generates the viewpoint image descriptor and the 3D audio descriptor (step c) according to the viewpoint requested by the viewer, and then transmits the generated viewpoint image descriptor and the 3D audio descriptor to the multi-view adaptive server unit 600. The multi-view adaptation server 600 generates a view image and 3D audio corresponding to the view point requested by the receiving view converting unit 800 (step d), and transmits it to the multi-view data transmitting unit 700. . The multi-view data transmitter 700 transmits uncompressed data and compressed data, and decodes the multi-view data decoder 900 to decode the multi-view image storage unit 1000 and the 3D audio storage unit 1300. ).

The intermediate view image generating unit 1100 generates the image of the intermediate view by taking the decoded multiview image data from the multiview image storage unit 1000 (step e). These viewpoint images are reproduced through the multi-view image reproducing unit 1200. Similarly, the 3D audio synthesizer 1400 takes 3D audio data decoded from the 3D audio storage 1300, synthesizes 3D audio of a multichannel or a stereo channel, and reproduces the 3D audio through the 3D audio playback unit 1500.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes, and substitutions may be made by those skilled in the art without departing from the essential characteristics of the present invention. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the embodiments and the accompanying drawings. . The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

According to the present invention, the multi-view image and the 3D audio data are combined, but by playing different audio according to the view image viewed by the viewer, more realistic media can be supplied, and only the image of the desired view can be selectively transmitted from the receiving side. In addition, it is possible to generate a mid-point image and provide a smooth and natural view image according to the viewing interval of the monitor.

Claims (16)

In the apparatus for transmitting and receiving multi-view image data and 3D audio data, A multiview broadcast server unit including a multiview image multiplexer for multiplexing a part of encoded multiview image data, and an image audio multiplexer for multiplexing the remaining coded multiview image data and the encoded 3D audio data; A multi-view adaptive server unit for generating a viewpoint image requested by the receiving side and 3D audio corresponding thereto; An acquisition server unit for obtaining data of each viewpoint image from a camera, a control server unit for giving a control command to selectively transmit the viewpoint requested at the receiving side, and data of the viewpoint image commanded from the control server unit; A multi-view data transmission unit which has an IP (Internet Protocol) multicast processing unit which takes the data from the unit and transmits it in the form of uncompressed data and transmits it in the form of uncompressed data; A viewpoint conversion unit for generating a descriptor of a viewpoint image and a descriptor of 3D audio according to a viewpoint requested by the viewer; And An intermediate viewpoint image generation unit that generates an intermediate viewpoint image through the decoded multiview image data. Multi-view image and 3D audio transmission and reception apparatus comprising a. The method of claim 1, wherein the multi-view adaptive server unit A view descriptor parser for parsing a view descriptor transmitted from the view transform unit; An image resource adaptor for adaptively converting a multi-view resource parsed through the viewpoint descriptor parser to a receiving terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptive resource Rv; An image descriptor adaptor for adaptively converting the multi-viewpoint image descriptor transmitted from the viewpoint converter to a receiving terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptive descriptor Dv; An audio resource adaptor configured to adaptively convert the audio resource parsed through the viewpoint descriptor parser into a receiving terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptive resource Ra; And An audio descriptor adaptor for adaptively converting an audio descriptor transmitted from the viewpoint converter to a reception terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptive descriptor Da. Multi-view image and 3D audio transmission and reception apparatus comprising a. The method of claim 1, wherein the viewpoint converting unit A viewpoint determination unit for selecting at least one viewpoint desired by the viewer from the plurality of viewpoint images received; And A viewpoint descriptor generator which generates a descriptor of a selected viewpoint from the viewpoint determiner and transmits the descriptor to the multi-view adaptive server unit Multi-view image and 3D audio transmission and reception apparatus comprising a. The method of claim 1, wherein the intermediate view image generation unit A camera parameter calculator configured to calculate external and internal parameters of the camera; A depth information calculator for finding initial depth information by finding matching points between neighboring viewpoint images; A region dividing unit for determining a reference image between neighboring viewpoint images and dividing the regions of neighboring viewpoint images into regions having different color information based on color information of the reference image; While generating the depth layer by reflecting the initial depth information calculated by the depth information calculation unit in the area divided by color through the area dividing unit, it is repeated to assign the representative depth information for the same area to all areas A depth layer prediction unit for generating a depth hierarchy by grouping representative depth information and extracting one representative depth value from the group; A depth information improver which repeatedly allocates depth to the depth layer generated from the depth layer predictor; And An intermediate view maker for generating an image of an arbitrary view point using depth information optimally generated by the depth information improver and camera parameters calculated by the camera parameter calculator. Multi-view image and 3D audio transmission and reception apparatus comprising a. The method of claim 1, A multi-view image acquisition unit for obtaining a multi-view image from a plurality of cameras; A multi-view image encoder for encoding multi-view image data obtained through the multi-view image acquisition unit; A 3D audio acquisition unit for acquiring multi-channel 3D audio from the plurality of microphones; 3D audio encoder for encoding the 3D audio data obtained through the 3D audio acquisition unit Multi-view image and 3D audio transmitting and receiving device further comprising. delete The multi-view data transmission unit of claim 1, wherein A viewpoint data collection unit which receives all viewpoint data including each viewpoint image and generation time information of 3D audio from the acquisition server unit; And A synchronous multiplexer for synchronously multiplexing the viewpoint data using the generation time information Multi-view image and 3D audio transmission and reception apparatus characterized in that it further comprises a compressed data transmission. The method of claim 1, A j-view decoder for decoding j-views from the received n multi-view coded data; A viewpoint data collection unit for collecting the data decoded through the j viewpoint decoder; And 3D audio decoder for decoding the received multi-channel audio data A multi-view image and 3D audio transmitting and receiving apparatus further comprising a multi-view data decoding unit having n, wherein n≥j. The method of claim 8, A multi-view image storage unit for storing the data collected through the viewpoint data collection unit to generate a viewpoint image in a frame unit for each viewpoint; A 3D audio storage unit for synchronizing with a multi-view image and storing decoded 3D audio data for 3D audio synthesis; A 3D audio synthesizing unit for synthesizing 3D audio corresponding to the viewpoint image selected by the viewer from the 3D audio storage unit; And 3D audio playback unit for playing back 3D audio data synthesized through the 3D audio synthesis unit through a multi-channel speaker or a stereo speaker Multi-view image and 3D audio transmitting and receiving device further comprising. The method of claim 9, wherein the 3D audio synthesis unit A surround panning unit for synthesizing 3D audio corresponding to the selected viewpoint image through surround panning according to a viewer's viewpoint selection; 3D parameter extraction unit for extracting 3D parameters from the multi-channel audio data to give a 3D effect in stereo playback; And A down mixing unit for generating stereo audio data using the 3D parameters extracted through the 3D parameter extractor Multi-view image and 3D audio transmission and reception apparatus comprising a. In the method for transmitting and receiving multi-view image data and 3D audio data, (a) a multiview image multiplexing step of multiplexing a part of encoded multiview image data; (b) a video audio multiplexing step of multiplexing the remaining coded multiview image data and the encoded 3D audio data; (c) generating a descriptor of a viewpoint image and a descriptor of 3D audio according to the viewpoint requested by the viewer; (d) The uncompressed data transmission step of transmitting the requested view image in the form of uncompressed data, and multiplexing all view images and the corresponding 3D audio into one data stream and transmitting the resultant view image in the form of compressed data. A multi-view data transmission step including a compressed data transmission step; (e) generating a viewpoint image requested by the receiver and 3D audio corresponding thereto; And (f) generating an intermediate viewpoint image through the decoded multi-view image data Multi-view image and 3D audio transmission and reception method comprising a. The method of claim 11, wherein step (e) Parsing the view descriptor transmitted through the step (c); An image resource adapting step of adaptively converting a multi-view resource parsed through the viewpoint descriptor parsing into a receiving terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptive resource Rv; An image descriptor adaptation step of adaptively converting the multi-viewpoint image descriptor transmitted through the step (c) to a reception terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptation descriptor Dv; An audio resource adaptation step of adaptively converting the audio resource parsed through the viewpoint descriptor parsing to a receiving terminal characteristic or a viewer's reproduction preference characteristic to generate an adaptive resource Ra; And The audio descriptor adaptation step of adaptively converting the audio descriptor transmitted through the step (c) to the receiving terminal characteristic or the viewer's reproduction preference characteristic to generate the adaptive descriptor Da. Multi-view image and 3D audio transmission and reception method comprising a. The method of claim 11, wherein step (c) Selecting at least one viewpoint desired by a viewer from a plurality of received viewpoint images; And Creating and transmitting the descriptor of the selected time point Multi-view image and 3D audio transmission and reception method comprising a. The method of claim 11, wherein step (f) Calculating external and internal parameters of the camera; Finding initial depth information by finding a matching point between neighboring viewpoint images; Determining a reference picture between neighboring viewpoint images, and dividing the region of neighboring viewpoint images into an area having different color information based on color information of the reference image; The depth layer is generated by reflecting the initial depth information in the region divided by color, but the task of allocating the representative depth information for the same region is repeated in all the regions, the representative depth information is grouped, and the representative depth value in the group. Extracting one to generate a depth hierarchy; Iteratively performing depth allocation to the depth hierarchy; And Generating an image at any point in time using the depth information optimally generated through the depth allocation and the calculated camera parameters. Multi-view image and 3D audio transmission and reception method comprising a. delete 12. The method of claim 11, wherein the multiplexing into one data stream in the compressed data transmission step is performed. A multi-view image and 3D audio transmission and reception method, characterized in that performed in a synchronized form using the time information of each viewpoint image and 3D audio.

Images