KR101646867B1 - Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof - Google Patents

Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof Download PDF

Info

Publication number
KR101646867B1
KR101646867B1 KR1020150024880A KR20150024880A KR101646867B1 KR 101646867 B1 KR101646867 B1 KR 101646867B1 KR 1020150024880 A KR1020150024880 A KR 1020150024880A KR 20150024880 A KR20150024880 A KR 20150024880A KR 101646867 B1 KR101646867 B1 KR 101646867B1
Authority
KR
South Korea
Prior art keywords
audio data
position information
viewpoint
information
image
Prior art date
Application number
KR1020150024880A
Other languages
Korean (ko)
Inventor
김성권
Original Assignee
서울과학기술대학교 산학협력단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 서울과학기술대학교 산학협력단 filed Critical 서울과학기술대학교 산학협력단
Priority to KR1020150024880A priority Critical patent/KR101646867B1/en
Application granted granted Critical
Publication of KR101646867B1 publication Critical patent/KR101646867B1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/301—Automatic calibration of stereophonic sound system, e.g. with test microphone
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00—Details of transducers, loudspeakers or microphones
    • H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
    • H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
    • H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00—Stereophonic arrangements
    • H04R5/027—Spatial or constructional arrangements of microphones, e.g. in dummy heads
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Stereophonic System (AREA)

Abstract

An apparatus for implementing an FTV stereophonic sound using microphone location information according to the disclosed present invention comprises: a plurality of microphones which are installed at predetermined locations; an audio reception unit which receives audio information from the microphones, and generates pieces of location information about respective pieces of audio data; an image location information detection unit which detects location information of an image viewpoint via a stereoscopic image signal processing unit when the image viewpoint is designated by a user; a comparison unit which detects location information of corresponding audio data identical to the location information of the image viewpoint by comparing the pieces of location information of the pieces of audio data with the location information of the image viewpoint, and transmits the corresponding audio data; a stereophonic sound signal processing unit which signal-processes the corresponding audio data, transmitted by the comparison unit, into 3D sound information; and a stereophonic sound output unit which controls a speaker to output a stereophonic sound signal-processed by the stereophonic sound signal processing unit. According to the present invention, a stereophonic sound associated with a stereoscopic image can be implemented by using location information of microphones when an FTV service is provided, thereby providing an effect in which a further realistic FTV service can be provided.

Description

FIELD OF THE INVENTION [0001] The present invention relates to an apparatus for realizing an FTV stereophonic sound using microphone position information,

The present invention relates to an apparatus and method for implementing a stereophonic sound, and more particularly, to an apparatus and method for implementing stereophonic sound interacting with a stereoscopic image when providing a free-view TV (FTV) service using position information of a microphone .

With the development of digital media related technology, the audience level of users has become higher, so that realistic and realistic media viewing technology is required, and stereoscopic video and stereophonic technology are dramatically developed. It is also changing to 3D (3-Demention Audio / Video) service which is active and participates in passive and one-sided AV (Audio / Video) service.

MPEG has been experimenting with 3DAV technology since 2001. In MPEG, many applications and technologies related to 3D video have been proposed. Interactivity was the main common key. The viewers can select a desired point of view, and the restored image can be viewed according to the viewpoint of the data captured from the actual image.

3DAV has three application areas: Omni-directional video, Free viewpoint video, and Bidirectional stereo video.

Unlike forward video, free-view video is a technology that enables viewers to freely select an arbitrary viewpoint and a line of sight. A free-view video is a free-view television (FTV) And FTV is also regarded as 3DTV.

FIG. 1 is a schematic diagram of an FTV system according to functions. After a correction process (correction) is performed for a sequence process for compensating geometrical errors between images by acquiring images from a plurality of cameras, Decoding process, and then an intermediate image is generated to reproduce an image at an arbitrary point in time.

However, there are a lot of researches on realization of stereoscopic images when providing FTV service, but there is little research on realization of stereo sound interfacing with stereoscopic images. In order to realize the stereoscopic sound in the center of the screen, the sound source must be changed simultaneously with the change of the screen. For example, when a screen at a certain position is designated as a viewpoint and enlarged at a certain point in time, the sound source at the corresponding location must be reproduced through a speaker to provide a more effective FTV service. However, There is a difficulty in realizing interlocking stereophonic sound.

Particularly, when a plurality of microphones for sound source acquisition, such as sports broadcasts, are installed wirelessly due to installation location or space restrictions, this difficulty becomes even more serious.

Korean Patent Publication No. 10-2010-0006300 (Published Jan. 19, 2010) Korean Patent Publication No. 10-2010-0018802 (published Feb. 18, 2010)

'Free - view TV and 3D Video Standardization Trend / Hyo - sung, Icheon' (Information and Communication Standardization News). Pp. 116-94 (published in March / April 2008)

The present invention provides an FTV stereophonic sound device using microphone position information for realizing stereophonic sound interfaced with a stereoscopic image in providing an FTV service using location information of a microphone, It has its purpose.

According to an aspect of the present invention, there is provided an apparatus for realizing an FTV stereophonic sound using microphone position information, comprising: a plurality of microphones installed at a predetermined position; An audio receiver for receiving audio information from the microphone and generating position information for each audio data; An image position information detector for detecting position information of the image viewpoint from a stereoscopic image signal processor when a user designates a viewpoint; A comparing unit for comparing the position information of the viewpoint with the position information of the audio data to detect position information of the audio data coinciding with the position information of the viewpoint and transmitting the corresponding audio data; A stereo sound signal processing unit for signal processing the audio data transmitted from the comparison unit to three-dimensional acoustic information; And a stereo sound output unit for outputting the stereo sound signal processed by the stereo sound signal processing unit to the speaker.

According to a preferred embodiment of the present invention, the microphone is applied to a wireless microphone, and the audio receiver stores location information of each of the microphones in advance and generates location information on the audio data when the audio data is received from each microphone .

If there is no audio data coinciding with the position of the view point, the comparator detects two audio data at a position closest to the position of the view point and transmits the audio data to the signal processor .

According to another aspect of the present invention, there is provided an apparatus for implementing an FTV stereophonic sound using microphone position information, the apparatus comprising: a) receiving audio data from a plurality of microphones and generating position information on the received audio data; ; b) encoding and decoding audio data including the position information; c) if the user designates a viewpoint, detecting location information of the viewpoint; d) comparing the position information of the viewpoint with the position information of the decoded audio data, detecting position information of the audio data coinciding with the position information of the viewpoint, Transmitting; e) signal processing the transmitted audio data into three-dimensional acoustic information; And f) outputting the signal processed stereo sound to a speaker.

In the step d), if there is no audio data coinciding with the position of the image viewpoint, two audio data positions closest to the position of the image viewpoint are detected and transmitted.

According to the present invention, stereophonic sound interlocking with a stereoscopic image can be realized by providing position information of a microphone to provide an FTV service, thereby providing a more realistic FTV service.

FIG. 1 is a block diagram illustrating a general FTV system according to functions,
FIG. 2 is a block diagram of an FTV system including an FTV stereophonic sound utilizing apparatus using microphone position information according to an embodiment of the present invention.
FIG. 3 is a conceptual diagram for explaining a service of the FTV system of FIG. 2,
4 is a flowchart illustrating an FTV stereophonic sounding method using microphone position information according to an embodiment of the present invention.

These and other objects, features and other advantages of the present invention will become more apparent by describing in detail preferred embodiments of the present invention with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT Hereinafter, an apparatus and method for implementing an FTV stereophonic sound using microphone position information according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 2, the FTV system according to the embodiment of the present invention includes a stereoscopic image implementation device and a stereophonic sound implementation device.

Although the scope of the present invention is not limited to this, the FTV stereophonic apparatus using the microphone position information according to the embodiment of the present invention and the FTV system including the FTV stereophonic sound system may be used in various fields such as a sports relay, Or an area or a space is very wide so that it is preferable to apply to a case where a plurality of microphones for sound source acquisition are installed wirelessly due to installation position or space limitation.

The stereoscopic image implementing apparatus includes a video acquiring unit 110 having a plurality of cameras (a camera 1, a camera 2, and a camera M), a video receiver 120 for receiving an image from the video acquiring unit 110, A video decoding unit 140 that generates a decoded video data by decoding the video data transmitted from the video encoding unit 130 and includes a memory; And a stereoscopic image output unit 180 for outputting the processed three-dimensional image to a display unit.

The stereoscopic image signal processing unit 170 reproduces a stereoscopic image at this position when there is an image viewpoint designation from the user through the user interface 50. [

The configuration and operation of the stereoscopic image realizing device are well known in the technical field of the present invention, and are not a gist of the present invention, and thus a detailed description thereof will be omitted.

The stereophonic sounding apparatus includes an audio acquiring unit 210, an audio receiving unit 220, an audio encoding unit 230, an audio decoding unit 240, a video position An information detection unit 250, a comparison unit 260, a stereo sound signal processing unit 270, and a stereo sound output unit 280.

The audio receiver 220 receives audio information from the microphone and generates position information for each audio data.

As described above, although the scope of the present invention is not limited to this, according to a preferred embodiment of the present invention, the microphones of the audio acquisition means 210 are wireless microphones. Accordingly, the audio receiver 220 wirelessly receives audio information from the microphones.

When the microphones and the audio receiver 220 are connected by wire, it is easy to receive the position information of the microphones. However, when the microphones and the audio receiver 220 are connected wirelessly, Is not easy to receive. Therefore, according to the present invention, the position information of each microphone installed at a predetermined position is stored in advance in the audio receiver 220, and therefore, when audio data is input from each microphone in which position information is stored, .

The audio encoding unit 230 encodes the audio data, and the audio decoding unit 240 decodes the audio data transmitted from the audio encoding unit 230 to generate decoded audio. Although not shown, the audio decoding unit 240 is provided with a memory, so that the voice obtained from each microphone is stored together with the position information.

As described above, the stereoscopic image signal processor 170 enlarges and reproduces the stereoscopic image at the viewpoint position when there is a designation of a viewpoint from the user through the user interface 50. The stereoscopic image signal processor 170, The detection unit 250 detects the position information of the image when the user designates a viewpoint.

The comparing unit 260 compares the position information of the designated viewpoint from the user with the position information of the audio data stored in the memory of the audio decoding unit 240, After detecting the position information of the sound, the audio data is transmitted to the stereo sound processing unit 270.

On the other hand, in order to realize an ideal stereo sound, an infinite number of sound sources must be acquired by arranging an infinite number of microphones. However, it is impossible to arrange an infinite number of microphones in reality. Due to a spatial problem or a cost problem, It is inevitable to dispose of them. Accordingly, there may be no sound data coinciding with the position of the view point designated by the user. In this case, after detecting two audio data at a position closest to the position of the view point, And transmits it to the signal processing unit 270 so that interpolation processing can be performed.

The stereo sound signal processing unit 270 processes the decoded sound into three-dimensional sound information that is available to the three-dimensional speaker, and the stereo sound output unit 280 outputs the processed stereo sound to the speaker.

Hereinafter, an operation of the FTV stereophonic sounder using the microphone position information having the above-described configuration and a method of implementing the FTV stereophonic sound using the microphone position information according to the embodiment of the present invention will be described with reference to FIG. 3 and FIG.

First, the audio receiver 220 wirelessly receives audio data from a plurality of microphones and generates position information for each audio data. As described above, the audio receiver 220 stores position information of each microphone, The location information of the audio data can be known (S310).

Thereafter, the audio encoding unit 230 and the audio decoding unit 240 encode and decode audio data including the position information (S320).

If a user designates a viewpoint, the image position information detector 250 detects the position information of the image viewpoint at step S330.

For example, in FIG. 3, when the user wants to enlarge the 'Point A' portion of the screen 60 and enlarge this portion through the user interface 50, the 'Point A' portion becomes the viewpoint, The stereoscopic image signal processing unit 170 processes the image so that the viewpoint of the image is enlarged and outputted, and the image position information detector 250 detects the position information (J: 3) of the image viewpoint.

The comparing unit 260 compares the position information J (3) of the image viewpoint with the position information of the audio data stored in the memory of the audio decoding unit 240, J: 3), and transmits the audio data to the stereo sound signal processing unit 270 (S340).

As described above, there may be no audio data coinciding with the position of the view point. In this case, in order to process the interpolation method, two pieces of sound data at positions closest to the position of the view point And then transmitted.

Then, the stereophonic sound signal processing unit 270 processes the decoded sound into three-dimensional sound information that is available to the three-dimensional speaker (S350). Finally, the stereophonic sound output unit 280 outputs the processed stereophonic sound to the speaker (S360).

 As described above, according to the present invention, stereophonic sound interlocking with a stereoscopic image can be implemented by providing the FTV service using the location information of the microphone, thereby providing a more realistic FTV service.

Although the preferred embodiments of the present invention have been described, the present invention is not limited to the specific embodiments described above. It will be apparent to those skilled in the art that numerous modifications and variations can be made in the present invention without departing from the spirit or scope of the appended claims. And equivalents should also be considered to be within the scope of the present invention.

110. Video acquiring means 120. Video receiving unit
170. Stereoscopic image signal processing unit 180. Stereoscopic image output unit
210. Audio acquisition means 220. Audio reception unit
250. Image position information detection unit 260. Comparison unit
270. Stereophone signal processing unit 280. Stereophone output unit

Claims (5)

A plurality of wireless microphones installed at a predetermined position outdoors;
An audio receiver for receiving audio information from the wireless microphone and generating position information on the audio data when the position information of each of the plurality of microphones is stored in advance and the audio data is received from each microphone;
An image position information detector for detecting the position information of the image viewpoint from the stereoscopic image signal processor when there is a designation of a viewpoint at which a screen of a certain position of the TV screen is enlarged from the user;
A comparing unit for comparing the position information of the viewpoint with the position information of the audio data to detect position information of the audio data coinciding with the position information of the viewpoint and transmitting the corresponding audio data;
A stereo sound signal processing unit for signal processing the audio data transmitted from the comparison unit to three-dimensional acoustic information; And
And a stereo sound output unit for outputting the stereo sound signal processed by the stereo sound signal processing unit to the speaker.
delete The apparatus according to claim 1,
And detects two audio data at a position closest to a position of the view point and transmits the detected audio data to the signal processing unit when there is no audio data coinciding with the position of the view point. FTV stereophonic implementation using location information.
a) receiving audio data from a plurality of wireless microphones installed at a predetermined position outdoors, storing location information of each of the plurality of microphones in advance, and generating position information on each audio data received from each microphone ;
b) encoding and decoding audio data including the position information;
c) detecting position information of the image viewpoint when a viewpoint of a viewpoint of a predetermined position in the screen of the TV is specified by the user;
d) comparing the position information of the viewpoint with the position information of the decoded audio data, detecting position information of the audio data coinciding with the position information of the viewpoint, Transmitting;
e) signal processing the transmitted audio data into three-dimensional acoustic information; And
and f) outputting the signal-processed stereo sound to a speaker.
5. The method of claim 4, wherein in step d)
And detects and transmits two pieces of audio data at a position closest to a position of the image viewpoint when there is no audio data coinciding with a position of the image viewpoint. Acoustic implementation method.
KR1020150024880A 2015-02-23 2015-02-23 Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof KR101646867B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150024880A KR101646867B1 (en) 2015-02-23 2015-02-23 Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150024880A KR101646867B1 (en) 2015-02-23 2015-02-23 Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof

Publications (1)

Publication Number Publication Date
KR101646867B1 true KR101646867B1 (en) 2016-08-09

Family

ID=56712652

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150024880A KR101646867B1 (en) 2015-02-23 2015-02-23 Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof

Country Status (1)

Country Link
KR (1) KR101646867B1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3605531A4 (en) * 2017-03-28 2020-04-15 Sony Corporation Information processing device, information processing method, and program
CN111684823A (en) * 2018-02-07 2020-09-18 索尼公司 Transmission device, transmission method, processing device, and processing method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080098819A (en) * 2007-05-07 2008-11-12 광주과학기술원 A method and apparatus for view-dependent multi-channel audio processing for a multi-view camera system
KR20100006300A (en) 2008-07-09 2010-01-19 한국전자통신연구원 Method for file formation according to freeview av service
KR20100018802A (en) 2008-08-07 2010-02-18 한국전자통신연구원 Method and apparatus for free-viewpoint video contents offering according to scalable depth coding
KR20120116572A (en) * 2011-04-13 2012-10-23 김순종 The image which has spatial information creates and the system which remakes and the image creation remaking method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20080098819A (en) * 2007-05-07 2008-11-12 광주과학기술원 A method and apparatus for view-dependent multi-channel audio processing for a multi-view camera system
KR20100006300A (en) 2008-07-09 2010-01-19 한국전자통신연구원 Method for file formation according to freeview av service
KR20100018802A (en) 2008-08-07 2010-02-18 한국전자통신연구원 Method and apparatus for free-viewpoint video contents offering according to scalable depth coding
KR20120116572A (en) * 2011-04-13 2012-10-23 김순종 The image which has spatial information creates and the system which remakes and the image creation remaking method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
'자유시점 TV와 3차원 비디오 국제 표준화 동향/호요성, 이천' (정보통신표준화소식)TTA Journal. 통권 제116호 pp.86-94(2008년 3/4월 공개)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3605531A4 (en) * 2017-03-28 2020-04-15 Sony Corporation Information processing device, information processing method, and program
US11074921B2 (en) 2017-03-28 2021-07-27 Sony Corporation Information processing device and information processing method
CN111684823A (en) * 2018-02-07 2020-09-18 索尼公司 Transmission device, transmission method, processing device, and processing method
CN111684823B (en) * 2018-02-07 2022-07-01 索尼公司 Transmission device, transmission method, processing device, and processing method

Similar Documents

Publication Publication Date Title
US10257638B2 (en) Audio object processing based on spatial listener information
US11917221B2 (en) Encoding device and method, reproduction device and method, and program
CN1984310B (en) Method and communication apparatus for reproducing a moving picture
US20060279750A1 (en) Apparatus and method for converting image display mode
CN103491445A (en) Image display apparatus, mobile terminal and method for operating the same
CN103491404A (en) Mobile terminal and method for operating the same
KR102383117B1 (en) Display apparatus, display method and display system
CN107925840A (en) Method and apparatus for handling the audio signal associated with video image
KR101945813B1 (en) Image display apparatus, mobile terminal and method for operating the same
KR20110014821A (en) System and method for broadcasting scalable for three dimensional images broadcasting
KR20100040271A (en) Method for operating additional information of video using visible communication and apparatus for the same
US20130278781A1 (en) Image communication apparatus, image communication server and image processing method for image communication
KR101646867B1 (en) Apparatus for stereophonic realization of free-viewpoint television using microphone location and Method thereof
KR101005015B1 (en) A method and apparatus for an 3d broadcasting service by using region of interest depth information
US20190373244A1 (en) Method of transmitting 3-dimensional 360 degree video data, display apparatus using the method, and video storage apparatus using the method
WO2020031453A1 (en) Information processing device and information processing method, and video-audio output system
US11341976B2 (en) Transmission apparatus, transmission method, processing apparatus, and processing method
JP2012004991A (en) Broadcast receiving apparatus and control method for the same
US20120281073A1 (en) Customization of 3DTV User Interface Position
KR102255141B1 (en) Beam projector
KR101674187B1 (en) Apparatus for stereophonic acquisition for broadband interpolation and Method thereof
US10264241B2 (en) Complimentary video content
TW201101839A (en) System for encoding video and method, system for decoding video and method, system for displaying video and method
JP2024041721A (en) video conference call
KR101979270B1 (en) Image display apparatus, mobile terminal and method for operating the same

Legal Events

Date Code Title Description
E701 Decision to grant or registration of patent right
GRNT Written decision to grant
FPAY Annual fee payment

Payment date: 20190731

Year of fee payment: 4