KR102184131B1 - Multi channels transmitting system for dynamaic audio and controlling method - Google Patents

Abstract

The multi-channel transmission system for providing stereoscopic audio according to an embodiment records at the same time as recording for each camera location, and outputs audio at a corresponding location during channel conversion, so that the user can have a location flow of audio during playback by location. You can listen to the stereoscopic audio output while checking the image from a desired angle or position.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention Multi channels transmitting system for providing stereoscopic audio and controlling method thereof

The embodiments relate to a multi-channel transmission system for providing stereoscopic audio and a control method thereof.

Recently, the public prefers to play videos using mobile devices. In line with this preference, companies are offering broadcasting platforms such as V-app, AfreecaTV, and Youtube Live. The public watching these platforms are watching videos taken from one viewpoint, that is, one camera. However, recent viewers want to watch the video shot in the desired space.

Currently, a video service that provides a multi-channel image to a user by geometrically correcting and synthesizing a plurality of images obtained by photographing a subject in various channels with a plurality of cameras has been disclosed. These multi-channel images provide realistic images that go beyond the concept of high-definition, and through this, users feel more immersed in the media and can greatly enhance the effect of delivering image information in fields such as advertising, education, medical care, defense, and entertainment. have.

In the conventional multi-channel image, channel/time switching is a dimension that is simply reproduced by a predetermined merge method when producing a multi-channel image. That is, in the related art, a plurality of frames are acquired from a plurality of cameras, some of the acquired frames are selected, and these frames are merged to produce a single channel switching image. Since such a channel-switched video is a simple merge of the frames of a channel previously determined by the producer during video production, when the video file is played back, the merged frames exert a channel switching effect showing a single channel shift effect. According to such a multi-channel video of the prior art, the user is simply watching the previously produced channel switching effect, and the viewer watches the video while turning the channel to the desired point of time by manually manipulating visual switching or channel switching. It was impossible to do.

In addition, there is a need for a multi-channel or camera synchronization technique, an image correction and transmission method for capturing an image in various spaces, as well as a method for realistically providing stereoscopic audio generated in various spaces.

Embodiments provide a multi-channel transmission system for providing multi-view audio corresponding to a multi-view image and a control method therefor.

According to an embodiment, a subject is photographed for a predetermined time through a plurality of cameras, controlled to record audio input to each camera, and multi-channel images and recorded audio signals corresponding to the plurality of cameras are transmitted through a communication network. A camera control unit that transmits; And grouping and storing the multi-channel images transmitted from the camera control unit according to at least one criterion of time, channel, time and channel, and storing respective audio signals corresponding to the grouped images, and the user And a video server for transmitting the grouped video and audio signals corresponding to the grouped video through the communication network according to the request of the terminal.

The time and channel are mixed, and the first to Nth (N is a natural number of 2 or more) times corresponding to the first to Nth (where M is a natural number of 2 or more) channels are mixed, or the first to the first It is characterized in that the images of the Mth to the first channels corresponding to N times are mixed.

The time and the channel are mixed, and the images of the first to Mth channels corresponding to the Nth to the first times are mixed, or images of the Mth to the first channels corresponding to the Nth to the first time are mixed. It is characterized by mixing.

It is characterized in that images corresponding to a channel by time (y is a natural number) are grouped around an event corresponding to any one of the predetermined times.

Each of the audio signals corresponding to the grouped image is characterized in that the audio signals of different sizes recorded at different locations where the respective cameras are located.

When the grouped images are grouped for each channel, recorded audio signals having different sizes are output when the grouped images are reproduced.

The user request is a signal corresponding to zoom-in or zoom-out when playing the grouped video, and the audio size of the video being played is adjusted according to the signal corresponding to the zoom-in or zoom-out.

According to another embodiment, a control method of a multi-channel transmission system includes: photographing a subject for a predetermined time through a plurality of cameras, and controlling to record audio input to each camera; Transmitting multi-channel images and recorded audio signals corresponding to the plurality of cameras through a communication network; Grouping and storing the transmitted multi-channel images based on at least one of a mixture of time, channel, time and channel, and storing respective audio signals corresponding to the grouped images; And transmitting the stored grouped image and respective audio signals corresponding to the grouped image through the communication network according to the request of the user terminal.

Each audio signal corresponding to the grouped image is audio signals of different sizes recorded at different locations where the cameras are located, and when the grouped image is grouped by channel, the grouped image is played back. In this case, recorded audio signals of different sizes are output.

It includes a storage medium recording a program for executing the control method according to another embodiment in a computer.

The multi-channel transmission system for providing stereoscopic audio according to the embodiment may increase convenience of data transmission of a multi-channel image and reduce a processing speed when moving in space over time. In addition, by simultaneously recording and recording for each camera location, and outputting the audio of the corresponding location when switching channels, the user can have a positional flow of the audio when playing by location, and the user can check the video from a desired angle or position while checking the stereoscopic audio. You can listen to the output.

1 is a schematic diagram of a multi-channel transmission system 100 providing stereoscopic audio according to an exemplary embodiment.
2 is an exemplary diagram for generating a multi-channel image.
3 is a flowchart illustrating a method of controlling a multi-channel transmission system providing stereoscopic audio according to another embodiment.
4 is a schematic diagram of generating stereoscopic audio according to another embodiment.
5 and 6 are exemplary diagrams for outputting stereoscopic audio according to still other embodiments.
7 is a schematic diagram of the camera control unit 110 shown in FIG. 1.
8 is a schematic diagram of the image server 200 shown in FIG. 1.
9 to 11 are exemplary diagrams for explaining grouping of multi-channel images according to another embodiment.

The terms used in the embodiments have been selected as currently widely used general terms as possible while considering functions in the embodiments, but this may vary depending on the intention or precedent of a technician working in the art, the emergence of new technologies, etc. . In addition, in certain cases, there are terms that are arbitrarily selected, and in this case, the meaning will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents of the present embodiments, not a simple name of the term.

In the description of the embodiments, when a certain part is connected to another part, this includes not only a case in which it is directly connected, but also a case in which it is electrically connected with another component interposed therebetween. In addition, when a certain part includes a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, the term "... unit" described in the embodiments means a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

Terms such as “consisting of” or “comprising” used in the present embodiments should not be construed as necessarily including all of the various constituent elements or various steps described in the specification, and some constituent elements or some of them It should be construed that the steps may not be included, or may further include additional components or steps.

The description of the following embodiments should not be construed as limiting the scope of the rights, and what those skilled in the art can easily infer should be construed as belonging to the scope of the rights of the embodiments. Hereinafter, embodiments for illustration only will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a multi-channel transmission system 100 for stereoscopic audio transmission according to an exemplary embodiment.

Referring to FIG. 1, the transmission system 100 controls a plurality of cameras 111 to 113 and a plurality of cameras 111 to 113, processes multi-channel images captured from the plurality of cameras 111 to 113, and A camera controller 110 that transmits and processes audio recorded by a plurality of cameras 111 to 113, and an image server 200 that processes and stores multi-channel images and recorded audio signals transmitted from the camera controller 110 ). Each of the plurality of cameras 111 to 113 may be equipped with a microphone or may be equipped with a separate microphone to record a subject to be photographed and audio around the subject. The image server 200 receives a request for a multi-channel image, a request for a switching image, a request for a specific event, such as zoom-in or zoom-out, from the user terminal 150, and stores the multi-channel image and the multi-channel image, respectively. Audio signals corresponding to are transmitted to the user terminal 150.

The image server 200 may include an image processing device 130 and an image storage unit 140.

As shown in FIG. 2, a plurality of cameras 1 to N may be disposed to photograph a specific subject. A plurality of cameras arranged in an array around the subject receive a plurality of images photographing the subject from various angles. A method of arranging a plurality of cameras may be, for example, arranging N cameras in a line on substantially the same plane based on an arbitrary camera. In this case, N cameras may be sequentially placed on a circumference spaced a certain distance from the subject, or the distance between the two cameras disposed on both sides and the subject is matched based on the camera disposed in the center, and the remaining cameras It is also possible to make the distance to the subject different. Also, the subject may be a fixed subject or a moving subject. As described above, the plurality of cameras 1 to N may have built-in microphones or separate microphones to record the subject and audio around the subject. For example, you can record different outputs or different sounds at each camera position. The microphone converts audio data or voice electrical signals having waveform characteristics such as audio or sound waves. Here, the microphone is not limited to its type.

The plurality of cameras 1 to N and the camera control unit 110 may communicate with each other by wire or wirelessly, and may include a plurality of camera control units for controlling the plurality of cameras 1 to N.

The camera controller 110 may control the plurality of cameras 1 to N through a synchronization signal that synchronizes the plurality of cameras 1 to N. The camera control unit 110 temporarily stores images captured from a plurality of cameras 1 to N, and reduces the size of the captured image through codec change and enables fast transmission. In addition, the camera controller 110 may temporarily store audio signals recorded by a plurality of cameras 1 to N, and may compress the stored audio signals. A detailed configuration of the camera control unit 110 will be described with reference to FIG. 7.

The camera control unit 110 may include a camera driving control unit 111, an image conversion unit 112, an audio conversion unit 114, and a transmission unit 113.

The camera driving controller 111 controls camera synchronization and photographing.

The image conversion unit 112 reduces the size of images by changing a codec to facilitate fast transmission of images generated from a plurality of cameras 1 to N through the communication network 120. In addition, it is possible to determine a data transmission method suitable for the shooting site or for wired or wireless transmission.

The audio conversion unit 114 may reduce the size of audio signals by changing a codec to facilitate fast transmission of audio signals recorded from the plurality of cameras 1 to N through the communication network 120.

The transmission unit 113 transmits the converted images and audio signals to the image server 200 through the communication network 120.

The image server 200 groups and stores the multi-channel images transmitted from the camera control unit 110 and audio signals corresponding to each time based on at least one of a mixture of time, channel, time and channel, and stores the 150), the stored grouped video and corresponding audio signals are transmitted through the communication network. The detailed configuration of the image server 200 will be described with reference to FIGS. 8 to 11.

Referring to FIG. 8, the image server 200 includes an image processing device 110 and an image storage unit 140. The image processing apparatus 110 includes an image processing unit 131, an image conversion unit 132, a transmission unit 133, and an audio conversion unit 134.

The image processing unit 131 performs image correction on multi-channel images transmitted from the camera control unit 110, that is, images captured by a plurality of cameras. For example, since images captured by a plurality of cameras may be out of focus, image processing is performed so that the focus between the cameras is the same. The image processing unit 131 corrects the transmitted multi-channel images. Since the geometrical error of the array of N cameras appears as a visual shake in the process of reproducing a multi-channel image, at least one of the size or size, slope, or center position of each image may be corrected to remove this.

The image conversion unit 132 groups multi-channel images based on at least one criterion in which time, channel, time and channel are mixed. The image conversion unit 132 groups and groups several spaces into one. The grouping method can be performed according to various criteria. The transmission system according to the embodiment transmits all image data to transmit effective multi-channel images or switching images to the user terminal 150 without waste of data, and transmits grouped images to transmit only the data necessary for the user. To be. The image conversion unit 132 may group and group channel images for each time ±y (y is a natural number) based on the event at time t. For example, it may be a case in which an event occurs in channel 1 and t3. Here, the event may be a predetermined case, for example, a home run scene or an out scene in a baseball game, or an event at a user's request, for example, a zoom-in or zoom-out, or a case that the user desires.

As a basic method of grouping, channel images are grouped by time. Images of channels 1 to 5 captured at t1 are grouped into one to generate a grouped image. Similarly, images of channels 1 to 5 captured at t2 are grouped into one to generate a grouped image. In this way, grouping can be performed by time. For example, when a camera is arranged as shown in FIG. 2, the user terminal 150 can check images taken from all directions surrounding the subject at the same time. A method of grouping channel images by time has been described, but it is also possible to transmit each channel. Grouping may be performed in the order of cameras, that is, in the order of the first to fifth channels, but may be grouped in the reverse order, that is, in the order of the fifth to first channels.

In addition, grouping may be performed by time and channel combinations. In descending order, the first channel image photographed at t1, the second channel image photographed at t2, the third channel image photographed at t3, the fourth channel image photographed at t4, and the fifth channel image photographed at t5. Grouped images are created by grouping them into one. Similarly, the first channel image photographed at t2, the second channel image photographed at t3, the third channel image photographed at t4, the fourth channel image photographed at t5, and the fifth channel image photographed at t6 are grouped into one. To create a grouped image. In ascending order, the fifth channel image photographed in t1, the fourth channel image photographed in t2, the third channel image photographed in t3, the second channel image photographed in t4, and the first channel image photographed in t5 are Grouped images may be created by grouping them into one. Grouping of images obtained by mixing time and channel has been described, but is not limited thereto, and grouping is possible in various ways.

Referring back to FIG. 8, the grouped images converted by the image conversion unit 132 are stored in the image storage unit 140. When there is a request from the user terminal 150, the image processing apparatus 130 extracts images stored in the image storage unit 140 and transmits the extracted images to the user terminal 150 through the transmission unit 133.

The audio conversion unit 134 compresses the input audio signal. The audio signal compression may be an adaptive transform audio encoding apparatus using human auditory characteristics. In such compression or encoding processing, an audio signal in the time domain is transformed into the frequency domain. In addition, the signal on the frequency axis is divided into a frequency band corresponding to the frequency resolution capability of the hearing. And, using the human auditory characteristics, the optimal amount of information required for encoding in each frequency band is calculated. Then, the signal on the frequency axis is quantized according to the amount of information allocated to each frequency band. Among the adaptive transform audio encoding apparatuses, an MPEG (Moving Picture Experts Group)-2 AAC (Advanced Audio Coding) scheme standardized by an International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) may be used. Advanced Audio Coding (hereinafter referred to as AAC, standard document ISO/IEC 13818-7) is a standard lossy data compression method used in digital audio. AAC extends the sample frequency from 8Khz to 96Khz, extends up to 48 channels, and can variably allocate bits as needed even at a fixed bit rate, and change it to a Modified Discrete Cosine Transformation format. It enables efficient encoding.

In addition, the audio conversion unit 134 may store an audio signal corresponding to an image grouped by the image conversion unit 132 in an image storage unit or may provide the audio signal to the transmission unit 133.

Referring to FIG. 9, the image server 130 includes an image processing unit 131, an image conversion unit 132, a transmission unit 133, and an audio conversion unit 134, and the transmission unit 133 includes a message handler ( 133-1), a streamer 133-2, and a channel manager 133-3. The transmission unit 133 may be a streaming device and is described as being included in the image server 130, but it is of course possible to implement it as a separate device separated from the image server 130. A description of the same parts as in FIG. 8 will be omitted, and differences will be mainly described.

The transmission unit 133 transmits the processed image or the stored image in real time. For example, it may be a device for real-time streaming.

The message handler 133-1 performs session management and protocol management with the user terminal.

The streamer 133-2 transmits video and audio signals to the user terminal and is a set of images to be transmitted to the user terminal.

The channel manager 133-3 receives a signal from the user terminal, schedules an image in units of GOP, and delivers it to the streamer 133-2.

When there is no special request from the user terminal, for example, a channel change request or a channel movement request, the channel manager 133-3 delivers the scheduled video to the streamer 133-2 in each GOP unit. As shown in FIG. 10, channels 5, that is, images captured by the camera 5, are scheduled in the order of GOP1 to GOPN, and then transmitted to the streamer 133-2. The streamer 133-2 collects GOP images according to the scheduling of the channel manager 133-3 and transmits them to the user terminal.

As shown in FIG. 11, when a channel movement request is received from the user terminal, the channel manager 133-3 performs channel 5, that is, GOP1 to 3 among images captured by camera 5, GOP4 of channel 4, and channel 3 GOP5 of the channel 2, GOP6 of the channel 2, and GOP7 of the channel 7 are scheduled in the order and transmitted to the streamer 133-2. The streamer 133-2 collects GOP images according to the scheduling of the channel manager 133-3 and transmits them to the user terminal. At this time, along with the GOP images, different audio signals corresponding to each image are transmitted to the user terminal. Accordingly, the user listens to the audio signal recorded by camera 5 together with the GOP 1 to 3 video, listens to the audio signal recorded by the camera 4 with the GOP 4 video, and the audio signal recorded by the camera 3 with the GOP 5 video, You can listen to the audio signal recorded by camera 2 with GOP 6 video, and listen to the audio signal recorded by camera 1 with GOP 7 video. Accordingly, audio signals recorded differently from cameras 5 to 1 can be heard in three dimensions.

3 is a flowchart illustrating a method of controlling a multi-channel transmission system according to another embodiment.

Referring to FIG. 3, in step 300, a subject is photographed for a predetermined time through a plurality of cameras, and audio input to each camera is recorded. Here, as shown in FIG. 2, the cameras are arranged in a circle to photograph a subject, and each camera may include a built-in microphone or a separate microphone.

Referring to FIG. 4, with a sound source 400 disposed in the center, five cameras and microphones 410 to 450 are disposed from point A to point B. In this case, sounds input at the same time at each point have different sizes and frequencies. In an embodiment, each of the cameras and microphones 410 to 450 may not only photograph the same subject at each point, but also record audio obtained at the corresponding point.

In step 302, multi-channel images corresponding to the plurality of cameras and recorded audio signals are transmitted through a communication network.

In step 304, the transmitted multi-channel images are grouped and stored according to at least one criterion in which time, channel, time and channel are mixed, and audio signals corresponding to the grouped images are stored.

In step 306, the grouped image stored according to the request of the user terminal and respective audio signals corresponding to the grouped image are transmitted through the communication network.

In an embodiment, by simultaneously recording for each camera location and outputting the audio at the corresponding location when changing channels, the location of the audio can be flowed during playback by location, and the user can check the video at a desired angle or location while You can listen to the stereoscopic audio output.

Referring to FIG. 5, when a user zooms in or zooms out while playing a grouped image transmitted from a playback terminal (not shown), a zoom factor, for example, 100% to 250%, increases the audio output. You can increase it accordingly.

Referring to FIG. 6, changes in audio output through left and right speakers when switching from point A to point B shown in FIG. 4 are illustrated.

As for the audio recorded by the camera 420 shown in FIG. 4, the largest audio is output through the left speaker, and the smallest audio is output through the right speaker. In addition, as for the audio recorded by the camera 440 shown in FIG. 4, the largest audio is output through the right speaker, and the smallest audio is output through the left speaker. However, the cameras 410, 430, and 450 shown in FIG. 4 output audio as the sound source. Accordingly, when the user sees an image while switching from point A to point B, the user can listen to audio in which the audio output of the left and right speakers changes in three dimensions.

An embodiment may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery medium.

The above description is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily transformed into other specific forms without changing the technical spirit or essential features of the present invention. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (10)

A camera control unit for photographing a subject for a predetermined time through a plurality of cameras, controlling to record audio input to each camera, and transmitting multi-channel images and recorded audio signals corresponding to the plurality of cameras through a communication network; And
The multi-channel images transmitted from the camera control unit are grouped and stored according to at least one criterion in which time, channel, time and channel are mixed, and respective audio signals corresponding to the grouped images are stored, and A video server for transmitting the grouped video and audio signals corresponding to the grouped video through the communication network according to a request,
Each of the audio signals corresponding to the grouped images are audio signals of different sizes recorded at different locations where the respective cameras are located,
When the grouped video is grouped by channel, recorded audio signals of different sizes are output when the grouped video is played back,
The video server,
A message handler that performs session management and protocol management with the user terminal;
A streamer for transmitting the grouped video and audio signals corresponding to the grouped video to the user terminal; And
According to the request of the user terminal-the request is a channel change request-scheduling the grouped video in GOP units, and providing the scheduled videos in GOP units and audio signals corresponding to each to the streamer Includes a channel manager,

Multi-channel transmission system, characterized in that. The method of claim 1,
The criteria for mixing the time and channel are:
Images of the first to Mth (M is a natural number of 2 or more) channels corresponding to the first to Nth (N is a natural number of 2 or more) times are mixed, or the M to the first corresponding to the first to Nth times Mix the images of the channels,
Mixing images of the first to Mth channels corresponding to the Nth to the first times, or mixing the images of the Mth to the first channels corresponding to the Nth to the first hours,
The first to Nth times mean an order according to the passage of time,
The Nth to the first time means the reverse order according to the passage of time,
The first to Mth channels mean a channel order according to a change in the position of the camera,
The M th to the first channel means a reverse order of channels different from a change in the position of the camera. delete The method of claim 2,
A multi-channel transmission system comprising grouping images corresponding to a channel according to ±y (y is a natural number) time based on an event corresponding to any one of the first to Nth times. delete delete The method of claim 1,
The request of the user terminal,
In the case of reproducing the grouped video, the multi-channel transmission system is a signal corresponding to zoom-in or zoom-out, and the audio size of the video being played is adjusted according to the signal corresponding to the zoom-in or zoom-out. Photographing a subject for a predetermined time through a plurality of cameras, and controlling to record audio input to each camera;
Transmitting multi-channel images and recorded audio signals corresponding to the plurality of cameras through a communication network;
Grouping and storing the transmitted multi-channel images based on at least one of a mixture of time, channel, time and channel, and storing respective audio signals corresponding to the grouped images; And
Transmitting the stored grouped image and respective audio signals corresponding to the grouped image through the communication network according to a request of a user terminal,
Each of the audio signals corresponding to the grouped images are audio signals of different sizes recorded at different locations where the respective cameras are located,
When the grouped video is grouped by channel, recorded audio signals of different sizes are output when the grouped video is played back,
Perform session management and protocol management with the user terminal, and transmit the grouped video and audio signals corresponding to the grouped video to the user terminal,
In accordance with the request of the user terminal-the request is a channel change request-the grouped video is scheduled in GOP units, and the scheduled videos in the GOP unit and audio signals corresponding to each are provided. Control method of multi-channel transmission system. delete A recording medium on which a program for executing the method according to claim 8 on a computer is recorded.

Images