KR100748060B1 - Internet broadcasting system of Real-time multilayer multimedia image integrated system and Method thereof - Google Patents

Abstract

The present invention relates to an internet broadcasting system using a real-time multilayer video synthesizing system and an internet broadcasting providing method using the same, wherein an internet broadcasting system using a real-time multilayer video synthesizing system is an image input in real time in an internet broadcasting system via a network. Real-time multi-layer video synthesizing system for generating a multi-layer composite video by synthesizing data, sound data, and background video in real time, converting the multi-layer composite video into a streaming file according to a predetermined codec in real time, and transmitting it through the network; A web server for receiving a streaming file transmitted from a multi-layer video synthesis system and transmitting in real time through the network and a plurality of clients accessing the web server through the network to receive the streaming file. The Internet broadcasting system using a real-time multi-layer composite video system comprising the end is provided.

According to the present invention, there is an advantage in that it is possible to provide live internet broadcasting without any additional equipment or editing process, and also, there are advantages in that various special effects are processed in real time to provide higher quality live internet broadcasting.

Internet broadcasting, online broadcasting, real time compositing, multilayer video, video compositing

Description

Internet broadcasting system using real-time multilayer video synthesis system and method for providing internet broadcasting using same

1 is a schematic block diagram of an internet broadcasting system using a real-time multi-layer video synthesis system according to an embodiment of the present invention.

2 is a block diagram of a real-time multi-layer video synthesis system according to an embodiment of the present invention.

3 is a block diagram of a web server according to an embodiment of the present invention.

FIG. 4 is a flowchart illustrating a process in which a video is captured and synthesized in real time in a real-time multilayer video synthesizing system according to an exemplary embodiment of the present invention.

5 is a flowchart illustrating a method of providing real-time Internet broadcasting in a web server according to an exemplary embodiment of the present invention.

Explanation of symbols on the main parts of the drawings

100: real-time multilayer video synthesis system 102: network

104: Web server 106: Best production video DB

108: Use automatic video DB 110: High Scorer video DB

210: real-time multilayer video synthesizing apparatus 222: image processing unit

224: sound processor 226: file processor

230: project control unit 310: broadcasting control unit

320: database control unit

The present invention relates to an internet broadcasting system using a real-time multilayer video synthesizing system. More particularly, a video of a user acting according to the direction of a virtual director is synthesized into a real-time multi-layer synthetic video using a real-time multilayer video synthesizing system, and synthesized. By receiving the multi-layered composite video from a web server and transmitting it to a plurality of client terminals connected to the web server, an internet broadcast can enable a large number of viewers to watch a high quality internet broadcast video with video synthesis and various special effects in real time. The present invention relates to a system and a method for providing Internet broadcasting using the same.

In general, the Internet broadcasting system has a structure in which a broadcast video is produced in a separate studio for producing a broadcast video, and the internet video is transmitted by using a server.

In producing a broadcast video, convert the analog video input through the recording device into a digital video, synthesize the background to the digital video, or add various special effects to produce the final broadcast video, and transmit it to the web server Thus, a method of providing a plurality of viewers connected to a web server using a web server has been used.

In addition, in order to provide a video for broadcasting, a separate director had to produce a performance or a song or dance to the subject (Internet broadcaster) in accordance with the editing intention, and to shoot and broadcast it through a recording device, a large number of personnel There was a problem that it was necessary.

Since the production and editing of such broadcast videos required expensive maintenance and a lot of time, in reality, it was common for Internet broadcasting to edit and re-transmit the recorded video of the air broadcasting. There was a problem that there are many restrictions.

To solve these problems, a number of video producers are currently used, and most of them produce chromatic video using a chroma-key technique to synthesize a background and a user's appearance.

The chroma key technique is a video synthesis technique that adds special effects and text to a video. The chroma key technique is most frequently used when compositing with other videos by removing specific color parts and extracting only other elements.

However, when synthesizing a video using the existing chroma key technique, there is a problem that an expensive synthesis apparatus is required or real-time synthesis is difficult.

In addition, in the case of the existing video producer, there is no directing function, so users have to postpone the action arbitrarily, and thus, there is a problem that the awkward direction is made so that a proper video cannot be produced, and high quality special effects are added. There was a problem in that it is not possible to produce and service a composite image in real time.

The present invention generates a multi-layer composite video in real time using a real-time multi-layer video synthesis system that can give a special effect by synthesizing the background image to the video input in real time in order to solve the problems of the prior art as described above In addition, the present invention provides an internet broadcasting system using a real-time multilayer video synthesizing system that can provide a plurality of internet broadcasting viewers through a web server, and an internet broadcasting providing method using the same.

In order to achieve the object as described above, according to an embodiment of the present invention, in the Internet broadcasting system via a network, a multi-layer composite video by synthesizing the video data, audio data and background video input in real time in real time A real-time multi-layer video synthesizing system for generating the multi-layer composite video and converting the multi-layer video into a streaming file according to a predetermined codec in real time; A web server receiving the streaming file transmitted from the real-time multilayer video synthesizing system and transmitting the streaming file in real time through the network; And a plurality of client terminals connected to the web server via the network to receive the streaming file.

In order to solve the problems of the prior art as described above, according to another embodiment of the present invention, there is provided an audio / video input device, a display device, a key input device, a background screen, a background video database and a model video clip database A method of providing a real-time internet broadcasting using a web server having a real-time multilayer video synthesizing system and a model video database and an internet broadcasting system in which a plurality of client terminals are connected through a network, the method comprising: (a) the audio in the real-time multilayer video synthesizing system; A real-time multilayer video is generated by synthesizing the video inputted from the video / video input device and the background video, converting the real-time multilayer video into a streaming file, and transmitting the streaming file and the video composite information to the web server in real time. ; And (b) receiving the streaming file and the video synthesis information transmitted in the step (a) from the web server to construct a video for internet broadcasting and transmitting the video in real time. An internet broadcast providing method is provided.

Hereinafter, with reference to the accompanying Figures 1 to 3 will be described in detail a preferred embodiment of a real-time multi-layer video synthesis system according to the present invention.

1 is a schematic block diagram of an internet broadcasting system using a real-time multi-layer video synthesis system according to an embodiment of the present invention.

As shown in FIG. 1, the internet broadcasting system using the real-time multilayer video synthesis system according to the present invention includes a real-time multilayer video synthesis system 100, a network 102, a model video DB 106, and an automatic video DB used. It may include a web server 104 and a plurality of client terminals 120 including the 108 and the highest score video DB (110).

The real-time multilayer video synthesizing system 100 generates a multi-layer composite video by synthesizing image data, audio data, and background video input in real time, and converts the generated multi-layer composite video into a streaming file according to a predetermined codec in real time. To transmit through the network 102.

The web server 104 receives the streaming file transmitted from the real-time multilayer video synthesis system 100 and transmits it in real time through the network 102, and the client terminal 120 transmits the web server 104 through the network 102. ) To receive the streaming file.

The best-presenting video database 106, the automatic video database 108, and the high-scoring video database 110 are connected to the web server 104 to database the video data under the control of the web server 104, and the web server. According to the control of 104, corresponding video data is extracted and provided.

The best-presenting video database 106 is stored in the best-presenting video (generally animation) that has been produced by using a motion capture format in advance, and the user video database 108 stores user's data transmitted from the real-time multilayer video synthesizing system 100. The composite video is structured and stored together with the user's personal information.

In addition, the highest scorer video database 110 is configured by the real-time multi-layer video synthesis system 100, the video recorded the highest score among the broadcast video transmitted by the web server 104 is structured and stored.

The internet broadcast viewer accesses the web server 104 through the network 102 using the client terminal 120, and receives the internet broadcast data composed of the multi-layered composite video synthesized by the real-time multi-layered video synthesizing system 100. The display may be viewed using a display device (not shown) connected to the client terminal 120.

2 is a block diagram of a real-time multi-layer video synthesis system according to an embodiment of the present invention.

With reference to the accompanying drawings, it will be described in detail the configuration of a real-time multi-layer video synthesis system according to the present invention.

As shown in FIG. 2, the real-time multi-layer video synthesizing system 100 according to the present invention includes a real-time multi-layer video synthesizing apparatus 210, at least one display device 250, an audio input device 252, and at least one It may include a video input device 256, a background screen 258, and a key input device 260.

The real-time multi-layer video synthesizing apparatus 210 receives a user's control signal through the key input device 260, displays a model video corresponding to the user's control signal through the display device 250, and displays a video input device ( 256) and the audio input device 252 is controlled by receiving the user's video input in real time with the background screen 258 as a background, and synthesized with the background video in real time to generate a multi-layer composite video, and then convert it into a streaming file. Will be converted.

When the key operation of the user is input, the key input device 260 outputs a control signal corresponding to the key operation to the real-time multilayer video synthesizing apparatus 210, and the real time multilayer video synthesizing apparatus 210 inputs the control signal. According to the real-time video synthesis is performed.

The background screen 258 provides a background of a single color to obtain a background separated image for image synthesis according to the chroma key technique.

Typically, when image synthesis is performed using a human video, a blue or green background screen may be used.

The audio input device 252 converts an audio signal into an electrical signal and outputs the electrical signal to the real-time multilayer video synthesizing apparatus 210. The video input device 256 converts an optical signal into an electrical signal to real-time multilayer video synthesizing apparatus ( 210) is output to the side.

The display device 250 displays various types of information and a video output from the real-time multilayer video synthesizing apparatus 210. In order to obtain video synthesis information that can control the video synthesis process and the synthesis information as a whole, real-time multilayer video synthesis A video synthesis menu provided by the device 210 and an image of a virtual performer to be directed to the user may be displayed.

In addition, a plurality of display devices 250a to 250n are provided so that one exemplary display video of a virtual performer is displayed on one display device 250a, and a multi-layer composite video synthesized in real time is displayed on another display device 250b. It may be configured.

The real-time multilayer video synthesizing system according to the present invention may further include a position sensor 254 for measuring a user's position and outputting it as an electrical signal.

The position sensor 254 may be configured of various sensors such as a pressure sensor or an optical sensor. The position sensor 254 measures the position of the user and outputs the measured position to the real-time multilayer video synthesizing apparatus 210.

The real-time multilayer video synthesizing apparatus 210 selects one video input device suitable from among a plurality of video input devices as a photographing video input source according to the user's location information input from the position sensor 254.

In the case where the position sensor 254 is composed of a pressure sensor, the horizontal and vertical grids are partitioned on a fixed footrest (a foot where the user smokes from above), the pressure sensor is arranged on the intersection of each grid, and then the position coordinates for each pressure sensor It is possible to determine the position of the user by using the signal output from the pressure sensor when the user's foot depresses the pressure sensor.

In this case, the real-time multilayer video synthesizing apparatus 210 may select a video input apparatus 256 as an image signal input means capable of obtaining an optimal image according to a user's position among a plurality of connected video input apparatuses 256a to 256n. As the selection method, a method of matching each video input device 256a to 256n according to a predetermined user position may be used.

That is, the first video input device 256a is selected when the user's position is xx-yy, and the second video input device 256b is selected when the user's position is yy-zz. It is possible to obtain an optimal image according to.

As shown in FIG. 2, the real-time multilayer video synthesizing apparatus 210 according to the present invention includes a main controller 212, a RAM 214, a ROM 216, a storage unit 220, an input / output interface 218, The image processor 222, the sound processor 224, the file processor 226, and the communication controller 228 may be included.

The RAM 214 is used as a temporary storage place of various data generated during image processing and sound processing in a real time multilayer video synthesizing system, and a ROM and a program for driving a real time multilayer video synthesizing apparatus are included in the ROM 216. Algorithms and sound processing algorithms are stored.

The input / output interface 218 controls data input / output with an external input / output device and forms a path for data input / output.

The main controller 212 uses a program for driving the real-time multilayer video synthesizing apparatus 210 stored in the ROM 216, an image processing algorithm, and an audio processing algorithm to control signals output from the key input device 260. Accordingly, the user's photographed video input from the video input device 256 and the audio input device 252 is combined with the background video in real time to generate a multi-layer composite video.

The storage unit 220 physically stores the background video database 240 pre-recorded according to the various stories and the background video database 240 stored in the structured exemplary video of the virtual performer and the modeled video database 242 stored in the real time. The multi-layer video and the various information are synthesized to be stored.

The background video is an image to be synthesized with an image (ie, a user's video) input through an external input device when the user wants to synthesize a video in real time using the real-time multilayer video synthesizing apparatus 210. Speak video taken according to the story.

The background video database 240 is a database in which such a background video is structured and stored. The background video database 240 provides a list of the background videos stored through the display apparatus under the control of the main control unit 212. According to the control, the background video designated by the user is extracted and output to the main controller 212.

In addition, by adding an alpha channel to the pre-photographed background video to store in the storage means 220, it may be configured to enable more smooth resource utilization when real-time video synthesis.

The exemplary video is a virtual motion picture of the virtual director provided to enable the user to produce a natural video. The exemplary video is produced by an animation motion capture and stored in the exemplary video video database 242.

The exemplary rendering video database 242 displays a list of stored exemplary rendering videos on the display device 250, and when the user operates the key input device 260 to select the exemplary rendering video according to the video composition, the user's selection According to the exemplary rendering video according to the display device 250 is displayed.

The sound processor 224 digitizes the sound data input through the audio input device 252 and outputs digital sound data.

In addition, the sound processing unit 224 according to the present invention analyzes the analog sound data converted to digital, and configures and outputs sound information including sound information such as frequency and output of the sound, such sound information is the user When the acoustic visualization effect is selected as the moving picture synthesis information, it is used to generate the multi-layered synthesis moving picture.

The image processor 222 synthesizes the image data input through the video input device 256 and the background video stored in the background video database 240 in real time to output a multi-layer composite image.

More preferably, the image processor 222 according to the present invention may include an input video processing module 270, a background video processing module 272, an image synthesizing module 274, and a special effect module 276.

The input video processing module 270 converts analog image data input from the video input device 256 into digital image data, and sets color information of a specific color corresponding to a specific key value stored in the digital image data as transparent to separate the background. Create an image.

In this case, the key value used for the background separation is set as the color information of the background screen when the user's video is generated using the background screen of a specific single color (for example, blue, green, etc.).

The color information of the digitized image data includes 256 levels of values for each of the R (red), G (green), and B (blue) channels.

In order to separate a background from an input video, a photographing is performed on a single color background, the background color is set as a key value, and color information of a color corresponding to a key value among digital image data of the input video is made transparent. Typically, r: 0 g: 0 b: 0) is used to separate the background.

For example, when shooting a video on a blue background, a blue RGB channel value is stored as a key value stored in the memory, and all color information corresponding to blue among the digital image data of the video input in real time is a transparent value. Is converted to a background separation image.

In this case, the key value may be set to a specific RGB channel value, or may be set to an arbitrary range of RGB channel values for efficient separation of the background.

In addition, the color information of a specific range is set to transparent, and the specific range color information (the range that can be judged not the background, that is, the color information of the boundary range where the distinction between the user's image and the background image is not clear) is determined. The transparency may be adjusted to generate a more natural background separation image.

The background video processing module 272 extracts a background video corresponding to a control signal output from the key input device 260 from the background video database 240, and determines whether an alpha channel is present in the extracted background image. If the channel does not exist, an alpha channel is added and the rear background image and the front background image are output by changing the alpha value of the alpha channel according to the sound information output from the sound processor 224.

The background video processing module 272 processes and outputs the background video to be synthesized with the video input in real time.

That is, the background video processing module 272 processes and outputs the rear background image (the image synthesized as the background of the background separation image) and the front background image (the image synthesized on the front of the background separation image) of the video.

 The background video processing module 272 extracts a background video based on the video synthesis information from the background video database 240 and generates an alpha channel by determining whether an alpha channel exists in the extracted background video. If not, an alpha channel is added to the background video, and the alpha value of the alpha channel of the background video is changed according to a predetermined algorithm to generate and output a rear background image.

At this time, according to the sound information output from the sound processor 224, the alpha value (transparency) of the back background image alpha channel may be changed according to a predetermined algorithm, so that a composite image dynamically corresponding to the sound effect may be generated. It will be apparent to those skilled in the art.

In addition, the background video processing module 272 extracts the background video stored in the background video database 240 according to the video synthesis information or corresponds to the sound information output from the sound processor 224 in order to generate the front background image. An audio visualized image is generated, and the front background image is generated by outputting an alpha channel to the background video or the audio visualized image.

That is, when the user designates the background video to be used as the front background image, the front background image is generated based on the background video. If the user selects the sound visualization image as the front background image, the sound processor 224. Receiving the sound information output from the to generate a front background image based on the acoustic visualization image accordingly.

The process of the case where the user designates the background video to be used as the front background image is the same as the generation process of the back background video described above, and thus the detailed description thereof will be omitted.

If the user selects the sound visualization image as the front background image, the background image processing module 272 receives the sound information output from the sound processor 224 and generates a sound visualization image accordingly.

The sound visualization technology is to visually express sound information such as sound volume and frequency band, and refers to a technology for generating an image according to sound information according to a predetermined algorithm.

For example, it is possible to set color information and figure information according to frequency bands, and to configure an image displaying different colors and figures according to the input sound information. Can be configured to be adjusted.

In the present invention, an alpha channel is added to a background video or an audiovisualized image, and the alpha value of the alpha channel is changed according to the sound information output from the sound processor 224, and the front background dynamically changes in response to a change in sound. Create a video.

The image synthesizing module 274 synthesizes the background separated image output from the input video processing module 270 and the back background image and the front background image output from the background video processing module 272 in real time to output a multi-layer composite image. .

The special effect module 276 outputs a variety of special effects according to the video synthesis information to the multi-layer composite image output from the image synthesis module.

In this case, various image special effects such as three-dimensional stereoscopic effect, light source effect, and shade effect may be used as the special effect used.

In addition, when the real-time multi-layer video synthesis system according to the present invention further comprises a position sensor 254, the image processing unit 222 may further include a camera switching module 278 and the scene change module 280.

As described above, the position sensor 254 measures and outputs the position of the user.

The camera switching module 278 selects the most suitable video input device 256 among the plurality of video input devices 256 according to the user's position output from the position sensor 254 and receives a video signal to receive an input video processing module ( 270).

The scene change module 280 provides a scene change effect according to the change of the video input device 256 by the camera switching module 278.

The scene change module 280 analyzes the multi-layer composite image output from the special effect module 276 according to a predetermined algorithm and, when a scene change occurs, performs a scene change effect on the multi-layer composite image and outputs it.

The basic method for scene change detection is to determine that a scene change has occurred when the difference in feature values between successive frames is equal to or greater than a preset threshold.

However, if it is determined whether or not to change the scene only in this manner, the abrupt scene change can be detected relatively easily, but there is a problem that it is difficult to detect the scene change.

Accordingly, in the present invention, whether or not to change scenes is determined using an algorithm for detecting a gradual change of scene by observing changes in feature values before and after the change of scene.

HSV color histogram was used as a feature value in the present invention, and each component of the histogram was quantized to 4 bits to be less sensitive to small changes in the image.

In addition, in order to determine the difference between the frames, the change in the characteristic value between the frames is determined using a method of comparing the difference between X ² .

In addition, in the present invention, it is determined whether to change scenes by using the adaptive threshold, and the average of feature values in the window is set as the adaptive threshold by covering a local window having a length N before and after the current frame.

That is, M _T (adaptive threshold) = T _P (preselected threshold) * U _N (average value in the local window).

When the feature value of the current frame is larger than the adaptive threshold and at the same time the maximum value in the window, it is determined that the scene change has occurred.

In addition, in the present invention, in order to effectively detect a sudden scene change and a gradual scene change, a twin comparison is applied. In the case of a twin comparator, two threshold values are set so that a feature value is higher than or equal to a high threshold value. If it is determined that the scene change is abrupt, and if the threshold value is higher than or equal to the low threshold value, the feature value is accumulated in a section having a value higher than or equal to the low threshold value, and if the accumulated feature value is larger than the given threshold value, it is determined as a progressive scene change.

In order to solve the problem that the output image looks unnatural when the scene change is made, in the present invention, the scene change module 280 is used to determine whether the scene is changed, and when it is determined that there is a scene change, various special By using effects (e.g. fade-in / fade-out effects, transitions from the scene before the transition to the transitioned screen), more natural transition video (i.e. the transition between frames before and after the transition is not awkward and natural) Video) can be provided.

In addition, by receiving the video input device 256 switching signal output from the camera switching module 278, regards the switching of the video input device 256 as a scene change, it is configured to give a scene change effect according to the scene change May be

The file processing unit 226 synchronizes the multi-layer composite image output from the image processing unit 222 and the digital audio data output from the sound processing unit 224, combines them into a single file, and compresses it into a streaming file according to a predetermined codec.

In this case, compression may be performed according to various commercial codecs, and preferably, may be compressed into an asf file format capable of real-time Internet broadcasting according to the MPEG4 codec.

The communication controller 228 transmits the streaming file and the video composition information output from the file processor 226 to the web server 104 connected through the network 102 in real time.

In addition, the communication controller 228 may be configured to periodically update the data of the background video database 240 and the model video clip database 242 by receiving control signals and data transmitted from the web server 104.

In addition, the real-time multi-layer video synthesizing apparatus 210 according to the present invention stores a menu screen for selecting a video compositing option to the user, and the video compositing information desired by the user input through the menu screen, according to the stored video compositing information It may further include a project control unit 230 for totally controlling the real-time multi-layer video synthesis process.

Project control unit 230 according to the present invention may include a project control module 232 for controlling the overall video synthesis process.

When power is applied to the real-time multilayer video synthesizing system 100, the project control module 232 displays a video synthesizing menu through the display apparatus 250.

The video composition menu includes a list of background videos stored in the background video database 240 and a list of best performing videos stored in the best-presentation video database 242, video special effects, whether to switch cameras, visual sound effects, and various kinds. It is configured to allow the user to select the options required for video composition.

The project control module 232 receives and stores video composite information that the user manipulates and outputs the key input device 260, and controls all means in the real-time multi-layer video composite apparatus 210 according to the video composite information. The desired video is synthesized.

In addition, the project controller 230 according to the present invention may further include a motion analysis module 234.

The motion analysis module 234 compares the user's video with the modeling video, determines how much the user postpones (acting, dancing, etc.) to match the motion of the modeling video, and calculates the result as a composite system score. do.

The scoring of the user's motion is to extract only the basic motion (except when connected) among the movements of the modeling movie, and to convert the frame buffer's change value (the background is 0 and the subject is 1) into the database in the operation time and frame. Save the data.

That is, the basic motion is regarded as a captured still image, and the stored information of the frame buffer of the still image is divided into a background image and a subject model image and digitized to be used for calculating a composite system score.

When the user's image is input to the motion analysis module 234 through the image processor 222, the motion analysis module 234 extracts the basic motion from the user's image and converts the frame into data. Compared with, it determines whether the operation is matched.

On the other hand, when converting the user's video data, the user has to have a different body size, key, etc., unlike a virtual producer, so in this case, an error inevitably occurs even when the user's basic operation value is extracted and compared with the database value. Done.

In order to solve this problem, the present invention sets a standard category according to the physical condition, such as the user's physique, and if it exceeds or falls short of the category, the error is automatically reduced or enlarged within the standard category range according to certain data. Will be minimized.

The synthesis system scoring score (information) output from the motion analysis module 234 is displayed in real time through the display apparatus 250 and is transmitted to the web server 104 via the network 102.

3 is a block diagram of a web server according to an embodiment of the present invention.

With reference to the accompanying drawings, the configuration of the web server 104 according to an embodiment of the present invention will be described in detail.

As shown, the web server 104 according to the present invention is the main control unit 300, communication control unit 302, storage unit 304, RAM 306, ROM 308, broadcast control unit 310 and database It may include a control unit 320.

The RAM 306 is used as a temporary storage place of various data generated in the process by the web server 104, and the ROM 308 stores a program for operating the web server 104 and an internet broadcast control algorithm.

The main control unit 300 controls the overall Internet broadcasting in accordance with a program for driving the web server 104 stored in the ROM 308 and an internet broadcasting control algorithm.

The storage means 304 stores a user video database 108 in which a user video transmitted from a real-time multilayer video synthesis system is structured and stored, a model video clip database 106 stored in a structured model, and a variety of information. Will be.

The communication control unit 302 receives a streaming file transmitted from the real-time multilayer video synthesis system 100, video synthesis information, and synthesis system scoring information, and connects the broadcast data configured under the control of the main control unit through the network 102. To 120.

In addition, the communication control unit 302 is transmitted to the real-time multi-layer video synthesis system 100 via the network, if the model of the best-executed video or background video is additionally registered, the background video database ( 240 or the model video clip database 242 may be updated.

Database control unit 320 controls the construction and management of the database connected to the web server 104, the broadcast control unit 310 is a streaming file and video transmitted from the real-time multi-layer video synthesis system 100 through the communication control unit 302 The broadcasting information is composed by receiving the synthesis information and the synthesis system scoring information in real time.

The broadcast controller according to the present invention may more preferably include a video control module 312 and a real-time scoring module 314.

The real-time scoring module 314 calculates the synthesis system scoring information transmitted from the real-time multilayer video synthesis system 100 for each time zone, and outputs an average score and a cumulative score.

The video control module 312 uses the streaming information transmitted from the real-time multilayer video synthesis system 100 and the scoring information output from the real-time scoring module 314 and the best-presenting video stored in the best-presenting video database corresponding to the video synthesis information. To compose a broadcast video.

That is, the video control module 312 is a player screen for viewing the Internet broadcast displayed on the display means of the client terminal 120 (that is, the Internet broadcast screen to be played in one window), the best presentation video and multi-layer composite video and The broadcast screen is reconfigured to simultaneously display the synthesis system scoring information.

In addition, the Internet broadcasting system using a real-time multi-layer video synthesis system according to the present invention, the viewer scoring information input through the input device of the client terminal 120 in real time to the web server 104, the web server 104 May be configured to receive the scoring information transmitted from the client terminal 120, collect and process the scoring information data in real time, and include the same in the broadcasting video configuration.

The client terminal 120 receives viewer scoring information input by the user in real time while watching the Internet broadcast, and transmits the score to the web server 104 through the network 102.

The web server 104 receives the viewer scoring information, processes the synthesis system scoring information and the viewer scoring information according to a predetermined method, reconstructs the broadcasting video reflecting the viewer scoring information, and transmits the same to the client terminal 120. do.

In this case, the real-time scoring module 314 calculates and outputs the sum of the composite system scoring information transmitted from the real-time multi-layer video synthesis system 100 and the viewer scoring information transmitted from the client terminal 120 according to a predetermined algorithm. The video control module 312 receiving the sum score uses the sum score to construct a broadcast video.

In addition, the web server 104 according to the present invention may further include the highest scorer video database 110 is structured and stored in the video of the highest scorer according to each exemplary presentation video.

The highest scorer video database 110 is a database of structured and stored user videos that record the highest scores (composition system scoring information, viewer scoring information, or the sum of the total scores, etc.) according to each exemplary production video. Say.

In this case, the video control module 312 according to the present invention is configured to configure the broadcast video so that the highest scorer video according to the multi-layer composite video, best presentation video, scoring information and the best presentation video on one screen when configuring the broadcast video do.

In addition, when the transmission of the video is completed, the real-time scoring module 314 calculates a final score by processing the synthesis system scoring information transmitted from the real-time multilayer video synthesis system 100 and the viewer scoring information transmitted from the client terminal 120. When the final score of the composite video transmitted in real time is greater than or equal to the final score of the highest scorer, the final scorer video database 110 is updated. .

FIG. 4 is a flowchart illustrating a process in which a video is captured in real time and synthesized in a real time multilayer video synthesizing system according to an exemplary embodiment of the present invention, and FIG. 5 is a real time Internet performed in a web server according to an exemplary embodiment of the present invention. A flowchart illustrating a method of providing a broadcast.

Hereinafter, a method for providing Internet broadcasting using a real-time multilayer video synthesis system according to an exemplary embodiment of the present invention will be described in detail with reference to FIGS. 4 and 5.

FIG. 4 is a flowchart illustrating a process in which a video is captured and synthesized in real time in a real-time multilayer video synthesizing system according to an exemplary embodiment of the present invention.

When the user outputs a control signal that the user wants to perform internet broadcasting using the real-time multilayer video synthesizing system using the key input device 260, the real-time multilayer video synthesizing apparatus 210 receives the signal and controls the project control unit 230. Real-time multi-layer video synthesis process is performed.

The project controller 230 displays a video composition menu on the display device 250 in which a user can select various video composition options.

The user views various selection menus displayed on the display device 250, and manipulates the key input device 260 to create video composition information for his / her own desired video-background video designation, use of various special effects, and sound. Whether the visualization image is used, whether the second background image is used, and the like, is output, and the project controller 230 receives and stores the video composite information output from the key input device 260 (S400).

When receiving a control signal to start multi-layer video synthesis, the project control unit 230 extracts the best-presentation video according to the video synthesis information from the best-presentation video database () to display through the display device 250, and at the same time video input By using the device 256 and the audio input device 252, the user's training images, training contents, dances, accompaniment, advertising motions, etc. are captured and input in real time, and stored in accordance with the video composite information stored in real time It synthesize | combines (S402).

In more detail, the process of generating a multi-layer composite video in real time, the sound processor 224 converts analog sound data input in real time through the audio input device 252 into digital sound data and outputs the digital sound. Analyze the data and output sound information.

At the same time, the image processor 222 converts analog image data input in real time through the video input device 256 into digital image data, and generates a background separated image using color information of the stored background screen 258.

In addition, the image processor 222 extracts a video to be used as a background image according to the video synthesis information from the background video database 240 to process the alpha channel (determining the existence of the alpha channel, when there is no alpha channel) Alpha channel is added, and the alpha value (transparency) of the alpha channel is adjusted according to the sound information output from the sound processor 224. To print.

In addition, the image processing unit 222 at the same time according to the video synthesis information, if the user chooses to create and synthesize the front background image using the existing background video, the front background image in the same process as the process of the back background image When the user selects to use the acoustic visualization image as the front background image, the user generates and outputs the acoustic visualization image according to the sound information output from the sound processor 224, and outputs the audio visualization image by alpha channel processing. .

The image synthesizing module 274 synthesizes the background separation image, the front background image, and the rear background image into a single image and outputs the multi-layer composite image, and the special effect module 276 applies the image special effects according to the settings to the multi-layer composite image. To give output.

At this time, when the camera switching module 278 changes the video input device 256 according to the setting by using the user's location information input from the position sensor, the video switching device 278 changes the video input device 256 according to the video input of the user. The scene change occurs, and accordingly, the scene change module 280 gives the scene change effect to the multi-layer composite image output by the special effect module 276 and outputs the scene change effect.

The file processor 226 synchronizes the digital sound data output from the sound processor 224 and the multi-layer composite image output from the image processor 222, combines them into a single file, and compresses the output into a streaming file according to a specific codec. .

Next, the project control unit 230 extracts a still image that is the basic operation of the user's digital video output from the image processing unit 222, and compared with the basic still image of the model of the best rendering video stored in the model database () A score (synthesis system scoring information) of the user is calculated (S404).

At this time, the more accurate score can be calculated by reducing or enlarging the still image according to the user's physical condition.

The project controller 230 transmits the stored video composition information and the streaming file and the synthesis system scoring information output from the file processor 226 to the web server 104 through the network 102 using the communication controller 228. In operation S406, when the termination signal is not received by determining whether the termination signal is received, steps S402 to S406 are repeatedly performed (S408).

5 is a flowchart illustrating a method of providing real-time Internet broadcasting in a web server according to an exemplary embodiment of the present invention.

The web server 104 receives the streaming file data, the synthesis system scoring information, and the video synthesis information transmitted from the real-time multilayer video synthesis system 100 (S500).

The broadcast control unit 310 of the web server 104 extracts the best-presentation video from the best-presentation video database () according to the video synthesis information received in step S500, and extracts the best-execution video and the synthesis system scoring information and the streaming file. By using the Internet to configure a video for broadcasting, and transmits the configured Internet broadcasting video to the plurality of client terminals 120 connected to the web server 104 via the network 102 (S502).

In this case, the Internet broadcast video refers to a screen that appears on one screen of a media player that can watch a broadcast screen transmitted from the web server 104 in the client terminal 120.

That is, the Internet broadcast viewer is configured to view the exemplary presentation video, the synthesized user video, and the synthesis system scoring information in one window (one screen played in the media player) through the display unit of the client terminal 120. will be.

In addition, the broadcast controller 310 extracts the best score video selected from the best scorer video database 110 according to the video composition information received in step S500, the best score video and the synthesis system scoring information and streaming. You can compose a video for internet broadcasting with a file.

In this case, the viewer can watch the modeling video, the user synthesis video, the highest scorer video and the real-time scoring information on a single screen.

In addition, the broadcast control unit 310 of the web server 104 receives the viewer scoring information transmitted from the plurality of client terminals 120, and adds the total score by the synthesis system scoring information received in step S500 in a predetermined manner. The calculated score is reflected in the video for internet broadcasting (S504).

Internet broadcasting providing method using a real-time multi-layer video synthesis system according to the present invention, in addition to the mechanical scoring information transmitted from the video synthesis system, the Internet broadcast viewer to operate the client terminal 120 while watching the user's video, and transmits The viewer grading information may be used to evaluate the user's video.

When a viewer views a video of a user playing in real time through a display device of the client terminal 120 and wishes to give a point, the viewer may obtain viewer scoring information such as a certain icon included in an internet broadcasting media player. The viewer scoring information is transmitted to the web server 104 using various icons, input windows, etc. provided by the server, so that the user video can be evaluated in real time.

The web server 104 determines whether an end signal is transmitted from the real-time multilayer video synthesizing system 100, and if the end signal is not transmitted, repeats steps S500 to S504 (S506).

When the web server 104 receives the end signal, the web server 104 combines the transmitted streaming file into a single video file and stores the user's information and the video composition information in the user video database 108 (S508).

In addition, the broadcast control unit 310 of the web server 104 accumulates the sum score calculated in step S504, calculates the final sum score, and finalizes the video using the same best-presented video stored in the highest score video database 110. Compared to the total score (S510), if the final score of the highest scorer is higher than the final score, the database of the highest scorer video database 110 through the database control unit 320 is updated to the user's video (S512).

Through the process as described above, it is possible to generate a high-quality multi-layer composite image in real time, and to provide it to viewers over the Internet.

As described above, although the present invention has been described by way of limited embodiments and drawings, the present invention is not limited thereto and is intended by those skilled in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope of equivalent claims.

As described above, the present invention provides a system and method for internet broadcasting service through 3D video synthesis, which overcomes the limitations of the conventional video producer, which allows the karaoke user to follow the exemplary behavior of the virtual performer. It is easy to produce, and it is effective to produce high quality video with various background images and special effects at low cost.

In addition, the recorded video can be broadcast or broadcast live via the Internet, in this case, unlike the existing broadcast has the advantage of less network overload problems.

In addition, there is an advantage that can be used for audition by shooting a personal PR video or music video.

Claims (21)

In the Internet broadcasting system via a network, Real-time multi-layered video that generates real-time multi-layered composite video by synthesizing real-time input image data, sound data, and background video, and converting the multi-layered composite video into a streaming file according to a predetermined codec in real time. Synthesis system; A web server receiving the streaming file transmitted from the real-time multilayer video synthesizing system and transmitting the streaming file in real time through the network; And And a plurality of client terminals connected to the web server through the network to receive the streaming file. The real-time multilayer video synthesizing system includes a modeling video database and a background image database in which various modeling videos are structured and stored, and extracts a specific modeling video that the user should act according to the user's manipulation from the modeling video database. Provides to the user and generates a background separation image by separating the background from the user's image data input in real time, extracts a specific background video from the background video database, and uses the alpha value of the alpha channel of the background video to Generating a front background image and synthesizing the background separated image, the back background image and the front background image in real time; The method of claim 1, wherein the real-time video system, An audio input device for converting an acoustic signal into an electrical signal and outputting the electrical signal; At least one video input device for converting and outputting an optical signal into an electrical signal; A key input device for outputting a control signal according to a key operation; A background screen for generating a background separation video; A background video database in which a pre-recorded background video is structured and stored according to various stories; A modeling video database in which various modeling videos are structured and stored; A display device displaying a best-presentation video stored in the best-presentation video database; And The display device is controlled according to a control signal output from the key input device to display a best-presented moving picture, and the sound data input through the audio input device, the image data input through the video input device, and the storage means. A real-time multi-layer video synthesizing apparatus for synthesizing the stored background video in real time to give a special effect, converting the synthesized multi-layer video into a streaming file according to a predetermined codec, and transmitting the multi-layer video to the web server through the network; Internet broadcasting system using a real-time multi-layer video synthesis system, characterized in that. The apparatus of claim 2, wherein the real-time multilayer video synthesizing apparatus is provided. An input / output interface forming a data input / output interface with an external input / output device; RAM used as a temporary storage place for moving image data and data necessary for image processing; ROM, which stores operating system, image processing algorithm and sound processing algorithm; Storage means for storing various data and physically storing the background video database and the exemplary presentation database; An image processor for synthesizing the image data input through the video input device and the background video stored in the storage unit in real time and outputting a multi-layer composite image; A sound processor for digitizing the sound data input through the audio input device and outputting the digital sound data; A file processing unit for synchronizing the multi-layer composite image output from the image processing unit and the digital audio data output from the sound processing unit to combine into a single file and to compress it into a streaming file according to a predetermined codec; A communication control unit transmitting the streaming file output from the file processing unit to the web server connected through the network; And Real-time multi-layer characterized in that it comprises a main control unit for extracting the best-presented video corresponding to the control signal input through the key input device to display through the display means, and overall control of the synthesis and transmission process of the real-time multi-layer video Internet broadcasting system using video synthesis system. The method of claim 3, The sound processor may analyze sound data output from the audio input device and output sound information including frequency information and output information of the sound data. And the image processor generates and processes a background video according to the sound information output from the sound processor, and outputs a multi-layer composite image. The method of claim 4, wherein the image processor, An input video processing module for converting analog image data input from the video input device into digital image data, and outputting a background separated image by separating a background; Extracts a background video corresponding to the control signal output from the key input device from the storage means, determines whether an alpha channel exists in the background video, and adds an alpha channel when the alpha channel does not exist, and performs the sound processing. A background video processing module for outputting a back background image and a front background image by changing an alpha value of an alpha channel according to sound information output from the means; An image synthesizing module for synthesizing a background separation image output from the input video processing module and a rear background image and a front background image output from the background video processing module in real time to output a multi-layer composite image; And Internet broadcasting system using a real-time multi-layer video synthesizing system comprising a special effect module for giving a variety of special effects according to the user's settings to the multi-layer composite video output from the image synthesizing module. The method of claim 5, The real-time multilayer video synthesis system, Further comprising a position sensor for measuring and outputting the position of the user, The image processor, A camera switching module configured to receive a video signal by selecting a most suitable video input device among the plurality of video input devices according to a user's position output from the position sensor; And And a scene change module configured to generate a multi-layer composite image by applying a scene change effect according to the change of the video input device by the camera switching module. According to claim 3, The real-time video synthesizing apparatus, Providing a video composition menu in the form of a GUI (Graphic User Interface) through the first display device, receiving a control signal input through the key input device to store the video synthesis information corresponding to the control signal, the video And a project control unit for controlling the control unit to generate the multi-layer composite video according to the synthesis information. The method of claim 7, wherein the project control unit, And a project control module configured to receive a control signal input through the key input device, store video composite information corresponding to the control signal, and control the controller to generate a multi-layer composite video according to the video composite information. Real-time multi-layer video synthesis, characterized in that it further comprises a motion analysis module for assigning a score by determining whether the video inputted through the video input device and the best-presented video displayed through the first display unit by a predetermined algorithm; Internet broadcasting system using the system. The method of claim 8, wherein the motion analysis module, And comparing frame data of the model video with the frame data of the user's video input through the video input device to determine whether or not the video is matched. The method of claim 1, The client terminal transmits the viewer scoring information input through the input device to the web server in real time, And the web server receives the scoring information transmitted from the client terminal, collects and processes the scoring information data in real time and includes the processed information in the broadcasting video data. 11. The method of claim 10, wherein the web server. A modeling video database in which various modeling videos are structured and stored; A user automatic image database structured and stored in a multi-layer composite video transmitted from the real-time multi-layer video composite system; A storage means for physically storing the model video clip database and the automatic video image database, and storing various data; RAM used as a temporary storage place for moving image data and data necessary for image processing; A ROM storing an operating system and various programs for running the web server; A communication controller for performing data communication with the real-time multilayer video synthesizing system and the client terminal through a network; The streaming file and the video synthesis information and the synthesis system scoring information transmitted from the real-time multi-layer video synthesis system through the communication control unit in real time, and receives the viewer scoring information transmitted in real time from the client terminal, the streaming file And a broadcast controller configured to construct a broadcast video using the information and transmit the same through the communication controller. A database controller which combines the multi-layer composite video data transmitted from the real-time multi-layer video synthesis system with user information to construct and manage the user automatic video database, and to control the exemplary production video database; And Internet broadcasting system using a real-time multi-layer video synthesis system comprising a main control unit for controlling the overall Internet broadcasting in accordance with the operating system stored in the ROM. The method of claim 11, wherein the broadcast control unit, A real time scoring module for outputting scoring information by summing in real time the synthesis system scoring information transmitted from the real-time multilayer video synthesizing system and the viewer scoring information transmitted from the client terminal in a predetermined manner; And A video control module for constructing a video for broadcasting by using a streaming video transmitted from the real-time multi-layer video synthesizing system and a best-playing video stored in the best-presenting video database corresponding to video synthesis information and scoring information output from the real-time scoring module; Internet broadcasting system using a real-time multi-layer video synthesis system comprising a. The method of claim 12, The web server further includes a top scorer video database of structured and stored videos of the top scorers according to each model video, The video control module is an Internet broadcasting system using a real-time multi-layer video synthesis system, characterized in that for configuring the broadcast video including the highest scorer video stored in the highest scorer video database. The method of claim 13, When the transmission of the video is completed, the real-time scoring module calculates a final score by processing the synthesis system scoring information transmitted from the real-time multi-layer video synthesis system and the viewer scoring information transmitted from the client terminal. And comparing the final score of the highest scorer to the final score of the highest scorer, wherein the highest scorer video database is updated. Real-time multi-layer video synthesis system with audio / video input device, display device, key input device, background screen, background video database and best-play video database, web server with best-play video database and multiple client terminals In the real-time Internet broadcast providing method using an internet broadcasting system connected through (a) synthesizing the video inputted through the audio / video input device and the background video in the real-time multi-layer video synthesis system to generate a real-time multi-layer video, converting the real-time multi-layer video to a streaming file, the streaming file and the video Transmitting composite information to the web server in real time; And (b) receiving the streaming file and the video composite information transmitted in the step (a) from the web server to construct a video for Internet broadcasting, and transmitting the data in real time, In step (a), (a1) displaying a moving picture composition menu for collecting moving picture synthesis information on the display device; (a2) receiving video synthesis information such as exemplary directing video selection information, background video selection information, sound visualization, and special effect selection information output from the key input device; (a3) extracting a corresponding exemplary production video from the exemplary production video database according to the exemplary production video selection information received in step (a2), and displaying the same production video on the display device; (a4) A multi-layer composite image is generated by synthesizing a video input from the video input device in real time and a background video corresponding to the video composite information according to the video composite information received in step (a2). And converting the acoustic data input in real time from the audio input device into digital acoustic data, combining the multilayered composite image and the digital acoustic data into one file, generating a multilayered composite video, and streaming the multilayered composite video. Converting to a file; (a5) comparing the input video and the best presentation video according to a predetermined algorithm to score and display the same on the display device; and (a6) transmitting the video synthesis information, the streaming file, and the synthesis system scoring information in real time. delete The method of claim 15, wherein step (a4), (a4.1) converting analog image data and analog sound data input in real time into digital image data and digital sound data; (a4.2) generating sound information by analyzing the digital sound data converted in the step (a4.1); (a4.3) generating a background separated image by processing the digital image data converted in the step (a4.1); (a4.4) Extracting the background video or generating the background video according to the video synthesis information received in step (a2) and the sound information generated in step (a4.2) and alpha channel processing, Outputting a front background image; (a4.5) generating a multi-layer composite image by synthesizing the background separation image generated in step (a4.3) and the back background image and front background image generated in step (a4.4); (a4.6) applying a special effect to the multi-layer composite image generated in step (a4.5); (a4.7) combining the digital sound data converted in the step (a4.1) and the multi-layer composite image generated in the step (a4.6); And (a4.8) Internet broadcasting providing method using a real-time multi-layer video synthesis system comprising the step of converting the file combined in the step (a4.7) in real time to a streaming file. The method of claim 15, wherein the step (a5) comprises comparing the frame data of the exemplary motion picture displayed on the display device with the frame data of the user video input through the video input device to determine whether or not a match and scoring. Internet broadcasting providing method using a real-time multi-layer video synthesis system. The method of claim 15, wherein step (b) comprises: (b1) receiving in real time a streaming file, video synthesis information, and synthesis system scoring information transmitted from the real-time multilayer video synthesis system; (b2) extracting a model video corresponding to the video composite information received in step (b1) from the model video video database; And (b3) real-time multi-layer video synthesis, comprising the step of constructing a broadcast video using the exemplary presentation video extracted in step (b2) and the streaming file and synthesis system scoring information received in step (b1). Internet broadcasting method using the system. The method of claim 19, The web server further includes a high scorer video database, In the step (b3), the broadcast video using the best rendering video extracted in the step (b2), the streaming file and the synthesis system scoring information received in the step (b1) and the highest scorer video corresponding to the video synthesis information Internet broadcast providing method using a real-time multi-layer video synthesis system, characterized in that the configuration. The method of claim 20, wherein after step (b3), (b4) receiving viewer scoring information transmitted from the client terminal and calculating scoring information in real time according to the system scoring information and a predetermined method; (b5) constructing a broadcast video by reflecting the scoring information on the broadcast video; And (b6) determining whether to receive a shooting end signal transmitted from the real-time multi-layer video synthesis system, and when shooting is finished, using the scoring information calculated in step (b4) to determine the highest score, If it is the Internet broadcast providing method using a real-time multi-layer video synthesis system characterized in that it further comprises the step of building the highest score video database using the streaming file.

Images