KR20210147231A - Apparatus and method for composing image and broadcasting system having the same - Google Patents

Abstract

Disclosed are a video synthesizing device and method, and a video broadcasting system including the same. The video synthesizing device and method, and the video broadcasting system including the same can perform natural synthesis by providing a video image, in which a content image is synthesized, to video data photographed at 360 degrees. The video synthesizing device and method, and the video broadcasting system including the same can also provide a smooth real-time broadcasting service by synthesizing and broadcasting the video data, which is a background video, after generating synthesized data in which the content image is synthesized. In addition, the present invention can provide a high-efficiency and high-performance video synthesizing device and method, and a video broadcasting system including the same by allowing a user to transform various types of content images using an authoring tool.

Description

Image synthesizing apparatus and method, and video broadcasting system including the same

The present invention relates to an apparatus and method for synthesizing an image, and to a video broadcasting system including the same, and to an apparatus and method for synthesizing a video for providing a video image synthesized with content images in real time, and to a video broadcasting system including the same.

Recently, with the spread of smartphones and the Internet, one-person media broadcasting is attracting attention as a new broadcasting trend, and the video media platform market is expanding worldwide.

Accordingly, in the video media platform market, various technologies are being developed to effectively provide various information to media images provided in real time, and among them, video synthesis technology that provides information by synthesizing content images with real-time video data is paying attention. .

Conventional image synthesis technology is provided in the form of synthesizing a two-dimensional content image with real-time image data. In addition, in the conventional 3D image synthesis technology, the result of 3D synthesis on the entire image data captured in 3D space in real time is rendered by pixel shading for every frame and transmitted. By doing so, there are disadvantages in that real-time transmission is difficult due to a large amount of computation, and expensive system resources are required.

An object of the present invention to solve the above problems is to provide a high-efficiency and high-performance image synthesizing apparatus.

In addition, another object of the present invention to solve the above problems is to provide a method for synthesizing an image with high efficiency and high performance.

Another object of the present invention to solve the above problems is to provide a high-efficiency and high-performance video broadcasting system.

An image synthesis method according to an embodiment of the present invention for achieving the above object includes the steps of receiving data photographed from an image camera in real time, storing the format-converted image data, and using an authoring tool, the image data generating synthesized data in which a 3D content image is inserted in the However, the generating of the synthesized data includes arranging the 3D content image on the image data and transforming the 3D content image using the authoring tool.

The apparatus and method for synthesizing an image according to embodiments of the present invention, and a video broadcasting system including the same may enable natural synthesis by providing a video image in which a content image is synthesized with video data captured at 360 degrees.

In addition, the apparatus and method for synthesizing an image according to the present invention, and a video broadcasting system including the same can provide a smooth real-time broadcasting service by synthesizing and broadcasting the image data, which is a background image, after generating the synthesized data in which the content image is synthesized. In addition, it is possible to provide a high-efficiency and high-performance image synthesizing apparatus and method, and a video broadcasting system including the same, by enabling a user to modify images of various types of content by means of an authoring tool.

1 is a block diagram of a video broadcasting system according to an embodiment of the present invention.
2 is a block diagram of an image synthesizing apparatus according to an embodiment of the present invention.
3 is a flowchart of an image synthesis method according to an embodiment of the present invention.
4 is a composite data image for explaining the function of an authoring tool in an image synthesis method according to an embodiment of the present invention.
5 is an image for explaining an authoring tool provided in the form of a menu bar in an image synthesis method according to an embodiment of the present invention.
6 is an image for explaining an image control tool among the image control tools according to an embodiment of the present invention.
7 is an image for explaining an image control tool in an image synthesis method according to an embodiment of the present invention.
8 is an image for explaining an image control tool in an image synthesis method according to an embodiment of the present invention.
9 is an image for explaining an image control tool in an image synthesizing method according to an embodiment of the present invention.
10 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.
11 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.
12 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.
13 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.
14 is an image for explaining a content editing tool in an image synthesis method according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term “and/or” includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

1 is a block diagram of a video broadcasting system according to an embodiment of the present invention.

Referring to FIG. 1 , the video broadcasting system S may output composite data obtained by synthesizing a content image with video data captured in real time from the video camera C. Referring to FIG.

According to an embodiment, the video broadcasting system S may include an image camera C, an image synthesizing apparatus 1000 , and an image output apparatus 5000 .

In addition, according to another embodiment, the video broadcasting system S is not limited to the above, and the image synthesizing apparatus 1000 and the image camera C combined with the functions of a separate image output apparatus 5000 and capable of image output ) may be included.

Hereinafter, for clearer explanation, features of each configuration of a video broadcasting system according to an embodiment of the present invention will be described.

The image camera C may be linked with an image synthesizing apparatus 1000 and an image output apparatus 5000 to be described later. Accordingly, the image camera C may transmit the captured data to the image synthesis apparatus 1000 and the image output apparatus 5000 , respectively.

According to an embodiment, the video camera C is a 360-degree photographing camera, and may acquire 360-degree photographing data in real time.

The video camera C is generally provided as a single camera, but may be provided as various devices including the video camera C capable of capturing 360 degrees. For example, the video camera C may be provided in the form of at least one of an IP camera, a CCTV, an omnidirectional camera, a portable terminal, and a Pan Tilt Zoom (PTZ).

The image synthesizing apparatus 1000 may receive captured data from the image camera C. Accordingly, the image synthesizing apparatus 1000 may synthesize a content image with the received captured data.

More specifically, the image synthesis apparatus 1000 may provide an authoring tool. Accordingly, the image synthesizing apparatus 1000 may generate synthesized data obtained by synthesizing the content image with the captured data received from the image camera C by using at least one function of the authoring tool.

Also, the image synthesis apparatus 1000 may be linked with the image output apparatus 5000 . Accordingly, the image synthesis apparatus 1000 may transmit the generated synthesized data to the image output apparatus 5000 .

As described above, the image synthesizing apparatus 1000 may also work with the image camera C. Accordingly, the image synthesis apparatus 1000 may control the operation of the image camera C according to a user input.

The image synthesizing apparatus A will be described in detail for each configuration with reference to FIG. 3 below.

2 is a block diagram of an image synthesizing apparatus according to an embodiment of the present invention.

Referring to FIG. 2 , the image synthesizing apparatus 1000 may include a memory 100 for storing at least one command and a processor 200 for executing at least one command in the memory.

In addition, the image synthesizing apparatus 1000 includes a transceiver 300 , an input interface device 400 , an output interface device 500 , and a storage device 600 that are connected to a network executed through the processor 200 to perform communication. ) and the like may be further included.

According to an embodiment, each of the components 100 , 200 , 300 , 400 , 500 , and 600 included in the image synthesizing apparatus 1000 may be connected by a bus 700 to perform communication with each other. .

Among the components 100 , 200 , 300 , 400 , 500 , 600 , and 700 of the image synthesis apparatus 1000 , the memory 100 may be configured as at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory 100 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

The memory 100 may include at least one instruction to be executed by the processor 200 to be described later. According to an embodiment, the at least one command includes a command to receive data photographed from a video camera in real time and store the format-converted video data, and a 3D content image to the video data by using an authoring tool. A command to generate the inserted composite data and a command to transmit the composite data to an image output device so that the video image in which the video data and the composite data are synthesized is broadcast by the video output device, The command to generate data may include a command to arrange the 3D content image on the image data and a command to transform the 3D content image using the authoring tool.

The processor 200 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed.

As described above, the processor 200 may execute at least one program command stored in the memory 100 . Features of the processor 200 for executing at least one command will be described in more detail when an image synthesis method to be described later is described.

The transceiver 300 may transmit data to or receive data from at least one interlocked device.

According to an embodiment, the image synthesizing apparatus 1000 may receive at least one piece of photographing data from the image camera C through the transceiver 300 . Here, the captured data may be a 360-degree video image.

Also, the image synthesis apparatus 1000 may transmit the synthesized data generated by the transceiver 300 to the image output apparatus 5000 .

The input interface device 400 may be configured to receive at least one command input from a user. According to an embodiment, the input interface device 400 may include any one of a mouse, a touch pen, and a keyboard.

The output interface device 500 may output at least one data.

According to an embodiment, the output interface device 500 may be provided as a display device. Accordingly, the image synthesizing apparatus 1000 outputs a synthesized image in which 3D content is synthesized according to a command of the processor 200 by using the output interface apparatus 500 without configuring a separate image output apparatus 5000 . can do.

The storage device 600 may be configured as at least one of a volatile storage medium and a non-volatile storage medium, like the memory 100 described above. For example, the storage device 600 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

According to the embodiment, when the configuration of a separate image output device 5000 is not provided, the storage device 600 pre-stores at least one photographing data received from the image camera C, and then the processor 200 The image data may be generated by synthesizing the synthesized data generated by the command to.

Referring back to FIG. 1 , the image output apparatus 5000 may interwork with the image camera C and the image synthesis apparatus 1000 as described above.

Accordingly, the image output apparatus 5000 may receive photographing data from the image camera C, and may receive synthesized data from the image output apparatus 1000 in real time.

Thereafter, the image output apparatus 5000 may synthesize and output the received photographing data and composite data. Accordingly, the image output device 5000 may broadcast 360-degree image data in which the content image is synthesized.

In the above, a video broadcasting system according to an embodiment of the present invention has been described. Hereinafter, an image synthesizing method by a processor operation of an image synthesizing apparatus in the image broadcasting system will be described.

3 is a flowchart of an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 3 , the processor 200 in the image synthesizing apparatus 1000 may receive at least one piece of photographing data from the image camera ( S1000 ). Here, the photographing camera may be a 360-degree camera, and the photographing data may be a 360-degree video image captured in real time.

The processor 200 may pre-process the received photographing data. In other words, the processor 200 may generate image data by converting the format of the received photographing data (S2000).

More specifically, according to the embodiment, the processor 200 may convert the photographed data into an equirectangular format image by using vertex shaders. In other words, the processor 200 may convert and store the captured data, which is a rectangular image, into sphere-shaped image data.

According to another embodiment of the present invention, when a separate configuration of the image photographing apparatus 5000 is not provided, the processor 200 may store the generated image data in the storage apparatus 600 . Accordingly, the processor 200 may synthesize the image data and the composite data stored in the storage device 600 and output the synthesized data by the output interface device 500 after generation of composite data, which will be described later.

The processor 200 may generate composite data in which a content image is synthesized based on the image data generated by the pre-processing step S2000 ( S3000 ).

More specifically, according to the embodiment, the processor 200 may generate composite data in which a content image is synthesized with image data using an authoring tool.

Here, the authoring tool may include at least one function for synthesizing a content image.

At least one function included in the authoring tool will be described in more detail below.

4 is a composite data image for explaining the function of an authoring tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 4 , the authoring tool may include at least one function for generating synthetic data.

According to an embodiment, the authoring tool may include an image control tool, a content control tool, and a content editing tool. Functions of the image control tool, the content control tool, and the content editing tools will be described in more detail below with reference to the drawings.

When the display device is provided as the output interface device 500 , the authoring tool may be provided in the form of a menu bar (Bar, A) or a pop-up window (B) at the top and bottom of the display screen.

5 is an image for explaining an authoring tool provided in the form of a menu bar in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 5 , when the authoring tool is provided in the form of a menu bar, the authoring tool generally exists in a hidden state. can be displayed.

In the following, the functions of the authoring tool will be described in more detail.

6 is an image for explaining an image control tool among the image control tools according to an embodiment of the present invention.

Referring to FIG. 6 , the image control tool may include a function for controlling execution of image data.

More specifically, as described above, the image synthesizing apparatus 1000 may control the operation of the image camera C by interworking with the image camera C. As illustrated in FIG.

According to an embodiment, when the user executes the first image control tool icon A in FIG. 6 by the input interface device 400 , the processor 200 receives the shooting data from the image camera C and , the pre-processed image data may be executed in the output interface device 500 .

According to another embodiment, when the user executes the second image control tool icon B in FIG. 6 by the input interface device 400, the processor 200 stops interworking with the image camera C, The output of the image data to the output interface device 500 may be stopped. For example, when the second image control tool icon B of FIG. 6 is executed, the output interface device 500 may output a black background screen.

7 is an image for explaining an image control tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 7 , the image control tool may include a function of controlling the rotation operation of the image camera C to change the background viewpoint of image data using the input interface device 400 .

More specifically, the user may control the rotation operation of the image camera C by executing the image control tool by the input interface device 400 . Accordingly, the user may change the background viewpoint of the photographing data received from the image camera C and output the image data of the changed composition through the output interface device 500 .

For example, when the input interface device 400 is a mouse device, the user clicks the right key of the mouse at a point P1 in the image data displayed on the output interface device 500 and clicks the corresponding point ( P1) can be set as the reference point. Thereafter, the user may change the background viewpoint of the image data output to the output interface device 500 by moving the mouse while the right key is clicked. For example, the processor 200 may change the background viewpoint of the image data output to the output interface device 500 by rotating the image camera C according to the movement distance and movement direction of the mouse.

8 and 9 are images for explaining an image control tool in an image synthesizing method according to an embodiment of the present invention. More specifically, FIG. 8 is an image in which a 3D content image is synthesized on image data, and FIG. 9 is an image in which a 2D content image is synthesized on image data.

Referring to FIGS. 8 and 9 , the image control tool may include a function of synthesizing an image with content on image data.

In other words, the image control tool may provide a function to arrange the content image on the image data.

According to the embodiment with reference to FIG. 8 , the user selects at least one 3D content image included in the menu bar by the input interface device 400 and arranges the selected 3D content image at a desired point on the image data. By doing so, it is possible to generate a composite image in which the 3D content image is synthesized with the image data.

According to another embodiment with reference to FIG. 9 , the user selects at least one 2D content image included in the menu bar by the input interface device 400 and arranges the selected 2D content image at a desired point on the image data. By doing so, it is possible to generate a composite image in which a two-dimensional content image is synthesized with image data.

10 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 10 , the content control tool may provide a function of changing the shape of a 3D content image disposed on image data.

According to an embodiment, the content control tool may be provided in the form of a pop-up window on the display screen when a 3D content image is clicked.

The function of the content control tool will be described in more detail with reference to FIGS. 11 to 13 below.

11 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 11 , the content control tool may provide a function of adjusting the size of the 3D content image.

According to an embodiment, the user may adjust the size of the 3D content image disposed on the composite image to a desired size by adjusting the slide bar of the Scale item among the functions of the content control tool provided as a pop-up window.

12 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 12 , the content control tool may provide a function of adjusting the rotation of the 3D content image.

According to an embodiment, the user can change the direction of the 3D content image arranged on the composite image to a desired direction by adjusting the slide bar of the Rotation item among the functions of the content control tool provided as a pop-up window. .

13 is an image for explaining a content control tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 13 , the content control tool may provide a function of adjusting the depth of the 3D content image.

According to an embodiment, the user adjusts the slide bar of the Depth item among the functions of the content editing tool provided as a pop-up window to determine the distance between the 3D content images when two or more 3D content images are provided. can be adjusted

The content control tool of the image synthesizing apparatus according to an embodiment of the present invention is not limited to the content control function disclosed in FIGS. 11 to 13, and may further include a control function such as deleting a 3D content image. have.

14 is an image for explaining a content editing tool in an image synthesis method according to an embodiment of the present invention.

Referring to FIG. 14 , the content editing tool may provide a function of individually editing content images arranged on image data.

For example, when the input interface device 400 is provided with a mouse, the user may right-click the mouse to execute the content editing tool in the form of a pop-up window on the display screen.

According to an embodiment, the content editing tool may provide a search function, a lock function, a show/hide function, a list deletion function, and the like.

More specifically, the finding function (A) may perform a function of finding a corresponding content image on the composite data. In other words, by executing the search function (A), the user may generate composite data focusing on the corresponding content image.

In addition, the lock function (B) may perform a function of preventing the deformation of the corresponding content image on the composite data. In other words, by executing the lock function B, the user may restrict selection or movement of the corresponding content image by the input interface device 400 .

The show/hide function (C) may be a function of determining whether to expose the corresponding content image on the composite data.

According to an embodiment, when the user deactivates the show/hide function (C), the corresponding content image may not be exposed when image data is output from the image output device 5000 , which will be described later.

The list deletion function (D) may perform a function of removing the corresponding content image from the composite data and the list list of the menu bar.

Referring back to FIG. 3 , the processor 200 may transmit the generated composite data to the image output apparatus 5000 ( S4000 ). Accordingly, the image output apparatus 5000 may output the image data obtained by synthesizing the image data transmitted from the image camera C and the synthesized data in real time.

In the above, an apparatus and method for synthesizing an image according to embodiments of the present invention, and a video broadcasting system including the same have been described.

The apparatus and method for synthesizing an image according to embodiments of the present invention, and a video broadcasting system including the same may enable natural synthesis by providing a video image in which a content image is synthesized with video data captured at 360 degrees.

In addition, the apparatus and method for synthesizing an image according to the present invention, and a video broadcasting system including the same can provide a smooth real-time broadcasting service by synthesizing and broadcasting the image data, which is a background image, after generating the synthesized data in which the content image is synthesized. In addition, it is possible to provide a high-efficiency and high-performance image synthesizing apparatus and method, and a video broadcasting system including the same, by enabling a user to modify images of various types of content by means of an authoring tool.

Accordingly, the apparatus and method for synthesizing an image according to an embodiment of the present invention, and a video broadcasting system including the same, can be used in various fields such as broadcasting, content production, games, and AR/VR, as well as in broadcasting production.

The operation of the method according to the embodiments of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. In addition, the computer-readable recording medium may be distributed in a network-connected computer system to store and execute computer-readable programs or codes in a distributed manner.

In addition, the computer-readable recording medium may include a hardware device specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although some aspects of the invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, wherein a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method may also represent a corresponding block or item or a corresponding device feature. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In embodiments, the field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

Although described above with reference to the preferred embodiment of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can.

S: Video broadcasting system C: Video camera
1000: video synthesis device 5000: video output device
100: memory 200: processor
300: transceiver device 400: input interface device
500: output interface device 600: storage device
700: Bus (BUS)

Claims (1)

receiving the captured data from the video camera in real time and storing the format-converted image data;
generating synthetic data in which a 3D content image is inserted into the image data using an authoring tool; and
transmitting the synthesized data to an image output device to broadcast a video image in which the image data and the synthesized data are synthesized by the image output device;
The step of generating the synthetic data comprises:
arranging the 3D content image on the image data; and
and transforming the 3D content image by using the authoring tool.

Images