KR20170006591A - Method of combining broadcasting image and object image captured by camera, set-top box and system - Google Patents

Abstract

An image synthesizing method includes: a step of receiving an image of an object by a camera of a first external device from the first external device; a step of transmitting the image of the object to a server; a step of receiving a matted image, which is obtained by removing a background of the image of the object, from the cloud server; a step of processing a preset area of the matted image to be translucent; and a step of combining the broadcasting image and the matted image.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method and apparatus for synthesizing an image of an object received from a broadcast image and a camera, a set-top box and a system,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a broadcast image and a method for compositing images of objects received from a camera, a set-top box and a system.

Video synthesis technology is an essential element for various contents production. For example, video composition techniques may be applied to apply special effects to content.

In order to synthesize images, it is required to extract objects in the image. The algorithm for extracting objects in the image is called a matting algorithm. Matting algorithms generally require a tri-map (Foreground, Background, Unknown Index) as an input value.

The image synthesis technology used in the conventional broadcast content production uses a chroma key, and a blue screen is installed in the background to extract the object in real time, synthesize the extracted object, and transmit the broadcast.

However, in an environment where a specific color scream can not be installed in the background, such as a chroma key, a matting technique using only image color information is required.

In addition, a simple matting algorithm is often used in a case where images are synthesized and broadcasted in real time. This simple matting algorithm has a problem of generating a large number of errors at the boundary of an object.

In this regard, Korean Patent Laid-Open Publication No. 2011-0123907 relates to an IPTV system and a content providing method thereof, in which a set-top box drives a camera to take an image, determines whether the captured image includes predetermined recognition information Discloses a technique of transmitting a unique code corresponding to recognition information to a content server, receiving content associated with an image photographed from a content server, synthesizing the content and the photographed image, and displaying the combined content on a display device.

We propose a method, a set-top box and a system for correcting in-body errors by using a skeleton image in a matting algorithm.

A mapped image is input to a guided filter to which a guidance image is applied to a binary segmentation result to derive the intensity of segmentation for each pixel of the mapped image and to calculate the intensity of the segmentation derived for each pixel A set-top box, and a system for correcting a boundary error by allocating a blending coefficient to each pixel based on a set of threshold values.

It is possible to change the media consumption from the provider center to the user center by executing the real time matting algorithm using the cloud server and synthesizing the broadcasting image in real time from the set top box.

According to an aspect of the present invention, there is provided an image processing method including receiving an image of an object photographed by a camera of a first external device from the first external device, Receiving a matted image in which the background of the image of the object is removed from the cloud server, translating a predetermined area of the mapped image into a semi-transparent state, And a step of synthesizing the image.

According to another embodiment of the present invention, there is provided an image processing apparatus including an object image receiving unit for receiving an image of an object photographed by a camera of a first external device from the first external device, an object image transmitting unit for transmitting the image of the object to a cloud server, A matched image receiving unit for receiving a background image of the object from the cloud server, a matched image receiving unit for semi-transparently processing a predetermined area of the mapped image, And a video synthesizer for synthesizing the set-top box.

Further, another embodiment of the present invention is an image synthesizing system including a set-top box and a cloud server, the set-top box receiving an image of an object photographed by a camera of a first external device from the first external device, And transmitting the image of the object to the cloud server, translucently processing a predetermined area of the matted image from which the background of the image of the object received from the cloud server is removed, The cloud server may generate a mapped image by removing the background of the image of the object received from the set-top box, and transmit the mapped image to the set-top box. have.

By using a skeleton image in a matting algorithm, a method, a set-top box and a system for correcting in-body errors can be provided.

A mapped image is input to a guided filter to which a guidance image is applied to a binary segmentation result to derive the intensity of segmentation for each pixel of the mapped image and to calculate the intensity of the segmentation derived for each pixel A set-top box and a system for correcting a boundary error by assigning a blending coefficient to each pixel on the basis of a plurality of pixels.

It is possible to change the media consumption from the provider center to the user center by executing the real time matting algorithm using the cloud server and synthesizing the broadcasting image in real time from the set top box.

1 is a configuration diagram of a video synthesis system according to an embodiment of the present invention.
2 is a block diagram of a set-top box according to an embodiment of the present invention.
3 is a diagram for explaining correction of an in-body error using an auxiliary image received from a second external device according to an embodiment of the present invention.
4 is a diagram for explaining correction of a boundary error using a guided filter according to an embodiment of the present invention.
5 is a view illustrating a composite image in which a boundary error is corrected according to an embodiment of the present invention.
FIG. 6 is a view for explaining the synthesis of an image of an object photographed by a camera of a first external device and a broadcast image according to an embodiment of the present invention.
7 is a flowchart illustrating a method of composing an image of a broadcast image and an object received from a camera according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "including" an element, it is to be understood that the element may include other elements as well as other elements, And does not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof.

In this specification, the term " part " includes a unit realized by hardware, a unit realized by software, and a unit realized by using both. Further, one unit may be implemented using two or more hardware, or two or more units may be implemented by one hardware.

In this specification, some of the operations or functions described as being performed by the terminal or the device may be performed in the server connected to the terminal or the device instead. Similarly, some of the operations or functions described as being performed by the server may also be performed on a terminal or device connected to the server.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of a video synthesis system according to an embodiment of the present invention. 1, the image synthesis system 1 may include a set top box 100, a cloud server 110, a first external device 120, and a second external device 130.

The set top box 100 may receive an image of an object photographed by the camera of the first external device 120 from the first external device 120. [ 1, the first external device 120 may be a camera provided in the set-top box 100 itself. Thus, the set-top box 100 may directly acquire an image of an object through the camera of the set-top box 100. [

In addition, the set-top box 100 may receive an auxiliary image for an image of an object from the second external device 130. [ The secondary image may be a skeleton image for at least one of the still frames of the object's image.

The set-top box 100 may transmit the image of the object and the auxiliary image to the cloud server 110.

The set top box 100 may receive a matted image from which the background of the object image is removed from the cloud server 110. The mapped image received from the cloud server 110 may be an image in which the in-body error is corrected based on the auxiliary image.

Inbadic error refers to the distortion of a part of the inbuilt area in the process of acquiring only the foreground by removing the background through the matching algorithm.

The set-top box 100 can semi-transparently process predetermined areas of the mapped image. The set top box 100 can correct the boundary error by semi-transparently processing the boundary area of the mapped image.

For example, the set-top box 100 inputs the mapped image into a guided filter to which a guidance image is applied to a binary segmentation result, thereby calculating the intensity of the segmentation for each pixel of the mapped image , And a blending coefficient may be assigned to each pixel based on the intensity of the segmentation derived for each pixel.

The set-top box 100 may correct the boundary error based on the blending coefficient. The blending coefficients may be assigned to each pixel in inverse proportion to the intensity of the segmentation. Specific details related thereto will be described later with reference to FIG.

The set-top box 100 may combine the broadcast image and the mapped image. The set-top box 100 may synthesize a mapped image that has been semi-transparent processed in a preset area into a broadcast image.

The cloud server 110 may generate a mapped image by removing the background image of the object received from the set top box 100. [ In generating the mapped image using the matting algorithm, the cloud server 110 may generate a mapped image in which the in-body error is corrected based on the auxiliary image.

For example, the cloud server 110 may perform a skeleton dilation based on the secondary image and extract an approximate object region through skeleton dilation in performing the matching algorithm . In addition, the cloud server 110 may synthesize the mat-map generated based on the approximate object region and the object image to generate mapped image corrected with the in-bard error.

The skeleton dilation may be to expand the skeleton image by a predetermined range.

The cloud server 110 may transmit the mapped image to the settop box 100. [

The first external device 120 may transmit the image of the object photographed through the camera of the first external device 120 to the settop box 100.

An example of the first external device 120 is a wireless communication device that is guaranteed to be portable and mobility. The first external device 120 may be a Personal Communication System (PCS), a Global System for Mobile communications (GSM), a Personal Digital Cellular (PDC) , PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, CDMA (Code Division Multiple Access) -2000, W-CDMA (W-CDMA), Wibro (Wireless Broadband Internet) smart phones, smartpads, tablet PCs, and the like.

As described above, the first external device 120 may be a camera provided in the set-top box 100 itself.

The second external device 120 may transmit an auxiliary image for the image of the object to the set top box 100. The secondary image may be a skeleton image for at least one of the still frames of the object's image.

For example, the second external device 120 may select at least one of a plurality of still frames of the image of the object, generate a skeleton image for the selected still frame, and transmit the skeleton image to the set top box 100. The skeleton image may be used by the cloud server 110 to correct the inborn error in generating a mapped image with the background of the image of the object removed.

An example of the second external device 120 is a wireless communication device that ensures portability and mobility. The second external device 120 may be a personal communication system (PCS), a global system for mobile communications (GSM), a personal digital cellular (PDC) , PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, CDMA (Code Division Multiple Access) -2000, W-CDMA (W-CDMA), Wibro (Wireless Broadband Internet) smart phones, smartpads, smartpads, tablet PCs, smart watches, and the like.

2 is a block diagram of a set-top box according to an embodiment of the present invention. 2, the set-top box 100 includes an object image receiving unit 200, an object image transmitting unit 210, an auxiliary image receiving unit 220, a matting image receiving unit 230, an image correcting unit 240, A blending coefficient assigning unit 260, and an image combining unit 270. [

The object image receiving unit 200 can receive an image of an object photographed by a camera of the first external device 120 from the first external device 120. [

The object image transmitting unit 210 may transmit the image of the object to the cloud server 110. [

The auxiliary image receiving unit 220 may receive the auxiliary image for the image of the object from the second external device 130. [ The secondary image may be a skeleton image for at least one of the still frames of the object's image.

The object image transmitting unit 210 may transmit the auxiliary image to the cloud server 110.

The matting image receiving unit 230 may receive a matted image from which the background of the object image is removed from the cloud server 110. [ The mapped image received from the cloud server 110 may be an image in which the in-body error is corrected based on the auxiliary image.

Inbadic error refers to the distortion of a part of the inbuilt area in the process of acquiring only the foreground by removing the background through the matching algorithm.

The image correcting unit 240 may semi-transparently process a predetermined area of the mapped image. The set top box 100 can correct the boundary error by semi-transparently processing the boundary area of the mapped image.

The image correcting unit 240 may include a guide filter 250 and a blending coefficient assigning unit 260.

The guide filter 250 may be one obtained by applying a guidance image to a binary segmentation result.

The blending coefficient assigning unit 260 may assign a blending coefficient to each pixel based on the intensity of the segmentation derived for each pixel of the mapped image.

For example, the image correcting unit 240 may input the mapped image into a guided filter to derive the intensity of the segmentation for each pixel of the mapped image, and assign a blending coefficient to each pixel have. The blending coefficients may be assigned to each pixel in inverse proportion to the intensity of the segmentation.

The image correction unit 240 may correct the boundary error based on the blending coefficient. For example, in the case of a pixel having a blending coefficient of a predetermined value or less, the image correcting unit 240 does not perform the boundary error correction for the pixel, and, for a pixel whose blending coefficient is equal to or larger than a predetermined value, The coefficient can be set, and the semi-transparent processing can be performed accordingly.

Accordingly, the boundary error correction process is not performed inside the object, and the error correction process can be performed outside the object. In particular, the more pixels outside the object, the closer it is to the background, the more translucent it can be.

Therefore, it is possible to correct the boundary error so that the image of the object in the synthesized image is not unnatural.

The image combining unit 270 may combine the broadcast image and the mapped image. The set-top box 100 may synthesize a mapped image that has been semi-transparent processed in a preset area into a broadcast image.

3 is a diagram for explaining correction of an in-body error using an auxiliary image received from a second external device according to an embodiment of the present invention.

The second external device 130 may transmit an auxiliary image for the image of the object to the set top box 100. The secondary image may be a skeleton image for at least one of the still frames of the object's image. The second external device 130 may be a smart watch interlocked with the first external device 120.

3, the second external device 130 receives at least one still frame of an image of an object or an object of the object from the first external device 120, and generates a skeleton image 300 Can be generated.

The skeleton image 300 for the still frame can be generated by the second external device 130 outputting at least one still frame and receiving the skeleton image 300 for the still frame from the user.

In contrast, the second external device 130 may directly generate the skeleton image 300 for at least one still frame.

When the set-top box 100 transmits the skeleton image 300 received from the second external device 130, the cloud server 110 performs a matting algorithm, thereby generating a skeleton delay based on the secondary image, , And extract the approximate object region 320 through the skeleton delays. In addition, the cloud server 110 may generate a mapped image in which the inborn error 330 is corrected by combining the approximate object region 320 and the matmap 320 generated based on the object image.

The skeleton dilation 310 may be to expand the skeleton image by a predetermined range.

4 is a diagram for explaining correction of a boundary error using a guided filter according to an embodiment of the present invention.

The set-top box 100 can semi-transparently process predetermined areas of the mapped image. The set top box 100 can correct the boundary error by semi-transparently processing the boundary area of the mapped image.

Referring to FIG. 4, the set-top box 100 can derive the segmentation intensity using a guide filter in which a guidance image is applied to a binary segmentation result of the mapped image 400. FIG.

In addition, the set-top box 100 can assign a blending coefficient to each pixel based on the intensity of the segmentation of each pixel. The blending coefficients may be assigned to each pixel in inverse proportion to the intensity of the segmentation.

For example, in the image 410 output through the guide filter shown in FIG. 4, the interior 420 is assigned a low blending coefficient because the intensity of the segmentation is high, and the exterior 320 is the intensity of the segmentation Is derived, a high blending coefficient can be assigned.

The intensity of the segmentation may correspond to a value between 0 and 1, and may have a different value for each pixel. Therefore, the blending coefficient of each pixel may also be different for each pixel.

The set-top box 100 can correct the boundary error based on the blending coefficient assigned to each pixel. For example, in the case of a pixel whose blending coefficient is equal to or less than a preset value, the set-top box 100 does not perform boundary error correction for the pixel, and for a pixel whose blending coefficient is equal to or larger than a predetermined value, And the translucent process can be performed accordingly.

Accordingly, the boundary error correction process is not performed on the inner part 420 of the object, and the error correction process can be performed on the outer part 430 of the object. In particular, in the outer portion 430 of the object, pixels closer to the background can be processed more semi-transparently.

Therefore, it is possible to correct the boundary error so that the image of the object in the synthesized image is not unnatural.

5 is a view illustrating a composite image in which a boundary error is corrected according to an embodiment of the present invention.

Referring to FIG. 5, reference numeral 500 denotes an image of an object synthesized in a broadcast image. In general, the user is familiar with blurring phenomenon when fast motion exists, such as reference numeral 510, when viewing an image. Accordingly, the set-top box 100 corrects the boundary error as indicated by reference numeral 520, and a natural synthesis effect can be obtained.

FIG. 6 is a view for explaining the synthesis of an image of an object photographed by a camera of a first external device and a broadcast image according to an embodiment of the present invention.

6, when the user photographs his or her own image 600 using the camera of the first external device 120 during the broadcast viewing, the set-top box 100 transmits the image of the object to the cloud server 110 And combines the mapped image and the broadcast image received from the cloud server 120.

Thus, a video 620 of the mapped object may be composited on the broadcast screen 610. Since this process is performed in real time, all the operations of the user can be output together with the broadcast image.

7 is a flowchart illustrating a method of composing an image of a broadcast image and an object received from a camera according to an embodiment of the present invention. The method of synthesizing the broadcast image according to the embodiment shown in FIG. 7 and the image of the object received from the camera includes the steps of time series processing in the system shown in FIG. Accordingly, the present invention is also applied to a method of synthesizing an image of an object received from a camera and a broadcast image performed according to an embodiment shown in FIG.

Referring to FIG. 7, in step S700, the set-top box 100 can receive an image of an object photographed by a camera of the first external device 120 from the first external device 120. FIG.

In step S710, the set-top box 100 may transmit an image of the object to the cloud server 110. [

In step S720, the set-top box 100 may receive a matted image from which the background of the image of the object is removed from the cloud server 110. [

In step S730, the set top box 100 may semi-transparently process a predetermined area of the mapped image.

In step S740, the set-top box 100 may combine the broadcast image and the mapped image.

The method of synthesizing the broadcast image and the image of the object received from the camera illustrated in FIG. 7 may be implemented in the form of a computer program stored in a medium or may include a command executable by a computer such as a program module executed by a computer But may also be embodied in the form of a recording medium. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium can include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: Set-top box
110: Cloud server
120: first external device
130: second external device

Claims (17)

A method for compositing an image of a broadcast image and an object received from a camera,
Receiving an image of an object photographed by a camera of a first external device from the first external device;
Transmitting an image of the object to a cloud server;
Receiving a matted image from which the background of the image of the object is removed from the cloud server;
Translucently processing a predetermined area of the mapped image; And
Synthesizing the broadcast image and the mapped image
And the image synthesis method.
The method according to claim 1,
Receiving an auxiliary image for an image of the object from a second external device; And
Transmitting the secondary image to the cloud server
Further comprising the steps of:
3. The method of claim 2,
Wherein the auxiliary image is a skeleton image for at least one of the still frames of the image of the object.
3. The method of claim 2,
Wherein the second external device is a smart watch interlocked with the first external device.
3. The method of claim 2,
Wherein the mapped image is corrected for an in-body error based on the auxiliary image.
The method according to claim 1,
The step of semitransparently processing a predetermined area of the mapped image
A step of correcting a boundary error by semi-transparently processing a border area of the mapped image
And the image synthesis method.
The method according to claim 6,
The step of correcting the boundary error may comprise:
Deriving a segmentation intensity for each pixel of the mapped image by inputting the mapped image into a guided filter to which a guidance image is applied to a binary segmentation result;
Assigning a blending coefficient to each pixel based on the intensity of the segmentation derived for each pixel;
And the image synthesis method.
8. The method of claim 7,
Wherein the blending coefficient is allocated for each pixel in inverse proportion to the intensity of the segmentation,
And corrects the boundary error based on the blending coefficient.
A set-top box for synthesizing a broadcast image and an image of an object received from a camera,
An object image receiving unit receiving an image of an object photographed by a camera of a first external device from the first external device;
An object image transmitting unit for transmitting the image of the object to the cloud server;
A matting image receiving unit receiving a matted image from which the background of the image of the object is removed from the cloud server;
An image correcting part for semitransparently processing a predetermined area of the mapped image; And
And an image synthesizing unit for synthesizing the broadcast image and the mapped image,
Lt; / RTI >
10. The method of claim 9,
An auxiliary image receiver for receiving an auxiliary image for an image of the object from a second external device;
Further comprising:
And the object image transmitting unit transmits the auxiliary image to the cloud server.
11. The method of claim 10,
Wherein the secondary image is a skeleton image for at least one of the still frames of the image of the object.
11. The method of claim 10,
And the second external device is a smart watch interlocked with the first external device.
11. The method of claim 10,
Wherein the mapped image is corrected for an in-body error based on the auxiliary image.
10. The method of claim 9,
Wherein the image correction unit corrects a boundary error by semi-transparently processing a boundary area of the mapped image.
15. The method of claim 14,
The image correction unit
A guided filter to which a guidance image is applied to a binary segmentation result; And
A blending coefficient assigning unit for assigning a blending coefficient to each pixel based on intensity of the segmentation derived for each pixel of the mapped image,
Further comprising:
Wherein the image corrector derives the intensity of segmentation for each pixel of the mapped image by inputting the mapped image into the guide filter.
16. The method of claim 15,
Wherein the blending coefficient is allocated for each pixel in inverse proportion to the intensity of the segmentation,
Wherein the image correction unit corrects the boundary error based on the blending coefficient.
A system for synthesizing a broadcast image and an image of an object received from a camera,
Set top box; And
Cloud server
, ≪ / RTI &
The set-
Receiving an image of an object photographed by a camera of the first external device from the first external device,
Transmitting an image of the object to the cloud server,
Translucently processes a predetermined area of a matted image from which the background of the image of the object received from the cloud server is removed,
And to combine the broadcast image and the mapped image,
The cloud server includes:
Generating a mapped image by removing the background of the image of the object received from the set-top box,
And transmit the mapped image to the settop box.

Images