KR101733125B1 - Method of chroma key image synthesis without background screen - Google Patents

Abstract

According to an embodiment of the present invention, a chroma key image combining method of a chroma key combining device includes: a step of taking a first image which does not include an object in the background and reading and maintaining color information of the first image; a step of taking a second image which includes the object in the background and reading and maintaining color information of the second image; a step of obtaining an object image from which the background is removed from the second image to extract only the object by removing a color value corresponding to the color information of the first image from the second image; a step of taking an infrared image which includes the object in the background and acquiring an object depth image in which the object is tracked in the infrared image based on depth information read from the infrared image; a step of correcting the object image using the object depth image; and a step of combining the corrected object image with a selected background image. Therefore, the object image necessary for chroma key combination can be accurately and easily obtained even in an environment without the background screen.

Description

{METHOD OF CHROMA KEY IMAGE SYNTHESIS WITHOUT BACKGROUND SCREEN}

The present invention relates to a chroma key image synthesis method that does not require a background screen, and more particularly, to a chroma key image synthesis method that extracts an object from a color image and an infrared image in which an object is photographed in a general background, The object contour of the infrared image is corrected through the color image and then synthesized with each other. Thus, even in an environment without a background screen, a chroma key image Lt; / RTI >

In a typical chroma key synthesizer, a specific color (chroma) is designated as a key, and a background screen image is displayed on an object captured through a camera based on a chroma key and a background screen image The background image is replaced with another background image to generate a synthesized image. A chroma key is used for image synthesis. After a specific chroma is designated as a key, a pixel corresponding to a key in the main image is replaced with another image. In this case, in an algorithm for performing a chroma key, a key means a range of a chroma value, not a single chroma value. Therefore, to perform a chroma key, a background screen to be a key is required, and a displacement of the chroma value of the background screen must be small. And the background screen and the objects (all objects that are separated from the screen, including people) must be separated from each other by chroma.

In such a chroma key technique, in order to separate a background screen and an object, it is necessary to plan in advance so that the color of the object does not become the same as the background color. In addition, even if the color of the object is different from the background color, the lighting condition must be planned in advance because it may be the same under certain lighting conditions. In addition, the camera technique must be planned in advance because the object may fall within the same chroma range as the background under a particular camera rendering technique. In addition, objects must be planned in advance because they may fall into the same color range as the background, due to the appearance of other objects, causing reflections, shadows, changes in lighting conditions, and the like.

In addition, in order to plan objects, cameras, lights, etc. in advance, it is necessary to be able to predict the chroma key value of the background before shooting. In addition, since all cameras distort the color and other states of the real thing to a certain degree, the chroma key value of the background screen is also used as a key value for the eye and a chroma key of the video signal coming through the camera Since the chroma key values are different, this difference should be minimized so that you can shoot with a pre-planned color scheme. In addition, since the chroma component of the background screen can be changed by objects, lighting, and a camera rendering technique, a key value to be a reference may be changed in this case. Therefore, it should be planned in advance so that such a phenomenon does not occur .

Thus, the chroma key technology according to the prior art requires a background screen. That is, when an object necessary for chroma key synthesis is photographed, the object is positioned on a background screen having a specific color, and then chroma keying is performed on the captured image to extract only the object, , It is necessary to take a picture on a background screen having a specific chroma. Therefore, in order to perform the conventional chroma key image synthesis, it is inconvenient that the image can be taken only in a specific space such as a room having a background screen. In order to solve the problems of the related art, it is required to develop a technique of performing chroma key image synthesis with flexibility and high accuracy in images taken without a background screen.

Patent Document 10-2003-0079316 (October 10, 2003)

The object of the present invention is to provide a method and an apparatus for correcting an object pixel hole of a color image by using an infrared image and extracting an object from a color image and an infrared image, The object contour of the image is corrected through the color image and then synthesized with each other to provide a chroma key image synthesis method that does not require a background screen that can acquire object images necessary for chroma key synthesis more conveniently and accurately even in an environment without a background screen .

According to another aspect of the present invention, there is provided a method of composing a chroma key of a chroma key combination apparatus, comprising: receiving a first image including no object in a background; Capturing and storing color information of the first image; Capturing a second image including the object on the background and reading and maintaining color information of the second image; Removing a color value corresponding to the color information of the first image from the second image and obtaining the object image from which the background is removed from the second image and extracting only the object; Capturing an infrared image including the object on the background and obtaining an object depth image in which the object is tracked in the infrared image through depth information read from the infrared image; Acquiring a chroma key object image by synthesizing the object image and the object depth image by correcting pixels of the object image and pixels of the object depth image through mutual pixel information; And compositing the chroma key object image with the selected background image.

를 연산하는 단계; 및 상기 r3, 상기 g3, 상기 b3 가 각각 0에 수렴하는 경우 해당 픽셀을 상기 배경으로 판독하고, 선정된 임계값 이상인 경우 해당 픽셀을 상기 객체로 판독하는 단계를 포함한다.Further, in the chroma key synthesis method of the chroma key combination apparatus according to an embodiment of the present invention, the chroma key corresponding to the color information of the first image may be removed from the second image, (YUV) value of each pixel of the first image corresponding to the RGB (YUV) value of each pixel of the second image to the size of the image Width * Height); One of the pixel values of the second image is denoted by r1, g1 and b1, one of the pixel values of the first image is denoted by r2, g2 and b2, and one of the pixel values of the object image is denoted by r3, g3, and b3,

; And reading the corresponding pixel as the background when r3, g3, and b3 respectively converge to zero, and reading the pixel as the object if the pixel is greater than the predetermined threshold value.

의 수식을 통해 상기 상기 r3, 상기 g3, 상기 b3를 보정하는 단계를 더 포함하고, 상기 수식에서, rt, gt, bt는 각 픽셀의 임계값이고, 선정된 시간 동안 촬영되는 배경 영상의 각 픽셀 값 변화를 감지하고, 상기 감지된 상기 각 픽셀 값 변화에서 최소값과 최대값을 연산하여 상기 각 픽셀의 임계값이 산출되는 것을 특징으로 한다.In addition, the chroma key synthesis method of the chroma key synthesizer according to an embodiment of the present invention may further include a step of calculating r3, g3, and b3,

Wherein rt, gt, and bt are threshold values of each pixel, and each pixel of the background image that is photographed during a predetermined period of time includes the step of correcting the r3, g3, And a threshold value of each pixel is calculated by calculating a minimum value and a maximum value in the detected change of each pixel value.

In another aspect of the present invention, there is provided a chroma key synthesizing method for a chroma key synthesizer, comprising the steps of: capturing an infrared image including the object on the background and reading depth information from the infrared image; The step of acquiring the depth information of the tracked object may include the steps of: removing noise of a contour line of the depth information image using pixel information of the object image; Matching a pixel corresponding to a pixel hole included in the object image using pixel information of the object depth image; And synthesizing the noise-removed object depth image and the object image matched with the pixel hole.

In addition, in the step of removing noise on the contour of the object depth image using pixel information of the object image in the chroma key combination method of the chroma key combination apparatus according to an embodiment of the present invention, Of the eight depths of the object depth image by checking the eight pixel values of the eight depth pixels and discarding the center pixel value if any one of the eight pixels has no value, And removing the masking image noise.

The step of matching the pixel corresponding to the pixel hole included in the object image using the pixel depth information of the object depth image of the chroma key combination method of the chroma key combination apparatus according to an embodiment of the present invention includes: (8) of the eight pixels are set as the center pixel value when any one of the eight pixels is a value, and the pixel value And a masking image hole filling technique for the image is performed.

According to the chroma key synthesis method of the chroma key composition apparatus of the present invention, it is possible to perform chroma key composition for an object required for chroma key image synthesis using only a color image and an infrared image for taking an object in a general background without shooting an object on a background screen such as a blue screen It is possible to obtain a more accurate and convenient image acquisition.

1 is a block diagram showing a configuration of a chroma key synthesizer according to an embodiment of the present invention.
2 is a flowchart showing a flow of a chroma key synthesis method according to an embodiment of the present invention.
3 is another flowchart showing the flow of a chroma key synthesis method according to a temporal example of the present invention.
4 is a diagram illustrating an example of a first image according to an embodiment of the present invention.
5 is a diagram illustrating an example of a second image according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an object image obtained through a pixel operation between a first image and a second image according to an embodiment of the present invention. Referring to FIG.
FIG. 7 illustrates a problem occurring in an object image according to an exemplary embodiment of the present invention. Referring to FIG.
8 is a view showing an object depth image obtained according to an embodiment of the present invention.
9 is a view showing an object depth image in which foreground masking image noise is removed from an object depth image according to an embodiment of the present invention.
10 is a diagram illustrating a chroma key object image obtained by synthesizing a color object image and an infrared object image according to an embodiment of the present invention.
11 is a diagram illustrating foreground masking of a chroma key object image according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing a configuration of a chroma key synthesizer according to an embodiment of the present invention.

A chroma key synthesizer that performs a chroma key image synthesis method that does not require a background screen in accordance with the present invention may be configured to include a computing device 110, a camera 121, an infrared camera 122, and a monitor 130 . The monitor 130 is a means for outputting a synthesized chroma key composite image, but is not limited to a monitor, and may output a chroma key composite image through another output means such as a smart terminal.

The camera 121 is not limited to a camera such as a camcorder, a digital camera, and a web camera as a means for capturing an image of a certain object, and a configuration for capturing an image and inputting the image to the computing device 110 is sufficient. The infrared camera 122 may also be configured to capture an infrared image and transmit the infrared image to the computing device 110.

The computing device 110 receives an image captured through the camera 121 and the infrared camera 122, generates an input image, tracks an outline and a movement of the object in the input image, and includes a target object to be extracted And an image synthesizing unit 113. The image synthesizing unit 113 synthesizes the synthesized image and the background image and synthesizes the synthesized image and the background image. The image input unit 111, the image processing unit 112, the image synthesizing unit 113, (115). The computing device 110 may be implemented as a device including a processor having an operating capability and an operating system (OS). For example, the computing device 110 may be implemented as a variety of devices including a processor and an operating system such as a PC, a MAC, a notebook PC, a tablet PC, a console game machine, a smart phone, and a smart pad.

FIG. 2 is a flowchart showing a flow of a chroma key synthesis method according to an embodiment of the present invention, and FIG. 3 is another flowchart showing a flow of a chroma key synthesis method according to a temporal example of the present invention.

According to an embodiment of the present invention, the chroma key synthesizer captures a first image that does not include an object in a background and reads and holds color information of the first image (step 210) ). The first image may be implemented such that no object is included as shown in FIG. And reads and stores the color value of the first image. Each pixel of the color image has three values of 8 bytes in RGB (YUV) color.

RGB: Red (8Byte) + Green (8Byte) + Blue (8Byte)

YUV: Y (8 bytes) + U (8 bytes) + V (8 bytes)

For a full HD video with a video size of 1920 x 1080, the size of the stored video may be 1,920 (width) x 1,080 (height) x 8 (bytes) x 3 = 49,766,400 bytes. The values of R, G, B, Y, U, and V of each color (RGB, YUV) have values of 0 to 255 since they are 8 bytes. For convenience of calculation, the above value may be expressed as a real number from 0 to 1.

The chroma key synthesizer captures a second image including the object on the background and reads and maintains color information of the second image (step 220). The second image is an image including an object having the same background as the first image as shown in FIG. The second image may also be stored in the memory as RGB (YUV) values.

The chroma key combination apparatus removes the color value corresponding to the color information of the first image from the second image and obtains the object image from which the background is removed from the second image and extracts only the object 230). The extracted object image may be implemented as shown in FIG.

를 연산하는 단계와, 상기 r3, 상기 g3, 상기 b3 가 각각 0에 수렴하는 경우 해당 픽셀을 상기 배경으로 판독하고, 선정된 임계값 이상인 경우 해당 픽셀을 상기 객체로 판독하는 단계를 포함한다. Step 230 includes matching the RGB (YUV) value of each pixel of the first image by the image size (Width * Height) corresponding to the RGB (YUV) value of each pixel of the second image, One of the pixel values of the second image is denoted by r1, g1 and b1, one of the pixel values of the first image is denoted by r2, g2 and b2, and one of the pixel values of the object image is denoted by r3, g3, and b3,

And reading the corresponding pixel as the background if the r3, the g3, and the b3 converge to 0, and reading the corresponding pixel as the object if the pixel is larger than the predetermined threshold value.

의 수식을 통해 상기 상기 r3, 상기 g3, 상기 b3를 보정하는 단계를 더 포함할 수 있다. 상기 수식에서, rt, gt, bt는 각 픽셀의 임계값이고, 선정된 시간 동안 촬영되는 배경 영상의 각 픽셀 값 변화를 감지하고, 상기 감지된 상기 각 픽셀 값 변화에서 최소값과 최대값을 연산하여 상기 각 픽셀의 임계값이 산출될 수 있다.After calculating r3, g3, and b3,

And correcting the r3, the g3, and the b3 through the following equation. In the above equation, rt, gt, and bt are threshold values of each pixel, and a change in each pixel value of a background image photographed for a predetermined time is detected, and a minimum value and a maximum value are calculated The threshold value of each pixel can be calculated.

의 각 값을 산출한다. 따라서 r3, g3, b3 이 0에 한없이 가까워 질 때 해당 픽셀은 배경으로 인식하게 되고 차이가 크면 전경으로 인식할 수 있다. 이 때, 제2 영상의 배경의 변화(조도 변화 및 기타 환경변화)로 인하여 의도치 않은 값이 생길 경우 이를 보정하기 위한, 각각의 칼라 값의 임계치 rt, gt, bt를 정해 이를 보정할 수 있다. 따라서 최종 알고리즘은 다음과 같다.In step 230, a corresponding one pixel RGB (YUV) value of the first image in the value RGB (YUV) of one pixel of the second image is performed by width * height, which is the size of the image. When one pixel of the second image is represented by r1, g1, and b1, one pixel of the first image is represented by r2, g2, and b2, and one pixel of the object image is represented by r3, g3,

Respectively. Therefore, when r3, g3, and b3 are close to zero, the corresponding pixel is recognized as the background, and if the difference is large, it can be recognized as the foreground. At this time, if an unintended value occurs due to a change in the background of the second image (change in illuminance and other environment), the threshold value rt, gt, bt of each color value for correcting the unintended value can be corrected and corrected . Therefore, the final algorithm is as follows.

If (r3 <0 && g3 <0 && b3 <0) condition is satisfied, it can be recognized as background pixel, otherwise it can be recognized as foreground pixel.

Since the values of rt, gt, and bt are obtained at the core of the color image chroma key in obtaining the threshold values rt, gt, and bt, in the present invention, two methods can be implemented in order to obtain the above values . First, in the case of the manual method, the user inputs the above three values and applies them to all the pixels. At this time, you can visually check the result value and specify three values that give the best result. This manual approach has the advantage of being able to obtain values at once. In the case of the automatic method, a background image is photographed for a predetermined time (about 10 seconds), and three threshold values of each pixel are obtained by finding the minimum and maximum values of the pixel values. This automatic method has an advantage of obtaining an accurate result because the threshold value is different for each pixel.

Since the foreground image is obtained on the basis of the color value of the corresponding color object image obtained in the step 230, if the foreground image is the same color as the background image, holes are formed like a red frame as shown in FIG.

In order to compensate for this, the chroma key synthesizer captures an infrared image including the object on the background and acquires an object depth image on which the object is tracked in the infrared image through depth information read from the infrared image Step 240). The object depth image can be obtained as shown in FIG.

The chroma key synthesizer can acquire depth information and provide skeleton tracking by analyzing characteristics of a dot pattern by projecting not only RGB images but also specific infrared dot patterns onto an object. The chroma key synthesizer can detect a pattern of infrared ray reflected by an object and generate a depth image using a light receiving sensor. However, due to the occurrence of a blind spot due to the difference in distance between the light emitting device and the light receiving sensor, the diffuse reflection of the infrared rays due to the smooth surface of the object is small and the infrared ray pattern can not be detected when the laser can not return to the light receiving sensor. The value may appear as a hole in the depth image. In addition, if the discontinuity of the depth information and the vertical direction of the infrared ray emitting part are vertical, a hole may be formed around the object boundary. Therefore, when each depth image is continuously reproduced as a moving image, In order to do so, it is possible to eliminate such fluctuation. The depth image can be generated by a pattern matching method by reading an infrared ray pattern inspected by an infrared ray projector with an infrared camera.

The chroma key synthesizer corrects the pixel of the object image and the pixel of the object depth image through pixel information of each other to synthesize the object image and the object depth image to obtain a chroma key object image (step 250) ).

In operation 250, the noise of the outline of the object depth image is removed using the pixel information of the object image, and the pixel depth of the object depth image is calculated using the pixel information of the object depth image. Matching the corresponding pixels with each other, and synthesizing the noise-removed object depth image and the object image matched with the pixel hole.

Wherein the chroma key combination apparatus identifies eight neighboring pixels on the basis of a center pixel and outputs the eight pixels of the object pixel based on the pixel information of the object image, The masking image noise of the object depth image can be removed by performing an erosion operation among the morphological operations of discarding the center pixel value.

Also, in the step of matching pixels corresponding to the pixel holes included in the object image using the pixel depth information of the object depth image, the chroma key synthesizer may calculate values of eight neighboring pixels based on the center pixel And performs a masking image hole filling technique on the object image through a dilation operation in an image processing morphological operation in which any one of the eight pixels is set as the center pixel value .

As shown in FIG. 9, the chroma key synthesizer may remove masking image noise through an erosion operation during the image processing morphology operation.

- reduction operation (Erosion algorithm)

As described above, by checking the values of eight neighboring pixels starting from the center pixel, if any one of the eight pixels does not exist, the center pixel value can be abandoned.

The chroma key synthesizer may perform a masking image hole filling technique through a dilation operation during the image processing morphology operation.

- Dilation algorithm (Dilation algorithm)

As described above, if the value of one of eight pixels is found by checking the values of eight neighboring pixels starting from the center pixel, the center pixel value can be obtained.

As shown in FIGS. 10 and 11, the chroma key synthesizer can synthesize two images by complementing the respective disadvantages of the color object image and the infrared object image. 11, the black color value is 0, the red color value is 255, and the black color value is 0 and the green color value is 255 in the foreground masking of the infrared image. In this case, when the values of a) image and b) image are combined, there are three values, 0, 255, and 510. If the result value is 0, it can be determined as the background image. If the result value is 255, it can be regarded as the foreground color. If the result is 510, the foreground color can be determined. If the result value is 255, it can be regarded as a foreground color in a) and b). Therefore, it is possible to determine the foreground if the 255 or more pixels are distributed. If the value is 0, the background can be determined.

The chroma key synthesizer may perform chroma key image synthesis by synthesizing the chroma key object image obtained according to this method with the selected background image (step 260).

The sleep state detection method according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

110: computing device
121: camera
122: Infrared camera
130: Monitor

Claims (7)

Capturing a first image that does not include an object in a background and reading and maintaining color information of the first image;
Capturing a second image including the object on the background and reading and maintaining color information of the second image;
(YUV) value of each pixel of the first image corresponding to the RGB (YUV) value of each pixel of the second image by the size of the image (Width * Height) One pixel value is referred to as r1, g1, and b1, one of the pixel values of the first image is referred to as r2, g2, and b2, and one of the pixel values of the object image is referred to as r3, g3, ,

Calculating a minimum value and a maximum value from a change in each pixel value detected in the step of calculating r3, g3, and b3, detecting a change in each pixel value of a background image photographed during a predetermined period of time, Rt, gt, and bt, which are threshold values of each pixel calculated and calculated, are applied

And correcting the r3, the g3, and the b3 through a formula of a pixel of interest, and if the r3, the g3, and the b3 converge to 0, the corresponding pixel is read as the background, The color value corresponding to the color information of the first image is removed from the second image to obtain the object image from which the background is removed and only the object is extracted from the second image step;
Capturing an infrared image including the object on the background and obtaining an object depth image in which the object is tracked in the infrared image through depth information read from the infrared image;
Acquiring a chroma key object image by synthesizing the object image and the object depth image by correcting pixels of the object image and pixels of the object depth image through mutual pixel information; And
Synthesizing the chroma key object image with the selected background image
Wherein the chroma-key synthesizing method comprises the steps of: delete delete The method according to claim 1,
The step of correcting the pixel of the object image and the pixel of the object depth image through pixel information of each other to synthesize the object image and the object depth image to obtain a chroma key object image,
Removing noise from a contour of the object depth image using pixel information of the object image;
Matching a pixel corresponding to a pixel hole included in the object image using pixel information of the object depth image; And
Synthesizing the noise-removed object depth image and the object image matched with the pixel hole
Wherein the chroma-key synthesizing method comprises the steps of: 5. The method of claim 4,
The method of claim 1, wherein removing the noise of the contour of the object depth image using the pixel information of the object image identifies eight neighboring pixels on the basis of the center pixel, And removing the masking image noise for the object depth image by performing an erosion operation among the morphological operations for discarding the center pixel value when there is no chrominance image synthesizing method . 5. The method of claim 4,
And matching the pixels corresponding to the pixel holes included in the object image using the pixel depth information of the object depth image to check the values of the neighboring eight pixels based on the center pixel, And a masking image hole filling method for the object image is performed through a dilation operation in an image processing morphological operation to set the center pixel value when there is a value of at least one value. &Lt; / RTI &gt; A computer-readable recording medium recording a program for executing the method of any one of claims 1, 4, 5, and 6.

Images