KR100661528B1 - Adjustive chroma key composition apparatus and method - Google Patents

Abstract

An adaptive chroma key synthesis apparatus and a method therefor are provided to be actively applied to various image conditions and enhance the quality of a chroma key. An image input unit(61) receives an image picked up through a camera and a 'background screen replacement image'. A pre-processing unit(62) scans an input image, spatially divides the entire picture per every frame, generates a separate chroma key for the respective divided pictures, tracks the change of the chroma key, and generates a chroma key. A chroma keying processing unit(63) performs a chroma keying process on the basis of the chroma key, replaces pixels corresponding to a background screen chroma key in the input image including an object with the 'background screen replacement image', and synthesizes the 'background screen replacement image'. An image output unit outputs the synthesized object and a 'background screen replacement image'.

Description

Adaptive chroma key composition apparatus and method

1 is a configuration diagram of a general chroma key synthesizing apparatus,

2 is an explanatory diagram showing an ideal chroma keying process;

3 is a schematic diagram showing a conventional chroma key synthesis process;

4A and 4B are explanatory diagrams showing a chroma keying process according to the prior art;

5 is a schematic diagram showing a chroma key synthesis process according to the present invention;

6 is a block diagram of an embodiment of an adaptive chroma key synthesizing apparatus according to the present invention;

7A and 7B are diagrams illustrating an embodiment of automatic chroma key adaptation among adaptive chroma key synthesis methods according to the present invention;

8A and 8B are diagrams illustrating an embodiment of real-time chroma key generation in an adaptive chroma key synthesis method according to the present invention;

9A to 9C are exemplary diagrams illustrating a process of applying a spatial division chroma key in an adaptive chroma key synthesis method according to the present invention;

10A to 10C are exemplary diagrams illustrating a chroma key application process based on luminance in an adaptive chroma key synthesis method according to the present invention.

* Explanation of symbols on the main parts of the drawing

61: image input unit 62: preprocessor

63: chroma key processing unit 64: post processing unit

65: video output unit

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an adaptive chroma key synthesizing apparatus and a method, and a computer readable recording medium recording a program for realizing the method, and more particularly, to various imaging conditions (camera, lighting, directing). Adaptive Chroma Key Synthesizing apparatus capable of actively adapting chroma key to a chroma key, and a method thereof, and a computer-readable recording medium recording a program for realizing the method. will be.

Color can usually be expressed separately by brightness (luma) and color difference (chroma).

A chroma key is used for image synthesis, and a specific color is designated as a key, and then a pixel corresponding to a key in the main image is replaced with another image. In this case, in an algorithm that performs a chroma key, a key means a range of chroma values, not a single chroma value.

Therefore, in order to perform a chroma key, a background screen to be a key is required, and a displacement of chroma values of the background screen should be small. In addition, the background screen and the object (all objects distinguished from the screen including a person) should be separated from each other on the basis of chroma.

Referring to the general chroma key synthesizing apparatus of FIG. 1, a specific color is designated as a key, and then the background of the object and the background screen image captured by the camera based on the chroma key is selected. The synthesized image is generated by replacing the screen image with another background image.

At this time, if the chroma of the image entered through the camera (chroma) is displayed in the color distribution, as shown in FIG. Ideally, the chroma component of the screen and the chroma component of the object would be completely separated without overlapping. That is, in FIG. 2, it can be seen that the chroma component of the screen is shifted to green so that the chroma component of the screen and the chroma component of the object are completely separated.

The key is set based on the screen chroma distribution and the range of keying is set to be chroma keying around the key. In this case, the method of setting the range is free, but as shown in FIG. 2, a rhombic range is set around the key.

As a prerequisite for successful chroma key technology, we will focus on image quality, key separation, and equipment.

First, since chroma key technology is an image handling technology, the image quality of a source should be good. The quality of the image is highly dependent on the shooting technique, the equipment, the lighting, the actor's costume, and the quality and condition of the background screen.

On the other hand, in order to separate the background screen (object) and the object (object) must be planned in advance so that the color of the object (object) does not match the color of the background (background). In addition, even if the color of the object is different from the color of the background, the lighting conditions must be planned in advance because they may be the same under certain lighting conditions. In addition, the camera technique must be planned in advance because the object can be in the same chroma range as the background under a particular camera rendering technique. In addition, the object may be in the same color range as the background by causing the reflection or shadow, the change of the lighting state, etc. due to the appearance of another object, so plan this in advance. In addition, in order to plan the object, camera, lighting, etc. in advance, it is necessary to be able to predict the chroma key of the background before shooting. In addition, all cameras distort the color and other conditions of the object to some extent, so that chroma key values of the background screen are also visually seen by the key values and images coming from the camera. Since the chroma key values of the signal are different, these differences must be minimized so that you can shoot with the color scheme planned in advance. In addition, since chroma components of the background screen may be changed by objects, lighting, camera rendering techniques, and the like, in this case, the reference key value may also change. Plan ahead to avoid this.

On the other hand, equipment is also important along with the production plan. That is, if you use a noisy camera, use a background screen with irregular colors, or a screen that reflects light, successful chroma keying cannot be performed. In addition, studios that can avoid unpredictable sunlight and lighting are also conditions that can improve the quality of chroma keying.

Next, the problems of the conventional chroma key technology will be described.

Looking at the equipment first, it is essential to use a camera that is classified as a broadcast to prevent noise and distortion by the camera. There is also a need for a dedicated screen that is less susceptible to illumination and objects and can maintain a uniform chroma distribution. There is also a need for studio lighting that can minimize shadows, color variations and changes in illumination. There is also a need for a highly planned studio. Conventionally, under these equipment conditions (fixed after the operation), the input image was accepted as it is (see Fig. 3).

In particular, when using a general camera, the auto exposure effect is described. As shown in FIGS. 4A and 4B, the screen and the object may be changed according to the passage of time, the movement of the object, and the camera. You can see that the color of the object changes significantly. Conventional chroma key techniques do not allow keying for screens that change over time. Specifically, as shown in FIG. 4A, as the space occupied by the white clothes increases, the exposure decreases gradually. As a result, the green screen observed by the eyes does not change, but is mounted through a camera. Green turns black and white clothes are dark. In addition, as shown in FIG. 4B, as the space occupied by the black clothes increases, the green screen becomes brighter, and the clothes which look black also appear to be light gray by the A / E and have a slightly reddish energy. do. Therefore, according to the conventional chroma key technique, it can be seen that keying is not possible for a screen that changes with time.

On the other hand, since only the key value is planned in advance, a lot of effort is required to maintain the key value. In addition, there are no chroma key products that can be equally applied to different studios with different conditions. In addition, there is no room for non-directed ad libs to intervene (unspecified users cannot appear). In addition, the deterioration of chroma key quality caused by lack of directing or equipment is inevitably compensated for in the latter part of the work (difficult to apply in real time).

On the other hand, there is no additional device other than the traditional chroma key replacement algorithm. In addition, all the problems that can occur can only be solved through the equipment, directing and post-production work.

On the other hand, coverage is limited to broadcasting, movies, weather forecasts, virtual reality systems, and the like. Also, due to equipment, studio and production problems, it is impossible to make products that can be used by various layers.

As such, the conventional chroma key technology has to rely on equipment (camera, lighting, directing, studio), directing, and post-production to complement chroma key quality.

The present invention has been proposed to solve the above problems, and can be adaptively adapted to various imaging conditions (camera, lighting, directing, studio, etc.) to increase the quality of chroma keys, adaptive chroma It is an object of the present invention to provide a Marquee synthesizing apparatus, a method thereof, and a computer-readable recording medium having recorded thereon a program for realizing the method.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. It will also be appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

A chroma key synthesizing apparatus of the present invention for achieving the above object comprises an image input means for receiving an image captured by a camera and a background screen replacement image; Pre-processing means for generating a chroma key by scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and tracking a change in the chroma key value; Chroma keying for chroma keying based on the chroma key to replace and synthesize a pixel corresponding to a background screen chroma key in the input image including an object with the background screen replacement image. Ing processing means; And image output means for outputting the synthesized object and the background screen substitute image.

delete

The present invention also provides a background screen by applying a luma component in addition to a chroma component when there is a common part (intersection) between the object and the background screen during the chroma keying process. And post-processing means for minimizing the common color gamut of the object.

Accordingly, a more preferred embodiment of the present invention is a chroma key synthesizing apparatus, comprising: image input means for receiving an image captured by a camera and a background screen substitute image; Pre-processing means for generating a chroma key by scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and tracking a change in the chroma key value; Chroma keying for chroma keying based on the chroma key to replace and synthesize a pixel corresponding to a background screen chroma key in the input image including an object with the background screen replacement image. Ing processing means; In the chroma keying process, when there is a common part (intersection) between the object and the background screen, a luma component is applied to the background screen and the object by applying a luma component in addition to the chroma component. Post-treatment step to minimize the; And image output means for outputting the synthesized object and the 'background screen replacement image'.

On the other hand, the present invention, in the chroma key (chroma key) synthesis method, the step of receiving an image captured by the camera and the 'background screen replacement image'; Scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and generating a chroma key by tracking a change in the chroma key value; Performing chroma keying on the basis of the chroma key, and replacing and synthesizing a pixel corresponding to a background screen chroma key with the 'background screen replacement image' in the input image including the object; And outputting the synthesized object and the 'background screen replacement image'.

delete

The present invention also provides a background screen by applying a luma component in addition to a chroma component when there is a common part (intersection) between the object and the background screen during the chroma keying process. And minimizing the common color gamut of the object.

On the other hand, the present invention, a chroma key (chroma key) synthesizer having a processor, a function for receiving an image captured by the camera and the background screen replacement image; Scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and generating a chroma key by tracking a change in the chroma key value; A function of performing chroma keying on the basis of the chroma key and replacing and synthesizing a pixel corresponding to a background screen chroma key with the 'background screen replacement image' in the input image including the object; In the chroma keying process, when there is a common part (intersection) between the object and the background screen, a luma component is applied to the background screen and the object by applying a luma component in addition to the chroma component. The ability to minimize; And a computer readable recording medium having recorded thereon a program for realizing a function of outputting the synthesized object and the background screen substitute image.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Looking at the chroma key synthesis process according to the present invention with reference to FIG.

If the existing method solves chroma key quality mainly by equipment, production, and post-production, the present invention intelligently adapts to the image by reinforcing the algorithm while maintaining the equipment and production conditions. The quality of chroma keys can be improved.

In other words, an expensive broadcast camera is not required for a chroma key, and a home camcorder or a surveillance camera can be used. It doesn't require a well-designed studio and can be used anywhere in a typical office, living room or study room. In addition, it is not significantly influenced by the type or location of the light, it can be applied as long as the light enough to identify the object. In addition, there is no need for a separate direction, and intelligently responds to the overlapping movements and colors of various environments, objects. In addition, there is no need for additional post-production, and a chroma keyed image can be generated in real time.

Therefore, as shown in Fig. 5, since any camera, any screen, and illumination are used randomly, the chroma key synthesizing apparatus of the present invention (see Fig. 6 below) is applied to a precondition and a key. No information is available.

6 is a block diagram of an embodiment of an adaptive chroma key synthesis apparatus according to the present invention.

As shown in FIG. 6, in the adaptive chroma key synthesizing apparatus according to the present invention, an image input unit 61 for receiving an image captured by a camera and a background screen substitute image, and an input image are scanned. Pre-processing unit 62 for generating a chroma key by tracking the change of the chroma key value every frame, and chroma keying processing based on the chroma key to display the background screen in the input image including the object. A chroma keying processor 63 for replacing and synthesizing a pixel corresponding to a mark with the "background screen replacement image", and an image output unit 65 for outputting the synthesized object and the "background screen replacement image". ). In particular, in the chroma keying process, when there is a common part (intersection) between the object and the background screen, a luma component is applied to the background screen and the object in addition to the chroma component. It further includes a post processor 64 for minimizing the area.

The preprocessing unit 62 further includes a function of spatially dividing the entire screen every frame and generating a separate chroma key for each division surface.

Then, the function of each component of the adaptive chroma key synthesizing apparatus according to the present invention will be described in detail.

In the preprocessing unit 62, a chroma key auto adjustment process (see FIGS. 7A and 7B), a realtime chroma key generation process (see FIGS. 8A and 8B), and a spatial division chroma key application process (See FIGS. 9A-9C).

First, a chroma key auto adjustment process (FIGS. 7A and 7B) is a process of recognizing itself without specifying a chroma value of a background screen in advance. That is, it is possible to calculate the optimal chroma key value at any place, at any lighting, under any camera and screen. The automatic adaptation button can be used while only the background screen is photographed through the camera. Pressing the (auto adjustment button) (701) scans the entire image (702), and selects only values close to a specific red-cyan (RGB) strain (703, 704) to generate chroma key values. (705). For example, a key value is calculated using a sample extracted for a green color.

However, an image photographed under poor conditions may appear in different colors depending on the condition even for the same object and screen. For example, when the lighting changes from fluorescent to incandescent or when a closed curtain is opened, when the camera's auto expose function is activated, when there is an additional appearance or exit of an object, and so on. Yes.

In this case, even if chroma keying is not possible with the key value of the screen obtained at first, the quality is deteriorated.

Accordingly, in order to overcome this problem, a realtime chroma key generation process is performed as shown in FIGS. 8A and 8B.

The real-time chroma key generation process (FIGS. 8A and 8B) scans the screen in real time to obtain chroma key values of the screen every frame. (Track changes in chroma key values).

In this case, a chroma key value obtained initially is used to distinguish an object from a screen in each frame. In addition, each frame utilizes chroma key values of the previous (previous) frame to distinguish an object from a screen.

That is, in generating a chroma key every frame, when there is no key of a previous frame for a new frame (801), the chroma key automatic adaptation process 802 (see FIG. 7B) is performed. Create and store the chroma key value of the previous frame. If there is a chroma key of the previous frame (801), after scanning the screen (803), the pixels expected to the screen are sampled using the chroma key of the previous frame. In operation 804, a chroma key of the frame is generated using the screen candidate pixels. Thereafter, a chroma keying process is performed (806).

Now, a process of applying the spatial division chroma key will be described with reference to FIGS. 9A to 9C.

In a generalized environment, even if a screen of a certain color is used, there is an uneven distribution of colors depending on the position of the light, the reflection, and the state of the object.

Therefore, when the deviation of the average value in the distribution of screen colors increases, the entire screen may not correspond to the calculated chroma key value. That is, as shown in FIG. 9A, since the upper part is bright and the lower part is dark (when it occurs very often in a general environment), the same chroma key cannot be applied to the upper part and the lower part.

Therefore, since only a chroma key for the entire screen is insufficient, the present invention obtains a separate chroma key for each divided surface by spatially dividing the entire screen. That is, as illustrated in FIG. 9B, a separate chroma key is applied by spatially dividing the entire screen. By doing this, it can be seen that a very even color distribution is seen from each division plane.

In this way, when a separate chroma key is obtained for each divided surface (FIG. 9B), an unbalance in color distribution due to lighting inequality or camera characteristics is eliminated.

That is, in generating a chroma key every frame, for a new frame, after scanning the screen (902), the chroma key value of the same division of the previous frame is loaded (903). By sampling 904 the screen candidate based on the value and sampling based on the systematic color feature of the screen (905), a chroma key is generated (906). Thereafter, a chroma keying process is performed (907).

Meanwhile, the post processor 64 performs a chroma key application process based on luminance (see FIGS. 10A to 10C).

The colored regions are magenta (M), blue (B), cyan (cu), green (G), and yellow (Y) as shown in FIG. 10A. It is divided into six representative hue of red (R) (red).

And, the representation of color is expressed using a method called YCbCr as shown in Figure 10b. In general, RGB is a method, which is a method of expressing a color by mixing colors of red light (R), green light (G), and blue light (B), which are three elements of light. However, YCbCr is a method of expressing a color using luminance Y, which is the intensity of color, color difference information Cb with blue, and color difference information Cr with red.

We will look at advanced chroma key techniques based on luma.

Chroma keys are literally keying techniques based on chroma components of color. Therefore, even if the luma components are different, if the chroma components are the same, the same key is recognized. Therefore, as shown in FIG. 10A, when a common part (intersection) of the screen and the object occurs, the existing chroma key technique recognizes this area as the screen.

However, since the human visual system is more sensitive to luma than chroma, even if the same chroma components are different in luma, they are recognized as different colors. In other words, the conventional chroma key technique overlooks this and treats the screen color reflected on the white paper in the same way as the screen.

In the conventional chroma key technique, all chroma areas that are identical to the screen are treated as screen regions, but in the chroma-based advanced chroma key technique, the ratio is lowered. It was.

That is, as shown in FIG. 10B, when a common part (intersection) of the screen and the object is generated, the common area may be minimized by applying a luma component in addition to chroma.

Referring to the processing procedure of FIG. 10C, after chroma keying is performed on the camera input image (pixel unit) (101), the camera input image is maintained for the object region (102, 103), and the screen is displayed. For the region, a pixel corresponding to the background screen chroma key in the input image including the object is replaced with a background screen replacement image (102 and 104). In this case, when a common part (intersection) of the screen and the object is generated, the common area may be minimized by applying a luma component in addition to chroma.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

According to the present invention as described above, even a non-expert who has a poor understanding of chroma key technology can perform image synthesis by simple operation, and can be implemented without expensive equipment and annual output, thereby making it easy to implement chroma key and the total cost of chroma key implementation. This can reduce costs (low-cost cameras, low-cost screens, no studio costs, no lighting costs, low-cost chromakey equipment, and no video expertise required).

In addition, according to the present invention, in addition to the existing broadcast, movie, virtual reality system, it is possible to create demand as an entertainment element in toys, karaoke, photo room, etc., and simulation and broadcasting such as driving and training (personal broadcasting, small internet broadcasting station, video freelancer) Etc.) can be used.

Claims (9)

delete In a chroma key synthesizing apparatus, Image input means for receiving an image captured by a camera and a background screen substitute image; Pre-processing means for generating a chroma key by scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and tracking a change in the chroma key value; Chroma keying for chroma keying based on the chroma key to replace and synthesize a pixel corresponding to a background screen chroma key in the input image including an object with the background screen replacement image. Ing processing means; And And an image output means for outputting the synthesized object and the background screen substitute image. The method of claim 2, In the chroma keying process, if there is a common part of the object and the background screen, post-processing means for minimizing the common color gamut of the background screen and the object by additionally applying a luma component. Adaptive Chroma key synthesis device further comprising. The method of claim 2 or 3, The pretreatment means, Adaptive chroma key synthesizing apparatus characterized by using the chroma key of the previous frame to distinguish the object and the screen in each frame. In a chroma key synthesizing apparatus, Image input means for receiving an image captured by a camera and a background screen substitute image; Pre-processing means for generating a chroma key by scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and tracking a change in the chroma key value; Chroma keying for chroma keying based on the chroma key to replace and synthesize a pixel corresponding to a background screen chroma key in the input image including an object with the background screen replacement image. Ing processing means; In the chroma keying process, when there is a common part (intersection) between the object and the background screen, a luma component is applied to the background screen and the object by applying a luma component in addition to the chroma component. Post-processing means for minimizing; And And an image output means for outputting the synthesized object and the background screen substitute image. delete In the chroma key synthesis method, Receiving an image captured by the camera and a 'background screen substitute image'; Scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and generating a chroma key by tracking a change in the chroma key value; Performing chroma keying on the basis of the chroma key, and replacing and synthesizing a pixel corresponding to a background screen chroma key with the 'background screen replacement image' in the input image including the object; And Adaptive chroma key synthesis method comprising the step of outputting the synthesized object and the background screen replacement image. The method of claim 7, wherein In the chroma keying process, when there is a common part (intersection) between the object and the background screen, a luma component is applied to the background screen and the object by applying a luma component in addition to the chroma component. Adaptive Chroma Key synthesis method further comprising the step of minimizing. In a chroma key synthesizing apparatus having a processor, A function of receiving an image captured by a camera and a 'background screen substitute image'; Scanning the input image, spatially dividing the entire screen every frame, generating a separate chroma key for each divided surface, and generating a chroma key by tracking a change in the chroma key value; A function of performing chroma keying on the basis of the chroma key and replacing and synthesizing a pixel corresponding to a background screen chroma key with the 'background screen replacement image' in the input image including the object; A function of minimizing a common color gamut of the background screen and the object by additionally applying a luma component when there is a common part of the object and the background screen in the chroma keying process; And A computer-readable recording medium having recorded thereon a program for realizing a function of outputting the synthesized object and a background screen substitute image.

Images