JP2008527951A - Adaptive chroma key composition apparatus and method - Google Patents

Abstract

  The present invention relates to a chroma key synthesizing apparatus in which a video input means for receiving an image taken through a camera and an alternative video for a background screen, and an input video are scanned and then the entire screen is spatially divided for each frame. , After generating a separate chroma key for each divided surface, a pre-processing means for generating a chroma key by tracking a change in chroma key value, and a chroma keying process based on the chroma key, and an input image including an object And a chroma keying processing means for substituting and synthesizing the pixel corresponding to the background screen chroma key with the substitute image of the background screen, and at the time of the chroma keying process, a common part (intersection) of the object and the background screen is If present, apply a luminance component in addition to the chroma component to minimize the common hue area between the background screen and the object. Including a post-processing means for, and a video output means for outputting a substitute image of the synthesized objects and the background screen.

Description

  The present invention relates to an adaptive chroma key synthesizing apparatus and method, and a computer-readable recording medium on which a program for realizing the method is recorded. More specifically, the present invention relates to various video conditions (camera, illumination). An adaptive chroma key synthesizer that actively adapts to the quality of a chroma key by actively adapting it to a production, studio, etc., and a method thereof, and a computer-readable recording medium on which a program for realizing the method is recorded .

  The color (color) is usually shown separately in luminance (brightness (luma)) and color difference (color difference (chroma)).

In order to compose a video, a chroma key designates a specific color (chroma) as a key and then replaces a pixel corresponding to the key in the main video with another video. At this time, in the algorithm for performing the chroma key, the key means a range of chroma values, not a single chroma value.
Therefore, in order to perform the chroma key, a background screen as a key is required, and the displacement of the chroma value of the background screen is reduced. The background screen and the object (all objects including people and separated from the screen) should be separated from each other on the basis of chroma.

The general chroma key composition apparatus of FIG. 1 designates a specific color (chroma) as a key, and then, based on the chroma key, the object and the background screen image photographed through the camera. The synthesized video is generated by replacing the video.
At this time, as shown in FIG. 2, the chroma of the video that has entered through the camera is displayed in a color distribution. In an ideal case, the chroma component of the screen and the chroma component of the object are completely separated without overlapping. That is, in FIG. 2, the chroma component of the screen is tilted green, and the chroma component of the screen and the chroma component of the object are completely separated.

The key is set based on the distribution of the screen chroma, and the keying range is set so that the chroma keying is performed around the key. At this time, the method of setting the range is arbitrary, but usually, a diamond-shaped range is set around the key as shown in FIG.
The main preconditions for successful chroma key technology are image quality, key separation, and equipment.
Since the chroma key technology is a technology that handles video, the image quality of the video that becomes the source is the most important. The image quality depends greatly on the shooting technique and equipment, lighting, actor's costume, background screen condition and quality.

  On the other hand, in order to separate the background screen and the object, planning is performed in advance so that the hue of the object does not become the same as the hue of the background. Furthermore, even if the hue of the object is different from the hue of the background, it may be the same under specific lighting conditions, so the lighting conditions are planned in advance. In addition, since the object may enter the same chroma range as the background under a specific camera production technique, the camera technique is planned in advance.

  Since an object may enter a range of the same hue as the background by inducing reflection or shadow, a change in lighting state, etc. due to the appearance of another object, this is planned in advance. Also, in order to plan the object, camera, lighting, etc. in advance, the chroma key value of the background is predicted before shooting. In addition, since all cameras distort the actual hue and other conditions to some extent, the chroma key value of the background screen also has a difference between the key value visually seen and the chroma key value of the video signal coming through the camera. The difference can be minimized and the image can be taken as planned in advance.

  Since the chroma component of the background screen changes depending on the object, lighting, camera production technique, etc., the key value used as a reference also changes in this case, so that a plan is made in advance so that such a phenomenon does not occur. On the other hand, equipment is also important along with the production plan. That is, if a camera with severe noise, a background screen with a non-constant hue, or a screen that reflects illumination is used, successful chroma keying cannot be performed. Studios that avoid unpredictable sunlight and lighting are also conditions that improve the quality of chroma keying.

  In the following, the problems of conventional chroma key technology will be considered. First of all, it is essential to use a camera that must be classified for broadcasting in order to prevent noise and distortion caused by the camera. There is also a need for a dedicated screen that is less affected by lighting and objects and maintains a uniform chroma distribution. In addition, there is a need for studio lighting that minimizes changes in shadows and hues, and changes in illumination, and a highly planned studio.

As shown in FIG. 3, conventionally, the input video is accommodated as it is under such equipment conditions (fixed after operation).
In particular, when using a general camera, the auto exposure effect is caused by the time flow, the movement of the object, and the color of the screen and the object depending on the camera, as shown in FIGS. . Conventional chroma keying techniques do not allow keying for screens that change over time.
More specifically, as shown in FIG. 4, the exposure gradually decreases as the space occupied by the white clothes increases, and as a result, there is no change in the green screen observed by the eyes, but it passes through the camera. Green turns black and white clothes are darker. In addition, as shown in FIG. 5, the green screen becomes brighter due to the increase in space occupied by black clothes, and the clothes that appear black also appear light gray due to A / E and contain a little red. Thus, according to conventional chroma key techniques, keying is not possible for a screen that changes over time.

  On the other hand, since only pre-planned key values are used, great effort must be made to maintain the key values. Also, there are no chroma key products that can be applied to different studios with different conditions. In addition, there is no room for ad-libs that are not directed (unspecified users cannot appear). Furthermore, if the production and equipment are inadequate, the deterioration of the chroma key quality that occurs will be supplemented by the work in the second half (real-time application is difficult).

On the other hand, there are no other additional devices outside of the traditional chromakey alternative algorithm. In addition, all problems that occur will be solved through equipment, production and work in the second half.
On the other hand, the scope of application is limited to broadcasting, movies, weather forecasts, virtual reality systems, and the like. Also, products that can be used by various levels cannot be created due to problems with equipment, studios, and production.
Thus, the conventional chroma key technology depends on equipment (camera, lighting, production, studio) and production, and work in the second half in order to complement the quality of the chroma key.

US Pat. No. 5,838,310 JP-A-5-336539 International Publication No. 98/37702 Pamphlet Korean Patent Application Publication No. 10-2004-0110748

  The present invention has been proposed in order to solve the above-mentioned problems, and adaptive chroma key composition that actively adapts to various video conditions (camera, lighting, production, studio, etc.) to improve chroma key quality. It is an object of the present invention to provide an apparatus and a method thereof, and a computer-readable recording medium on which a program for realizing the method is recorded.

  In order to achieve the above-described object, the chroma key synthesizing apparatus of the present invention scans an input image, an image input means for receiving an image captured through a camera and an alternative image of a background screen. Then, after dividing the entire screen every frame and generating a separate chroma key for each divided plane, a preprocessing means for tracking a change in chroma key value and generating a chroma key, and Chroma keying processing means for performing chroma keying processing based on the chroma key, and substituting and synthesizing pixels corresponding to the chroma key of the background screen with the substitute image of the background screen in the input video including the object, Including a synthesized object and a video output means for outputting a substitute video of the background screen. To.

Further, the present invention applies a luminance component in addition to the chroma component to the common between the background screen and the object when there is a common part (intersection set) between the object and the background screen during the chroma keying process. It further includes post-processing means for minimizing the hue area.
Accordingly, a more preferred embodiment of the present invention is to provide a chroma key synthesizing apparatus, a video input means for receiving an image captured through a camera and an alternative image of a background screen, and every frame after scanning the input video. The entire screen is divided into spaces, a separate chroma key is generated for each divided plane, a pre-processing means for generating a chroma key by tracking changes in chroma key values, and a chroma keying process based on the chroma key And a chroma keying processing means for substituting and synthesizing a pixel corresponding to a chroma key of the background screen with an alternative image of the background screen in the input video including the object, and at the time of the chroma keying process, the object and the background screen When there is a common part (intersection set), the background screen is applied by applying a luminance component in addition to the chroma component. Characterized in that it comprises a post-processing means for minimizing a common hue area and object, and an image output means for outputting a substitute image of the synthesized objects and the background screen.

  On the other hand, the present invention provides a chroma key composition method in which an image captured through a camera and an alternative image on a background screen are input, and after the input image is scanned, the entire screen is spatially divided for each frame. After generating a separate chroma key for the surface, tracking the change of the chroma key value to generate the chroma key, and performing chroma keying processing based on the chroma key, corresponding to the chroma key of the background screen in the input video including the object Substituting and synthesizing the pixel to be replaced with the substitute image of the background screen, and outputting the substitute image of the synthesized object and the background screen.

  Further, the present invention provides a common hue between the background screen and the object by applying a luminance component in addition to the chroma component when there is a common part (intersection set) between the object and the background screen during the chroma keying process. The method further includes minimizing the area.

On the other hand, the present invention provides a chroma key synthesizing device equipped with a processor to receive an image captured through a camera and an alternative image of a background screen, and after the input image is scanned, the entire screen is divided into spaces every frame. Then, after generating a separate chroma key for each divided surface, a function for generating a chroma key by tracking changes in the chroma key value, and a chroma keying process based on the chroma key, and an input image including the object as a background When there is a common part (intersection) between the object and the background screen at the time of the chroma keying process, and the function of substituting and synthesizing the pixel corresponding to the screen chroma key with the substitute image of the background screen, other than the chroma component A function of minimizing a common hue area between the background screen and the object by applying a luminance component to the object, and the synthesized object and background To provide a recording medium to be read in the recorded computer program for realizing the function of outputting a clean alternative image.
Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

  The present invention enables non-experts with inferior understanding of chroma key technology to synthesize video by simply operating, and can be implemented without expensive equipment and performance output. Reduces the total cost of implementing the chroma key (low-priced camera, low-priced screen, studio-free, lighting-free, low-priced chroma key equipment, no need for image specialists).

  In addition to existing broadcasting, movies, and virtual reality systems, the present invention creates demand as an element of entertainment in toys, karaoke, photo booths, etc., and simulates and broadcasts driving, training, etc. This is used for large-scale Internet broadcasting stations, video flickers, etc.).

FIG. 6 shows a chroma key composition process according to the present invention.
Assuming that the existing method solves the chroma key quality mainly by equipment and production, and the work in the latter half, the present invention is intelligently converted into an image through reinforcement of the algorithm while maintaining the conditions of equipment and production. Increase the quality of the chroma key by adapting.
That is, because of the chroma key, an expensive broadcasting camera (camera) is unnecessary, and a home video camera (camcoder), a monitoring camera, or the like is used. In addition, a well-designed studio is unnecessary, and it can be used anywhere, such as a general office, living room, or study room. In addition, the present invention can be applied as long as it is not affected greatly by the type and position of the illumination and can identify the thing. Furthermore, there is no need for a separate performance, and it intelligently reacts to various environments, movements of objects, and duplication of hues. Also, a separate second half of the work is unnecessary, and the chroma key application video can be created in real time.
Therefore, as shown in FIG. 6, since any camera, any screen, and illumination are used randomly, the chroma key composition apparatus (see FIG. 7) of the present invention relates to preconditions and keys. I do not know any information.

FIG. 7 is a block diagram of an embodiment of an adaptive chroma key composition apparatus according to the present invention.
As shown in FIG. 7, the adaptive chroma key synthesizing apparatus according to the present invention includes a video input unit 61 for receiving an image captured through a camera and an alternative image of a background screen, and after scanning the input image. The change of the chroma key value is tracked every frame, the pre-processing unit 62 for generating the chroma key, and the chroma keying process based on the chroma key, and the pixel corresponding to the chroma key of the background screen in the input image including the object is obtained. It includes a chroma keying processing unit 63 for substituting and synthesizing with the substituting video of the background screen, and a video output unit 65 for outputting the synthetic object and the substituting video of the background screen. In particular, when there is a common part (intersection) between the object and the background screen during chroma keying processing, in order to minimize the common area between the background screen and the object by applying a luminance component in addition to the chroma component The post-processing unit 64 is further included.
The pre-processing unit 62 further includes a function of spatially dividing the entire screen for each frame and generating a separate chroma key for each divided surface.

Hereinafter, functions of each component of the adaptive chroma key composition apparatus according to the present invention will be described in more detail.
In the pre-processing unit 62, a chroma key auto adjustment process (see FIG. 8 and FIG. 9), a real-time chroma key generation process (see FIG. 10 and FIG. 11), and An application process (see FIGS. 12 to 14) of chroma key for space division is performed.

  As shown in FIGS. 8 and 9, the process of automatically adapting the chroma key is a process of recognizing itself without specifying the chroma value of the background screen in advance. That is, the optimal chroma key value is calculated even at any place, any lighting, any camera and under the screen, but when an automatic adaptation button (auto adjustment button) is pressed while only the background screen is photographed through the camera ( 701), the entire video is scanned (702), and only the values close to the specific red R, green G, and blue B series are extracted (703, 704) to generate a chroma key value (705). For example, the key value is calculated using samples extracted for green series colors.

However, images shot under inferior conditions have other hues depending on the conditions even if they are the same object and screen. For example, when the lighting changes from a fluorescent lamp to an incandescent lamp, or when a closed curtain is opened, the automatic exposure function of the camera operates, or there is an additional appearance or exit of an object.
In this case, the chroma keying cannot be performed with the screen key value obtained first, or even if it can, the quality is deteriorated.
Accordingly, in order to overcome this problem, a real-time chroma key generation process is performed as shown in FIGS.

As shown in FIGS. 10 and 11, the real-time chroma key generation process scans the screen in real time to obtain the chroma key value of the screen every frame (tracks the change of the chroma key value).
At this time, the chroma key value obtained first is used to distinguish the object and the screen in each frame. In addition, the chroma key value of the previous (previous) frame is used to distinguish the object and the screen in each frame.
That is, when the chroma key is generated for each frame and there is no key of the previous frame for the new frame (801), as shown in FIG. 9, the previous frame is subjected to the chroma key automatic adaptation process (802) as shown in FIG. Generate and store chroma key values. If there is a chroma key for the previous frame (801), the screen is scanned (803), and the pixel expected for the screen is sampled using the chroma key for the previous frame (804). A chroma key for the frame is generated (805). Thereafter, a chroma keying process is performed (806).

With reference to FIGS. 12 to 14, the process of applying the spatial division chroma key will be described.
In a generalized environment, even if a screen of a certain color is used, it has an uneven distribution of hues depending on the position of illumination, reflection, and the state of the object.
Therefore, if the deviation from the average value becomes large in the hue distribution of the screen, the entire screen cannot correspond to the calculated chroma key value. That is, as shown in FIG. 12, the upper side is bright and the lower side is dark (this occurs very often in a general environment), so the same chroma key cannot be applied to the upper side and the lower side.
Therefore, since only the chroma key for the entire screen is insufficient, in the present invention, the entire screen is spatially divided and a separate chroma key is requested for each divided surface. That is, as shown in FIG. 13, the entire screen is divided into spaces and a separate chroma key is applied. In this way, when only the divided surfaces are viewed, a very uniform color distribution is obtained.
In this way, when a separate chroma key is requested for each divided surface (FIG. 13), the hue distribution imbalance due to illumination unevenness, camera characteristics, and the like is eliminated.

That is, in generating a chroma key for each frame, after scanning the screen for a new frame (902), the chroma key value of the same division plane of the previous frame is loaded (903) based on the previous key value Screen candidates are sampled (904), and chroma keys are generated (906) by sampling based on the sequence color characteristics of the screen (905). Thereafter, a chroma keying process is performed (907).
On the other hand, as shown in FIGS. 15 to 17, the post-processing unit 64 performs a chroma key application process based on luminance. As shown in FIG. 15, the chromatic color region is divided into six representative shades of magenta M (magenta), blue B, cyan C (cyan), green G, yellow Y, and red R.
The hue is expressed using the YCbCr method as shown in FIG. Usually, the RGB method is used, but this method is a method of expressing colors by mixing the colors of red light R, green light G, and blue light B, which are the three elements of light. However, YCbCr is a method of expressing a color with luminance Y, which is the intensity of the color, information Cb on the color difference between blue and information Cr on the color difference with red.

The chroma key technique developed based on luminance will be described. Since the chroma key is a keying technique based on the chroma component of the color, the same key is recognized as the same chroma component even if the luminance component is different. Therefore, as shown in FIG. 15, when a common part (crossing set) between the screen and the object occurs, this area is recognized as a screen by the existing chroma key technique.
However, since the human visual system is more sensitive to luminance than chroma, even if the same chroma component is used, if the luminance is different, it is recognized as a different color. That is, since the existing chroma key technique overlooks this, the color of the screen reflected on the white paper is also treated like a screen.

In the existing chroma key technique, the same chroma area as the screen is treated as the area of the screen, but in the chroma key technique developed based on the luminance, the ratio is lowered.
That is, as shown in FIG. 16, when a common part (crossing set) between the screen and the object occurs, the common area can be minimized by applying the luminance component in addition to the chroma.

  As shown in FIG. 17, the procedure of this process is as follows. After the chroma keying process is performed on the input image (pixel unit) of the camera (101), the input image of the camera is maintained for the object area. (102, 103) For the screen area, the pixel corresponding to the chroma key of the background screen is replaced with the substitute image of the background screen in the input image including the object (102, 104). At this time, if a common part (crossing set) between the screen and the object occurs, the common area can be minimized by applying a luminance component in addition to the chroma (105).

The method of the present invention as described above may be stored in a recording medium (Side-ROM, RAM, ROM (Read Only Memory), flexible disk, hard disk, magneto-optical disk, etc.) in a form embodied by a program and read by a computer. it can. Such a process can be easily performed by a person having ordinary knowledge in the technical field to which the present invention belongs, and will not be described in further detail.
As described above, the present invention can be variously replaced, modified and changed by those having ordinary knowledge in the technical field to which the present invention belongs without departing from the technical idea of the present invention. It is not limited by the drawings.

It is a block diagram of a general chroma key composition device. It is explanatory drawing which showed the ideal chroma keying process. It is the general | schematic diagram which showed the synthetic | combination process of the conventional chroma key. It is explanatory drawing which showed the process of the chroma keying by a prior art. It is explanatory drawing which showed the process of the chroma keying by a prior art. FIG. 3 is a schematic diagram illustrating a chroma key composition process according to the present invention. 1 is a configuration diagram of an embodiment of an adaptive chroma key composition apparatus according to the present invention. FIG. FIG. 5 is an explanatory diagram of an embodiment illustrating a chroma key automatic adaptation process in an adaptive chroma key composition method according to the present invention; FIG. 5 is an explanatory diagram of an embodiment illustrating a chroma key automatic adaptation process in an adaptive chroma key composition method according to the present invention; It is explanatory drawing of one Example which showed the real-time chroma key production | generation process among the synthesis methods of the adaptive chroma key by this invention. It is explanatory drawing of one Example which showed the real-time chroma key production | generation process among the synthesis methods of the adaptive chroma key by this invention. It is explanatory drawing of the Example which showed the space division chroma key application process among the synthesis methods of the adaptive chroma key by this invention. It is explanatory drawing of the Example which showed the space division chroma key application process among the synthesis methods of the adaptive chroma key by this invention. It is explanatory drawing of the Example which showed the space division chroma key application process among the synthesis methods of the adaptive chroma key by this invention. FIG. 6 is an explanatory diagram of an embodiment showing a process of applying a chroma key based on luminance in an adaptive chroma key composition method according to the present invention; FIG. 6 is an explanatory diagram of an embodiment showing a process of applying a chroma key based on luminance in an adaptive chroma key composition method according to the present invention; FIG. 6 is an explanatory diagram of an embodiment showing a process of applying a chroma key based on luminance in the adaptive chroma key composition method according to the present invention;

Explanation of symbols

61 Video input unit 62 Pre-processing unit 63 Chroma keying processing unit 64 Post-processing unit

Claims (7)

In the chroma key composition device, a video input means for receiving an image captured through the camera and an alternative image of the background screen;
After scanning the input video, the entire screen is divided into spaces for each frame, a separate chroma key is generated for each divided plane, and a preprocessing means for generating a chroma key by tracking changes in chroma key values When,
Chroma keying processing means for performing chroma keying processing based on the chroma key and substituting and synthesizing pixels corresponding to the background screen chroma key in the input image including the object with the substitute image of the background screen;
An adaptive chroma key synthesizing apparatus comprising: the synthesized object and video output means for outputting a substitute video of the background screen. If a common part between the object and the background screen exists during the chroma keying process, post-processing means for minimizing a common hue area between the background screen and the object by additionally applying a luminance component The adaptive chroma key synthesizing device according to claim 1, further comprising: The adaptive chroma key composition apparatus according to claim 1 or 2, wherein the preprocessing means uses a chroma key of a previous frame in order to distinguish an object and a screen in each frame. In the chroma key composition device, a video input means for receiving an image captured through the camera and an alternative image of the background screen;
After scanning the input video, the entire screen is divided into spaces for each frame, a separate chroma key is generated for each divided plane, and a preprocessing means for generating a chroma key by tracking changes in chroma key values When,
Chroma keying processing means for performing chroma keying processing based on the chroma key and substituting and synthesizing pixels corresponding to the background screen chroma key in the input image including the object with the substitute image of the background screen;
At the time of the chroma keying process, if there is a common part (intersection set) between the object and the background screen, the luminance component is applied in addition to the chroma component to minimize the common hue area between the background screen and the object. Post-processing means,
An adaptive chroma key synthesizing apparatus comprising: the synthesized object and video output means for outputting a substitute video of the background screen. In the chroma key composition method, receiving an image captured through the camera and an alternative image of the background screen;
After scanning the input video, the entire screen is spatially divided every frame, a separate chroma key is generated for each divided surface, a change in the chroma key value is tracked, and a chroma key is generated,
Performing a chroma keying process based on the chroma key, substituting and synthesizing a pixel corresponding to a background screen chroma key with an alternative image of the background screen in an input image including an object;
An adaptive chroma key composition method, comprising: outputting the synthesized object and a substitute image of a background screen. During the chroma keying process, if there is a common part (crossing set) between the object and the background screen, a step of minimizing a common hue region between the background screen and the object by applying a luminance component in addition to the chroma component The adaptive chroma key synthesis method according to claim 5, further comprising: A chroma key composition device with a processor that receives images taken through the camera and a substitute image for the background screen;
After scanning the input video, the entire screen is divided into spaces for each frame, a separate chroma key is generated for each divided plane, and a chroma key is generated by tracking changes in chroma key values;
A function of performing chroma keying processing based on the chroma key, substituting and synthesizing a pixel corresponding to a background screen chroma key with an alternative image of the background screen in an input video including an object
At the time of the chroma keying process, when there is a common part between the object and the background screen, a function of additionally applying a luminance component to minimize a common hue area between the background screen and the object;
A recording medium that is read by a computer that records a program for realizing the synthesized object and a function of outputting a substitute image of a background screen.

Images