WO2007097517A1

WO2007097517A1 - Adjustive chroma key composition apparatus and method

Info

Publication number: WO2007097517A1
Application number: PCT/KR2006/005585
Authority: WO
Inventors: Dong-Kyo Keum; Youn-Ki Jeong; Nako Ahn
Original assignee: Upnon Co., Ltd.
Priority date: 2006-02-27
Filing date: 2006-12-20
Publication date: 2007-08-30
Also published as: KR100661528B1; JP2008527951A

Abstract

An adjustive chroma key composition apparatus comprising: an image input means that receives a photographed image and a background screen substitution image; a pre-processing means that scans the received images, space-divides the overall screen for each frame, generates a separate chroma key value for each of space division planes, traces change of the chroma key value, and generates a background screen chroma key based on the traced change of the chroma key value; a chroma keying processing means that processes chroma keying based on the chroma key, and substitutes pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; a post-processing means that minimizes a common color region (intersection) of the background screen image and the object image by applying a luminance component in addition to a chroma component when the chroma keying is processed; and an image output means that outputs a composition of the object image and the background screen substitution image.

Description

ADJUSTIVE CHROMA KEY COMPOSITION APPARATUS AND

METHOD

Technical Field

[1] The present invention relates to an adjustive chroma key composition apparatus and method and a computer-readable recording medium processing the method, and more particularly, to an adjustive chroma key composition apparatus and method which is capable of improving quality of a chroma key by actively adjusting the chroma key to various image conditions (for example, camera, illumination, direction, studio, etc.), and a computer readable recording medium processing the method. Background Art

[2] Color may be typically separated into brightness (or luma) and color difference (or chroma).

[3] A chroma key is a kind of image compositing technique in which a particular chroma is specified as a key and pixels corresponding to the specified chroma in a main image are substituted with a different image. A key means not a single chroma value but a range of chroma values in a chroma key execution algorithm.

[4] Accordingly, a background screen is needed as the key in order to execute the chroma key, and displacement of chroma values of the background screen has to be small. In addition, the background screen and an object (all bodies distinguishable from the background screen, including men) have to be separated from each other.

[5] As shown in Fig. 1, a general chroma key composition apparatus generates a composes image by specifying a particular chroma as a key and substituting a background screen image in a photographed picture, which consists of an object image and the background screen image, with a different background image based on the specified chroma.

[6] Fig. 2 shows a chroma distribution of the photographed picture. It is ideal if chroma components of the background screen image are completely separated from chroma components of the object image. It can be seen from Fig. 2 that the chroma components of the background screen image are so biased to green color that the chroma components of the background screen image are completely separated from the chroma components of the object image.

[7] The key is set based on a background screen chroma distribution, and a range of keying is set to be chroma-keyed around the set key. Here, the range of keying may have a indeterminate form, but typically has a diamond form centering around the key, as shown in Fig. 2. [8] Now, prior conditions for a successful chroma key technology may include image quality, key separation, equipments, etc., which will be described in detailed below.

[9] First, since the chroma key technique is a technique to deal with an image, quality of a source image has to be high. The quality of image depends greatly on a photographing technique, a photographing equipment, illumination, an actor's costume, condition and quality of the background screen, etc.

[10] On the other hand, in order to separate the background screen image from the object image, it has to be so designed in advance that color of the object image is equal to color of the background screen image. In addition, even if the color of the object image is different from the color of the background screen image, since they may become equal to each other under a certain illumination condition, an illumination condition has to be designed in advance. In addition, since the object image may fall within the same chroma range as the background screen image under a certain camera direction technique, a camera technique has to be designed in advance. In addition, since the object image may fall within the same chroma range as the background screen image due to reflection, shadow, illumination change, etc., which may be caused by a different object, such effects by the different object have to be considered in advance. In addition, in order to design and consider the illumination condition, the camera technique and the effects by the different object in advance, chroma key values of the background screen image have to be estimated in advance before the picture is photographed. In addition, since a camera distorts color and other states of an object to some degrees, a visible chroma key value of the background screen image is different from a chroma key value of an image signal inputted through the camera. Such a difference has to be minimized to photograph the picture according to a predetermined color design. In addition, chroma components of the background screen image may be changed due to the object, the illumination, the camera direction technique, etc., and a reference chroma value may also be changed accordingly. Therefore, it has to be so designed in advance that the reference chroma value is not changed.

[11] In addition to the camera direction technique, equipments are also important. A camera with noise, a background screen with irregular color or a background screen that reflects illumination makes it difficult to achieve a successful chroma keying. In addition, a studio that can keep out of unpredictable sunlight or illumination assists in improving quality of chroma keying.

[12] Now, a problem of the conventional chroma key technique will be described.

[13] First considering equipments, it is requisite to use a broadcasting camera with no noise and distortion, a dedicated screen that is less effected by illumination and objects and can maintain a uniform chroma distribution, and studio illuminators that can minimize shadow, color change, illumination change, etc. and a highly-planned studio. Conventionally, inputted images have been received as it is under such equipment conditions.

[14] In particular, considering an auto exposure effect of a general camera, it can be seen that color of a screen and an object is remarkably changed with time, when the object moves, or depending on the kind of camera, as shown in Figs. 4 and 5. It is impossible for the conventional chroma key technique to achieve the chroma keying for the screen whose color is changed with time. More specifically, as shown in Fig. 4, when exposure becomes reduced as a space occupied by white clothes increases, although a green screen observed by eyes remains unchanged, green color of the screen turns into visible black-like color and white clothes becomes dark through the camera. In addition, as shown in Fig. 5, as a space occupied by black clothes increases, the green screen becomes bright and the black clothes appears as bright and slightly-reddish gray color by A/E. Accordingly, it can be seen that it is not possible for the conventional chroma key technique to obtain the keying for the screen that is changed with time.

[15] On the other hand, since the convention chroma key technique uses only a predetermined key value, it takes many efforts to maintain the predetermined key value. In addition, there is no chroma key technique which can be equally applied to various studios having different conditions. In addition, there is no room for intervention of an ad lib (that is, impossibility of intervention of unspecified persons). In addition, quality of chroma key, which may be deteriorated due to unskilled direction or insufficient equipments, has no option but to make up for during later work (that is, difficulty of real time work).

[16] On the other hand, there is no additional means except for traditional chroma key substitution algorithms. In addition, all possible problems can not help but be overcome through equipments, direction and later work.

[17] On the other hand, a scope of application to broadcasting, movies, weather forecast, virtual reality systems is limited. In addition, it is difficult to provide products which can be used by people of a variety of classes due to problems of equipments, studio and direction.

[18] The above described conventional chroma key technique can not help but depend on the equipments (for example, cameras, illumination, direction, studios, etc.) in order to compensate for deteriorated equality of chroma key. Disclosure of Invention Technical Problem

[19] The present invention has made to overcome the above problems, and it is an object of the present invention to provide an adjustive chroma key composition apparatus and method which is capable of improving quality of a chroma key by adjusting the chroma key to various image conditions (for example, camera, illumination, direction, studio, etc.), and a computer readable recording medium processing the method.

[20] The above and/or other objects, advantages and aspects of the present invention will become apparent and more readily appreciated from the following description of the exemplary embodiments, and it will be readily appreciated that the above and/or other objects, advantages and aspects of the present invention are realized through means set forth in the appended claims and combination thereof. Technical Solution

[21] To achieve the above objects, according to an aspect, the present invention provides an adjustive chroma key composition apparatus comprising: an image input means that receives a photographed image and a background screen substitution image; a preprocessing means that scans the received images, space-divides the overall screen for each frame, generates a separate chroma key value for each of space division planes, traces change of the chroma key value, and generates a background screen chroma key based on the traced change of the chroma key value; a chroma keying processing means that processes chroma keying based on the chroma key, and substitutes pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; and an image output means that outputs a composition of the object image and the background screen substitution image.

[22] Preferably, the adjustive chroma key composition apparatus further comprises a post-processing means that minimizes a common color region of the background screen image and the object image by additionally applying a luminance component when the chroma keying is processed.

[23] Preferably, the pre-processing means distinguishes the object image from the background screen image in a frame using a chroma key of a previous frame immediately before the frame.

[24] According to another aspect, the present invention provides an adjustive chroma key composition apparatus comprising: an image input means that receives a photographed image and a background screen substitution image; a pre-processing means that scans the received images, space-divides the overall screen for each frame, generates a separate chroma key value for each of space division planes, traces change of the chroma key value, and generates a background screen chroma key based on the traced change of the chroma key value; a chroma keying processing means that processes chroma keying based on the chroma key, and substitutes pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; a post-processing means that minimizes a common color region (intersection) of the background screen image and the object image by applying a luminance component in addition to a chroma component when the chroma keying is processed; and an image output means that outputs a composition of the object image and the background screen substitution image.

[25] According to still another aspect, the present invention provides an adjustive chroma key composition method comprising the steps of: receiving a photographed image and a background screen substitution image; scanning the received images, space-dividing the overall screen for each frame, generating a separate chroma key value for each of space division planes, tracing change of the chroma key value, and generating a background screen chroma key based on the traced change of the chroma key value; processing chroma keying based on the chroma key, and substituting pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; and outputting a composition of the object image and the background screen substitution image.

[26] Preferably, the adjustive chroma key composition method further comprises minimizing a common color region (intersection) of the background screen image and the object image by applying a luminance component in addition to a chroma component when the chroma keying is processed.

[27] According to still another aspect, the present invention provides a computer readable recording medium recorded with a program to cause a processor of an chroma key composition apparatus to perform an adjustive chroma key composition method comprising the steps of: receiving a photographed image and a background screen substitution image; scanning the received images, space-dividing the overall screen for each frame, generating a separate chroma key value for each of space division planes, tracing change of the chroma key value, and generating a background screen chroma key based on the traced change of the chroma key value; processing chroma keying based on the chroma key, and substituting pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; and outputting a composition of the object image and the background screen substitution image.

Advantageous Effects

[28] According to the present invention, an unskilled man who can not well understand the chroma key technology can simply compose an image without using expensive equipment and direction capability, which results in ease implementation of the chroma key and reduction of total costs for implementation of the chroma key (inexpensive camera, inexpensive screen, no studio cost, no illumination cost, in- expensive chroma key equipment, no skilled man power, etc.).

[29] In addition, according to the present invention, the adjustive chroma key composition apparatus is capable of creating demands for entertainment in toyshops, singing rooms furnished with karaokes, self-photographing rooms, etc., in addition to existing broadcasting, movie, virtual reality systems. In addition, this apparatus can be used in driving, training simulations or broadcastings (individual broadcastings, small Internet broadcasting stations, video freelances, etc.) Brief Description of the Drawings

[30] Fig. 1 is a view showing a configuration of a general chroma key composition apparatus.

[31] Fig. 2 is an explanatory view showing an ideal chroma distribution of a picture.

[32] Fig. 3 is a schematic view showing a conventional chroma key composition process.

[33] Figs. 4 and 5 are explanatory views showing a conventional chroma keying process.

[34] Fig. 6 is a schematic view showing a chroma key composition process according to an embodiment of the present invention.

[35] Fig. 7 is a view showing a configuration of an adjustive chroma key composition apparatus according to an embodiment of the present invention.

[36] Figs. 8 and 9 are explanatory views showing a chroma key automatic adjustment process of an adjustive chroma key composition method according to an embodiment of the present invention.

[37] Figs. 10 and 11 are explanatory views showing a real time chroma key generation process of the adjustive chroma key composition method according to the embodiment of the present invention.

[38] Figs. 12 to 14 are explanatory views showing a space division chroma key application process of the adjustive chroma key composition method according to the embodiment of the present invention.

[39] Figs. 15 to 17 are explanatory views showing a luminance-based chroma key application process of the adjustive chroma key composition method according to the embodiment of the present invention. Mode for the Invention

[40] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following detailed description of the present invention, concrete description on related functions and/or constructions will be omitted if it is deemed that the functions and/or constructions may unnecessarily obscure the gist of the present invention.

[41] First, a chroma key composition process according an embodiment of the present invention will be described with reference to Fig. 6.

[42] The conventional chroma key technique has overcome the problem of deterioration of the quality of chroma key, however, a chroma key technique of the present invention is capable of improving the quality of chroma key by intelligently adjusting the chroma key through an improved algorithm, with conditions of such equipments and direction unchanged.

[43] That is, in the invention, a home camcoder or a monitoring camera may be used for the chroma key without requiring an expensive broadcasting camera. In addition, the chroma key technique of the present invention may be used not only sturdios but also in any places, for example, offices, living rooms, study rooms, etc. In addition, the chroma key technique of the present invention has little dependency on kind and position of an illuminator and may be used if only illumination so sufficient as to make objects distinguishable from each other is present. In addition, the chroma key technique of the present invention can well cope with diverse environments, movement of objects and overlapping colors without requiring separate direction. In addition, the chroma key technique of the present invention can make a chroma key application image in real time without requiring separate later work.

[44] Accordingly, as shown in Fig. 6, since any camera, any screen and any illumination may be used randomly, a chroma key composition apparatus (see Fig. 7) of the present invention is under a state where information related to preconditions and a chroma key is unknown.

[45] Fig. 7 is a view showing a configuration of an adjustive chroma key composition apparatus according to an embodiment of the present invention.

[46] As shown in Fig. 7, an adjustive chroma key composition apparatus according to an embodiment of the present invention includes an image input part 61 through which a photographed image and a background screen substitution image are inputted, a preprocessing part 62 that scans the inputted images, traces change of a chroma key value every frame, generates a chroma key based on the traced change of the chroma key value, a chroma keying processing part 63 that performs a chroma keying process based on the generated chroma key for substituting pixels, which correspond to a background screen chroma key, in the inputted image containing an object image with the background screen substitution image, and an image output part 65 that outputs a composition image including the object image and the background screen substitution image. Particularly, the adjustive chroma key composition apparatus may further include a post-processing part 64 that applies a luma component in addition to the chroma component to minimize a common region (intersection) of the object image and the background screen image.

[47] In addition, the pre-processing part 62 may further include a function of space- dividing the overall screen for each frame and generating a separate chroma key for each division plane.

[48] Now, functions of the components of the adjustive chroma key composition apparatus of the invention will be described in more detail.

[49] The pre-processing part 62 performs a chroma key automatic adjustment process

(see Figs. 8 and 9), a real time chroma key generation process (see Figs. 10 and 11) and a space division chroma key application process (see Fig, 12 to 14).

[50] First, the chroma key automatic adjustment process (Figs. 8 and 9) is a process of recognizing a chroma key value of the background screen image without specifying the chroma key value in advance. That is, this process can calculate an optimal chroma key value even under any places, any illumination, any cameras and any screens. Specifically, under a state where only the background screen is photographed, when an automatic adjustment button is pushed (Step 701), an overall image is scanned (Step 702), values close to particular RGB color are extracted (Steps 703 and 704), and a chroma key value is generated (Step 705). For example, the chroma key value is calculated with a sample extracted for green color.

[51] However, an object image and a screen image of a picture photographed under poor surroundings appear to have different colors in different conditions, for example, when an illuminator is changed from a fluorescent lamp to a glow lamp or a closed curtain is opened, when an automatic exposure function of a camera operates, when the object disappears or a different object appears, etc.

[52] In this case, chroma keying can not be obtained with an initial chroma key value of the screen, or although it is possible, quality of chroma keying may be deteriorated.

[53] Accordingly, to overcome this problem, the real time chroma key generation process is performed as shown in Figs. 10 and 11.

[54] The real time chroma key generation process (Figs. 10 and 11) is a process of obtaining the chroma key value of the screen for each frame by scanning the screen in real time (trace of change of the chroma key value).

[55] This process uses an initial chroma key value of a frame and a chroma key value of a previous frame to distinguish the object image from the screen image in each frame.

[56] In other words, when the chroma key value is generated for each frame, if there is no chroma key value of the previous frame (Step 801), the chroma key value of the previous frame is generated and stored through the chroma key automatic adjustment process (Step 802) (see Figs. 9). In addition, if there exists a chroma key value of the previous frame (Step 801), the screen is scanned (Step 803), pixels expected to form the screen are sampled using the chroma key value of the previous frame (Step 804), and then the chroma key value of the frame is generated with screen candidate pixels (step 805). Thereafter, the chroma key process is performed (Step 806). [57] Now, the space division chroma key application process will be described with reference to Figs, 12 to 14.

[58] Under general environments, even if a screen having uniform color is used, the screen may have an ununiform color distribution depending on position or reflection of illumination, or conditions of the object.

[59] Accordingly, when variation of a mean value on the color distribution of the screen becomes large, the overall screen can not cope with a calculated chroma key value. That is, as shown in Fig. 12, since an upper portion of the screen is bright and a lower portion thereof is dark (this frequently occurs in the general environments), the same chroma key value can not be applied to the upper and lower portions.

[60] Accordingly, since a chroma key value for the overall screen is insufficient to perform a complete chroma keying process, the overall screen is space-divided and separate chroma key values are obtained and applied for space division planes, as shown in Fig. 13. Thus, it can be seen from Fig. 13 that each space division plane has a uniform color distribution.

[61] In this manner, when the separate chroma key values are obtained for the space division planes (Fig. 13), unbalance of the color distribution of the screen due to irregular illumination, properties of cameras, etc.

[62] Specifically, when the chroma key value is generated for each frame, the screen is scanned (Step 902) and a chroma key value of the same space division plane of a previous frame is loaded for a new frame (Step 903), a screen candidate is sampled based on the chroma key value of the previous frame (Step 904) and then is further sampled based on color characteristics of the screen (Step 905), and then, a chroma key value of the new frame (Step 906). Thereafter, the chroma keying process is performed (Step 907).

[63] On the other hand, the post-processing part 64 performs a luminance-based chroma key application process (Figs. 15 to 17).

[64] As shown in Fig. 15, a chromatic color region is divided into 6 representative colors: magenta (M), blue (B), cyan (C), green (G), yellow (Y), and red (R).

[65] In addition, as shown in Fig. 16, color is represented using a YCbCr method of representing color using luminance (Y) and color difference signals Cb and Cr, instead of an ordinary RGB method of representing color by mixing three primary colors: red (R), green (G) and blue (B).

[66] Now, an improved luma-based chroma key technique will be described.

[67] The chroma key refers to a chroma-based keying technique. Accordingly, if keys have the same chroma component although they have different luminance components, they are recognized as the same key. Accordingly, as shown in Fig. 15, when there exists a common region (intersection) of the screen and the object, the conventional chroma key technique recognizes this region as the screen.

[68] However, since a human's vision system is more sensitive to the luma than the chroma, if colors have different luma components although they have the same chroma components, they are recognized as different from each other. That is, since the conventional chroma key technique disregards this point, it treats screen color reflected by a white paper as if the screen color is the screen.

[69] Although the conventional chroma key technique treats all the same chroma regions as the screen as a screen region, the improved luma-based chroma key technique does not treat all the same chroma regions as the screen region.

[70] That is, as shown in Fig. 16, the common region (intersection) of the screen and the object may be minimized by applying the lima component in addition to the chroma component.

[71] Now, the luma-based chroma key application process will be described with reference to a flow chart of Fig. 17.

[72] After a chroma keying process is performed for a picture (in the unit of pixel) inputted from a camera (Step 101), the inputted picture is maintained for an object region (Steps 102 and 103), and pixels, which correspond to the background screen chroma key, are substituted with a background screen substitution image for a screen region in the inputted picture including the object image (Steps 102 and 104). At this time, the common region (intersection) of the screen and the object can be minimized by applying the lima component in addition to the chroma component (Step 105).

[73] The above-described adjustive chroma key composition method can be implemented by a program that can be stored in a computer readable recording medium (for example, a CD-ROM, a RAM, a ROM, a floppy disk, a hard disk, a magneto- optical disk, etc.). This implementation is easily practiced by those skilled in the art and therefore, detailed explanation of which is omitted herein.

[74] While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims and equivalents thereof.

Claims

[1] An adjustive chroma key composition apparatus comprising: an image input means that receives a photographed image and a background screen substitution image; a pre-processing means that scans the received images, space-divides the overall screen for each frame, generates a separate chroma key value for each of space division planes, traces change of the chroma key value, and generates a background screen chroma key based on the traced change of the chroma key value; a chroma keying processing means that processes chroma keying based on the chroma key, and substitutes pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; and an image output means that outputs a composition of the object image and the background screen substitution image.

[2] The adjustive chroma key composition apparatus according to Claim 1, further comprising a post-processing means that minimizes a common color region of the background screen image and the object image by additionally applying a luminance component when the chroma keying is processed.

[3] The adjustive chroma key composition apparatus according to Claim 1 or 2, wherein the pre-processing means distinguishes the object image from the background screen image in a frame using a chroma key of a previous frame immediately before the frame.

[4] An adjustive chroma key composition apparatus comprising: an image input means that receives a photographed image and a background screen substitution image; a pre-processing means that scans the received images, space-divides the overall screen for each frame, generates a separate chroma key value for each of space division planes, traces change of the chroma key value, and generates a background screen chroma key based on the traced change of the chroma key value; a chroma keying processing means that processes chroma keying based on the chroma key, and substitutes pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; a post-processing means that minimizes a common color region (intersection) of the background screen image and the object image by applying a luminance component in addition to a chroma component when the chroma keying is processed; and an image output means that outputs a composition of the object image and the background screen substitution image.

[5] An adjustive chroma key composition method comprising the steps of: receiving a photographed image and a background screen substitution image; scanning the received images, space-dividing the overall screen for each frame, generating a separate chroma key value for each of space division planes, tracing change of the chroma key value, and generating a background screen chroma key based on the traced change of the chroma key value; processing chroma keying based on the chroma key, and substituting pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; and outputting a composition of the object image and the background screen substitution image.

[6] The adjustive chroma key composition method according to Claim 5, further comprising minimizing a common color region (intersection) of the background screen image and the object image by applying a luminance component in addition to a chroma component when the chroma keying is processed.

[7] A computer readable recording medium recorded with a program to cause a processor of an chroma key composition apparatus to perform an adjustive chroma key composition method comprising the steps of: receiving a photographed image and a background screen substitution image; scanning the received images, space-dividing the overall screen for each frame, generating a separate chroma key value for each of space division planes, tracing change of the chroma key value, and generating a background screen chroma key based on the traced change of the chroma key value; processing chroma keying based on the chroma key, and substituting pixels, which correspond to the background screen chroma key, with the background screen substitution image in the received image including an object image; and outputting a composition of the object image and the background screen substitution image.